I try to get csv data from kibana server by making get request. After this server send me response with data which i write the csv file and save somewhere.
Every time csv file with single line "pending" inside.
Kibana logs shows Status code 503
But if i put url in browser i can get csv file with correct data.
Probably i need to wait more for response from kibana.
From my perspective the problem is next: server can send me the response with single line "pending" which means that it needs more time for preparing right response.
I tried to increase client time but it doesn't work as well

client := http.Client{
    Timeout: 10 * time.Second,
}

The Idea with go routines and channels/wait group was next: they force Get request to wait for getting right data instead of "pending" and 503 status code

import (
    "encoding/json"
    "errors"
    "fmt"
    "io/ioutil"
    "log"
    "net/http"
    "sync"
)

type KibanaReportResponse struct {
    Path string `json:"path"`
}

var urlKibanaBase = "http://localhost:5601"
var urlKibanaPost = urlKibanaBase   "/api/reporting/generate/csv_searchsource?"
var urlParameters = "jobParams=(browserTimezone:Europe/Berlin,columns:!(),objectType:search,searchSource:(fields:!((field:'*',include_unmapped:true)),index:ec074c00-1f62-11ec-8056-8d208a1f6e77,parent:(filter:!(),index:ec074c00-1f62-11ec-8056-8d208a1f6e77,query:(language:kuery,query:'')),sort:!((_score:desc)),trackTotalHits:!t,version:!t),title:'Discover search [2021-09-27T09:19:44.977+02:00]')"

var urlWithParam = urlKibanaPost   urlParameters
func main() {
    var wg sync.WaitGroup
    wg.Add(1)
    pathCsvFile := getCsvPathFromKibana(urlWithParam)
    go getCsvFile(urlKibanaBase, pathCsvFile, &wg)
    defer wg.Wait()
}
func getCsvPathFromKibana(urlKib string) string {
    resKibana := KibanaReportResponse{}
    client := &http.Client{}
    if req, err := http.NewRequest("POST", urlKib, nil); err != nil {
        log.Println("Given a method, URL, andoptional body are wrong", err)
    } else {
        req.Header.Add("kbn-xsrf", "true")
        if res, err := client.Do(req); err != nil {
            log.Println("Probably problems depends on client policy or HTTP connection", err)
        } else {
            if err := json.NewDecoder(res.Body).Decode(&resKibana); err != nil {
                log.Println("Problem by Json decoding \n", err)
            } else {
                return resKibana.Path
            }
        }
    }

    return resKibana.Path
}
func getCsvFile(urlKibanaBase string, pathCsvFile string, wg *sync.WaitGroup) error {
    defer wg.Done()

    res, err := http.Get(urlKibanaBase   pathCsvFile)
    if err != nil {
        return err
    }
    defer res.Body.Close()
    switch res.StatusCode {
    case 200:
        dataBody, err := ioutil.ReadAll(res.Body)
        if err != nil {
            return err
        }
        err = ioutil.WriteFile("data.csv", dataBody, 0666)
        if err != nil {
            return err
        }
        return nil
    case 503:
        fmt.Println("probably 503/pending")
        return errors.New("probably 503/pending")
    }

    return nil
}

curl request

curl -v localhost:5601/api/reporting/jobs/download/ku5k3rxz00xs7fac46c0k12u 

* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 5601 (#0)
> GET /api/reporting/jobs/download/ku5k3rxz00xs7fac46c0k12u HTTP/1.1
> Host: localhost:5601
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< kbn-csv-contains-formulas: true
< kbn-max-size-reached: false
< content-disposition: inline; filename="Discover search [2021-09-27T09:19:44.977 02:00].csv"
< content-type: text/csv; charset=utf-8
< x-content-type-options: nosniff
< referrer-policy: no-referrer-when-downgrade
< kbn-name: 3ae0cbefece4
< kbn-license-sig: 58d9fcb437ac7d3ac54d538e6d5ff9a039fde738ed3a940fa530e6d8b6ef6740
< cache-control: private, no-cache, no-store, must-revalidate
< content-length: 9013976
< vary: accept-encoding
< accept-ranges: bytes
< Date: Wed, 29 Sep 2021 14:06:19 GMT
< Connection: keep-alive
< Keep-Alive: timeout=120
< 
{ [13865 bytes data]
100 8802k  100 8802k    0     0  57.6M      0 --:--:-- --:--:-- --:--:-- 57.6M
* Connection #0 to host localhost left intact

Screenshot that shows that i can get data by typing url path to the browser

Update:
I try to call my function recursively. But it is bad solution even if it's working now.

    case 503:
        getCsvFile(urlKibanaBase, pathCsvFile)

CodePudding user response：

Kibana Server needs some time to prepare response. I've added a bit of code for signaling the channel to wait some time and after waiting repeat request to client

if res.StatusCode != 200 {
                    time.Sleep(30 * time.Second)
                }