Get request returns data in Thunder client/Postman but gives blank data in Golang code-CodePudding

I am trying to fetch data from an API using Golang net/http. When I am using Thunder client from VS Code or even Postman, I am getting proper data but when I am trying to fetch the data from Golang code, I get an empty response.

The data is fetched in 2 steps:

Fetch cookies using an initial GET req (this part is working fine in both)
Use the cookies to make another GET req for fetching the required data. (This is the step which is giving blank response in Golang and is named Historical Data in the Postman link given below)

Here's the Golang code. The code might be a little long but just because of multiple lined of adding headers.

var BaseURL string = "https://www.nseindia.com"

func ReqConfig() *http.Request {
    req, _ := http.NewRequest("GET", BaseURL, nil)
    req.Header.Add("Accept", "*/*")
    req.Header.Add("Accept-Encoding", "gzip, deflate, br")
    req.Header.Add("Accept-Language", "en-GB,en-US;q=0.9,en;q=0.8")
    req.Header.Add("Connection", "keep-alive")
    req.Header.Add("Host", "www.nseindia.com")
    req.Header.Add("Referer", "https://www.nseindia.com/get-quotes/equity")
    req.Header.Add("X-Requested-With", "XMLHttpRequest")
    req.Header.Add("sec-fetch-dest", "empty")
    req.Header.Add("sec-fetch-mode", "cors")
    req.Header.Add("pragma", "no-cache")
    req.Header.Add("sec-fetch-site", "same-origin")
    req.Header.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36")
    fmt.Println(1, req.Header.Get("Cookie"))

    res, err := http.DefaultClient.Do(req)
    if err != nil {
        panic(err)
    }
    defer res.Body.Close()
    for _, cookie := range res.Cookies() {
        req.AddCookie(cookie)
    }

    // TODO: Remove the need to call this API twice. This is just a temporary fix.
    res, err = http.DefaultClient.Do(req)
    if err != nil {
        panic(err)
    }
    defer res.Body.Close()
    for _, cookie := range res.Cookies() {
        req.AddCookie(cookie)
    }


    cookies := req.Cookies()
    for i := 0; i < len(cookies); i   {
        for j := i   1; j < len(cookies); j   {
            if cookies[i].Name == cookies[j].Name {
                cookies = append(cookies[:j], cookies[j 1:]...)
                j--
            }
        }
    }
    req.Header.Del("Cookie")
    for _, cookie := range cookies {
        req.AddCookie(cookie)
    }
    fmt.Println("Fetched cookies")

    return req
}


func HistoricalEQ(symbol string, from string, to string, series string) {
    req := ReqConfig()

    query := req.URL.Query()
    query.Add("symbol", symbol)
    query.Add("from", from)
    query.Add("to", to)
    query.Add("series", "[\"" series "\"]")
    req.URL.RawQuery = query.Encode()
    req.URL.Path = "/api/historical/cm/equity"

    client := &http.Client{Timeout: 40 * time.Second}
    res, err := client.Do(req)
    if err != nil {
        panic(err)
    }
    defer res.Body.Close()

    var data map[string]interface{}
    json.NewDecoder(res.Body).Decode(&data)

        // Prints `map[]` and not the whole json data which is provided in Postman req
    fmt.Println(data)
}


func main() {
    symbol := "PAYTM"
    series := "EQ"
    from_date := time.Date(2023, 1, 1, 0, 0, 0, 0, time.Local).Format("02-01-2006")
    to_date := time.Date(2023, 1, 24, 0, 0, 0, 0, time.Local).Format("02-01-2006")
    HistoricalEQ(symbol, from_date, to_date, series)
}

If you are able to fetch data from some other way in Golang only from GET https://www.nseindia.com/api/historical/cm/equity?symbol=PAYTM&series=["EQ"]&from=28-12-2022&to=28-01-2023, then that would also solve my issue. You can check out the website frontend at https://www.nseindia.com/get-quotes/equity?symbol=PAYTM. The GET req I am asking can be triggered by going to Historical data tab and clicking on filter button

CodePudding user response：

1️⃣ decoding error handling was missed

err := json.NewDecoder(res.Body).Decode(&data)
if err != nil {
    log.Fatalf("decode request: %v", err)
}

invalid character '\x1f' looking for beginning of value

2️⃣ looks like the response data has been compressed (gzip data start with the magic sequence 0x1f 0x8b). If you check response's Headers you see

...
Content-Encoding:[gzip]