I would like to be able to specify maxResults when using the golang BigQuery library. It isn't clear how to do this, though. I don't see it as an option in the documentation, and I have browsed the source to try to find it but I only see some sporadic usage in seemingly functionality not related to queries. Is there a way to circumvent this issue?

CodePudding user response：

I think there is no implemented method in the SDK for that but after looking a bit, I found this one: request

You could try to execute an HTTP GET specifying the parameters (you can find an example of the use of parameters here: query_parameters)

CodePudding user response：

By default the google API iterators manage page size for you. The RowIterator returns a single row by default, backed internally by fetched pages that rely on the backend to select an appropriate size.

If however you want to specify a fixed max page size, you can use the google.golang.org/api/iterator package to iterate by pages while specifying a specific size. The size, in this case, corresponds to maxResults for BigQuery's query APIs.

See https://github.com/googleapis/google-cloud-go/wiki/Iterator-Guidelines for more general information about advanced iterator usage.

Here's a quick test to demonstrate with the RowIterator in bigquery. It executes a query that returns a row for each day in October, and iterates by page:

func TestQueryPager(t *testing.T) {
    ctx := context.Background()

    pageSize := 5

    client, err := bigquery.NewClient(ctx, "your-project-id here")
    if err != nil {
        t.Fatal(err)
    }
    defer client.Close()

    q := client.Query("SELECT * FROM UNNEST(GENERATE_DATE_ARRAY('2022-10-01','2022-10-31', INTERVAL 1 DAY)) as d")
    it, err := q.Read(ctx)
    if err != nil {
        t.Fatalf("query failure: %v", err)
    }
    pager := iterator.NewPager(it, pageSize, "")
    var fetchedPages int
    for {
        var rows [][]bigquery.Value
        nextToken, err := pager.NextPage(&rows)
        if err != nil {
            t.Fatalf("NextPage: %v", err)
        }
        fetchedPages = fetchedPages   1
        if len(rows) > pageSize {
            t.Errorf("page size exceeded, got %d want %d", len(rows), pageSize)
        }
        t.Logf("(next token %s) page size: %d", nextToken, len(rows))
        if nextToken == "" {
            break
        }
    }

    wantPages := 7
    if fetchedPages != wantPages {
        t.Fatalf("fetched %d pages, wanted %d pages", fetchedPages, wantPages)
    }
}