Datastore transaction - hitting entity write limit-CodePudding

The Problem

Using the golang cloud.google.com/go/datastore package to create a transaction, perform a series of getMulti's, and putMulti's, on commit of this transaction I'm confronted with an an entity write limit error.

2021/12/22 09:07:18 err: rpc error: code = InvalidArgument desc = cannot write more than 500 entities in a single call

The Question

My question is how do you create a transaction with more than 500 writes?

While I want my operation to remain atomic, I can't seem to solve this write limit error for a transaction and the set of queries run just fine when I test on an emulator, writing in batches of 500.

What I've Tried

please excuse the sudo code but I'm trying to get the jist of what I've done All in one

    transaction, err := datastoreClient.NewTransaction(ctx)
    transaction.PutMulti(allKeys, allEntities)
    transaction.commit()
// err too many entities written in a single call

Batched in an attempt to avoid the write limit

    transaction, err := datastoreClient.NewTransaction(ctx)
    transaction.PutMulti(first500Keys, first500Entities)
    transaction.PutMulti(second500Keys, second500Entities)
    transaction.commit()
// err too many entities written in a single call

A simple regular putmulti also fails

    datastoreClient.PutMulti(ctx,allKeys, allEntities)
// err too many entities written in a single call

What Works

Non-atomic write to the datastore

    datastoreClient.PutMulti(ctx,first500Keys, first500Entities)
    datastoreClient.PutMulti(ctx,second500Keys, second500Entities)

here's the real code that I used for the write, either as a batched transaction or regular putMulti

    for i := 0; i < (len(allKeys) / 500); i   {
        var max int = (i   1) * 500
        if len(allKeys) < max {
            max = len(allKeys) % 500
        }

        _, err = svc.dsClient.PutMulti(ctx, allKeys[i*500:max], allEntities[i*500:max])
        if err != nil {
            return
        }
    }

Where I'm Lost

so in an effort to keeping my work atomic, is there any method to commit a transaction that has more than 500 entities written in it?

CodePudding user response：

Nothing you can do. This limit is enforced by the platform to ensure scalability and to prevent performance degradation. You can't write more than 500 entities in a single transaction.

It's possible to change the limit on Google's side, but nothing you can do on your side.