My application runs on Google App Engine and uses Google Cloud Datastore. I was alerted by one of my users that some entries they are associated with and had previously been seeing, are not appearing to them.

Indeed, when I query the Datastore for these entities (with a single property filter), they are not returned. I was able to find them via querying on a different property and after writing them to the datastore, the index is updated and they are returned in the query.

Perhaps technically my query is not guaranteed to return the entities as it is weakly consistent, but none of the entities were changed recently and usually any inconsistent results are resolved quite quickly. (it has been several days now)

So it seems like the index entries for this property on these entities were lost or damaged somehow. What to do? Wait and hope the index will be regenerated? I can write entities for this user to the datastore to regenerate the index...but doing it for all my users is not really an option.

The only similar case I can see on SA is this question: Seems the indexes are missing for new entities created since some time late June 1st, 2015 which resulted from this incident: https://status.cloud.google.com/incident/appengine/15015 but no similar incident occurred recently, according to the status dashboard.

CodePudding user response：

Have you changed the status of which properties are indexed?

If you do, previously existing entities will only be updated in the indices the next time they are put.

That could cause older entities to not show up in a query.

CodePudding user response：

An App Engine engineer kindly pointed me towards the root of the issue - my query filter property was on a UserProperty type.

From the docs at https://cloud.google.com/appengine/docs/legacy/standard/python/users/userobjects :

A User value with an email address that does not represent a Google account at the time it is created will never match a User value that represents a real user.

My user's email was not a Google Account but he likely recently created one, causing the user_id() to go from None to an integer id.

This means that from now, when I do a query like this:

u = User('name@non_google_domain.com')
Entity.all().filter('user_property', u)

Internally, u is now actually looked up by the combination of name@non_google_domain.com and the integer id, instead of the combination of name@non_google_domain.com and None, causing my entities not to be returned.

Indeed, examining e._entity['user_property'] of a returned entity, the new ones are of the form User('name@non_google_domain.com', _user_id='12345678912345678900') and the old ones are User('name@non_google_domain.com').

UserProperty is no longer recommended for use, because of issues like this.