`vendor.add` doesn't expose dependencies to dev

Migrating a legacy project from 2.7 and Ubuntu 18.04 [piecemeal, Python 3.10 & 22.04 next!] from vendored dependencies to requirements.txt. Removed dependencies from project root and enumerated them in my requirements.txt.

My requirements.txt contains google-cloud-storage==1.44.0 and was venv-2-7/bin/python -m pip install -t lib -r requirements.txt with a appengine_config.py in same dir as app.yaml with:

# From https://cloud.google.com/appengine/docs/legacy/standard/python/tools/using-libraries-python-27
import os

from google.appengine.ext import vendor

vendor.add('lib')
vendor.add(os.path.join(os.path.dirname(os.path.realpath(__file__)), 'lib'))

How do I resolve this error? - Attempted venv-2-7/bin/python -c 'import google.cloud.storage' which worked, but:

$ venv-2-7/bin/python /google-cloud-sdk/platform/google_appengine/dev_appserver.py --host 127.0.0.1 .

Errors [from PyCharm & manually] with:

ImportError: No module named google.cloud.storage

CodePudding user response：

I can reproduce your error.

I used to have problems with protobuf (the error was somewhat similar to what you're getting for storage). The solution was to update the google namespace package. I tried the same solution now (for storage) and your error went away

Update the code in your appengine_config.py to

    from google.appengine.ext import vendor
    import google, os

    lib_dir = os.path.join(os.path.dirname(__file__), 'lib')
    google.__path__.append(os.path.join(lib_dir, 'google'))

    vendor.add('lib')

After updating appengine_config.py, the line import google.cloud.storage no longer gives error of No module named cloud.storage.

CodePudding user response：

I didn't try to repro your error but believe you. From initial glance, it looks like your appengine_config.py is incomplete. This suffices when you have non-GCP 3P dependencies:

from google.appengine.ext import vendor

# Set PATH to your libraries folder.
PATH = 'lib'
# Add libraries installed in the PATH folder.
vendor.add(PATH)

However, if your requirements.txt has GCP client libraries, e.g., google-cloud-*, your appengine_config.py needs to use pkg_resources to support their use:

import pkg_resources
from google.appengine.ext import vendor

# Set PATH to your libraries folder.
PATH = 'lib'
# Add libraries installed in the PATH folder.
vendor.add(PATH)
# Add libraries to pkg_resources working set to find the distribution.
pkg_resources.working_set.add_entry(PATH)

To bring in pkg_resources, you need to add setuptools and grpcio to your app.yaml, and to use google-cloud-storage specifically, you need to also add ssl:

runtime: python27
threadsafe: yes
api_version: 1

handlers:
- url: /.*
  script: main.app

libraries:
- name: grpcio
  version: latest
- name: setuptools
  version: latest
- name: ssl
  version: latest

All of these 3P pkg games "go away" when you finally upgrade to Python 3 where your requirements.txt remains the same, but you delete appengine_config.py and replace your app.yaml with the following (if you're not serving static files):

runtime: python310

These same instructions can also be found in the App Engine documentation on the migrating bundled services page. That page basically says what I just did above for both Python 2 and 3.

When you're ready to upgrade to Python 3 and/or get off App Engine bundled services (NDB, Task Queue [push & pull], Memcache, Blobstore, etc.) to standalone Cloud equivalents (Cloud NDB, Cloud Tasks [push] or Cloud Pub/Sub [pull], Cloud Memorystore, Cloud Storage (GCS), etc.), or switch to Cloud Functions or Cloud Run, I've produced (well, still producing) a modernization migration series complete with code samples, codelab tutorials, and videos, all of which complement the official migration docs. You can find more info including links to all those resources as its open source repo. In particular, your inquiry covers GCS, and that migration is covered by "Module 16." The Mod16 app is a sample that works with GCS and is a migration of its analog Mod15 app based on Blobstore.

</ADVERTISEMENT>