No Such File Error When Trying to Create Local Cache of s3 Object-CodePudding

I'm trying to use Fsspec to create a local cache of a data file store in a public access bucket on AWS s3. The public access bucket is located here.

It is 100% necessary for me to do this in local file cache because this intended to scale and I don't want to physically download each individual file. I'm trying to do this with an API call fsspec that uses the underlying botocore framework. A simple, minimal reproducible example looks like this:

import fsspec

url = 'simplecache::s3://noaa-nbm-grib2-pds/blend.20211019/01/core/blend.20211019/01/core/blend.t01z.core.f001.co.grib2
'
of = fsspec.open_local(url, s3={'anon' : True}, filecache={'cache_storage':'/tmp/files'})

Running the above reproduces the error if all dependencies are installed. I've tried switching the url out with the coped link address of the file (the simplecache scheme was based off of some fsspec documentation here), but that still gives the following error:

ValueError: open_local can only be used on a filesystem which has attribute local_file=True

Is there a particular url that should be used for something such as this? Here's the direct url to the object (with injected parenthesis to prevent a link to immediately download a file)

# Remove parenthesis to get full file link which is a direct file download
https://noaa-nbm-grib2-pds.s3.amazonaws.com/blend.20211019/01/core/(blend.t01z.core.f001.co.grib2)

CodePudding user response：

The following works fine:

fsspec.open_local("simplecache::https://noaa-nbm-grib2-pds.s3.amazonaws.com/blend.20211019/01/core/blend.t01z.core.f001.co.grib2")

but directly accessing the file over the s3 interface fails with FileNotFound. This probably indicates that the permissions are not set up correctly, but fsspec is still behaving as you would expect.

>>> s3 = fsspec.filesystem("s3", anon=True)
>>> s3.info("s3://noaa-nbm-grib2-pds/blend.20211019/01/core/blend.20211019/01/core/blend.t01z.core.f001.co.grib2")
FileNotFoundError