I've been reading about dask and how it can work with files on s3. I was wondering what would happen if the worker's disk was too small to fully store the intermediate structures needed in a complex computation on incredibly large objects. Will dask spill temporary objects back to s3 and use it as more temporary disk space (if given the right permissions)? Or would the computation just fail?

CodePudding user response：

You could, in principle, set up a system to spill intermediates to S3, but there is no existing mechanism to automatically do this, and I have not heard of anyone having tried.