Does dask.s3fs actually use the cache?

Question

When you call S3FileSystem().open(..., fill_cache=True) it looks like it still goes to S3, still does the ssl handshake etc. Is this the intended behaviour that it does not cache?

Answer 1

The fill_cache option has a very specific meaning

    If seeking to new a part of the file beyond the current buffer,
    with this True, the buffer will be filled between the sections to
    best support random access. When reading only a few specific chunks
    out of a file, performance may be better if False.

i.e., it only makes a difference when you randomly access portions of a file while that file is kept open. An initial SSL connection/handshake is required regardless, the only thing that might change is the exact range of bytes that is requested.

Also note that s3ss uses a boto Session, so SSL connections should in general be pooled and reused where possible.