I have a use case where I need to make a S3 navigator which should allow users to navigate s3 files and view them without giving any sort of aws access. So users need not have aws credentials configured on their systems.
The approach I tried is to create a python app using tkinter and allow access to s3 using api gateway proxy to s3 docs. However, all this works fine for txt files in s3 but I have to read feather files and it's causing
s3_data=pd.read_feather("https://<api_gateway>/final/s3?key=naxi143/data.feather")
File "C:\Users\<User>\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\pandas\io\feather_format.py", line 130, in read_feather
return feather.read_feather(
File "C:\Users\<User>\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\pyarrow\feather.py", line 218, in read_feather
return (read_table(source, columns=columns, memory_map=memory_map)
File "C:\Users\<User>\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\site-packages\pyarrow\feather.py", line 239, in read_table
reader = _feather.FeatherReader(source, use_memory_map=memory_map)
File "pyarrow\_feather.pyx", line 75, in pyarrow._feather.FeatherReader.__cinit__
File "pyarrow\error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow\error.pxi", line 114, in pyarrow.lib.check_status
OSError: Verification of flatbuffer-encoded Footer failed.
error in my python code. Not sure if some settings are misconfigured on api gateway side.
s3_data=pd.read_feather("https://<api_gateway>/final/s3?key=naxi143/data.feather")
Is there any other way to make this work without involving aws credentials ?
Update
Looks like api gateway has a payload limit of 10 MB which leaves this solution out of scope for me as most of my data is more than that size. Is there any other way to achieve the same without using aws credentials ?