Multipart/form curl from S3 or cloud storage

Question

I am trying to make a POST request to API through Airflow using BashOperator.

I tried a cURL to upload a file into API.

When I tested it on Postman it works because the file is in my local filesystem(used Select file option on body tab), but in Airflow the file is S3 storage.

Is there any possibility to load files directly from S3?

I already tried to fill the field --form file=@'s3 path' but it didn't work.

Answer 1

You need to pull your file first from S3 and then POST it via the API. Both can be achieved via curl (providing that you have the right authentication/authorization).

You can even pipe the output of the first CURL into the second one.

You can find similar case here: use pipe for curl data just replace the "cat file" to use the right curl command that will GET the file from s3.