3

I am trying to write an API in python (Falcon) to accept a file from multipart-form parameter and put the file in MINIO object storage. The problem is I want to send the file to Minio without saving it in any temp location.

Minio-python client has a function using which we can send the file.

  `put_object(bucket_name, object_name, data, length)`

where data is the file data and length is total length of object. For more explanation: https://docs.min.io/docs/python-client-api-reference.html#put_object

I am facing problem accumulating the values of "data" and "length" arguments in the put_object function.

The type of file accepted in the API class is of falcon_multipart.parser.Parser which cannot be sent to Minio.

I can make it work if I write the file to any temp location and then read it from the desired location and send.

Can anyone help me finding a solution to this?

I tried reading file data from the Parser object and tried converting the file to bytes io.BytesIO. But it did not work.

def on_post(self,req, resp):
  file = req.get_param('file')
  file_data = file.file.read()
  file_data= io.BytesIO(file_data)

  bucket_name = req.get_param('bucket_name')

  self.upload_file_to_minio(bucket_name, file, file_data)



def upload_file_to_minio(self, bucket_name, file, file_data):

  minioClient = Minio("localhost:9000", access_key='minio', secret_key='minio', secure=False)

  try:
    file_stat = sys.getsizeof(file_data)
    #file_stat = file_data.getbuffer().nbytes
    minioClient.put_object(bucket_name, "SampleFile" , file, file_stat)

  except ResponseError as err:
    print(err)


Traceback (most recent call last):
  File "/home/user/.local/lib/python3.6/site-packages/minio/helpers.py", line 382, in is_non_empty_string
    if not input_string.strip():
AttributeError: 'NoneType' object has no attribute 'strip'
impromptu_user
  • 41
  • 4

1 Answers1

1

A very late answer to your question. As of Falcon 3.0, this should be possible leveraging the framework's native multipart/form-data support.

There is an example how to perform the same task to AWS S3: How can I save POSTed files (from a multipart form) directly to AWS S3?

However, as I understand, MinIO requires either the total length, which is unknown, or alternatively, it requires you to wrap the upload as a multipart form. That should be doable by reading reasonably large (e.g., 8 MiB or similar) chunks into the memory, and wrapping them as multipart upload parts without storing anything on disk.

IIRC Boto3's transfer manager does something like that under the hood for you too.

Vytas
  • 754
  • 5
  • 14