3

I'm trying to use ExAws library along with minio. I try to upload the file:

filepath
|> S3.Upload.stream_file
|> S3.upload(bucket_name, path)
|> ExAws.request!

and I get an error:

[error] #PID<0.1041.0> running Myapp.Endpoint terminated
Server: localhost:4000 (http)
Request: POST /resources/upload
** (exit) an exception was raised:
    ** (MatchError) no match of right hand side value: %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<InitiateMultipartUploadResult xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"><Bucket>user-1</Bucket><Key>docs/tsetup.0.9.51.tar.xz</Key><UploadId>35e9c8b0-ac34-4411-b39b-790a0104df93</UploadId></InitiateMultipartUploadResult>", headers: [{"Accept-Ranges", "bytes"}, {"Server", "Minio/DEVELOPMENT.GOGET (linux; amd64)"}, {"Vary", "Origin"}, {"X-Amz-Request-Id", "K4O7RNNSRLAMSL26"}, {"Date", "Tue, 11 Oct 2016 20:53:58 GMT"}, {"Content-Type", "text/xml; charset=utf-8"}, {"Transfer-Encoding", "chunked"}], status_code: 200}
        (ex_aws) lib/ex_aws/s3/upload.ex:39: ExAws.S3.Upload.initialize!/2
        (ex_aws) lib/ex_aws/s3/upload.ex:80: ExAws.Operation.ExAws.S3.Upload.perform/2
        (ex_aws) lib/ex_aws.ex:41: ExAws.request!/2
        (Myapp) web/models/resource.ex:68: Myapp.Resource.put_file_to_cloud/1
        (Myapp) web/controllers/resource_controller.ex:59: Myapp.ResourceController.upload/2
        (Myapp) web/controllers/resource_controller.ex:1: Myapp.ResourceController.action/2
        (Myapp) web/controllers/resource_controller.ex:1: Myapp.ResourceController.phoenix_controller_pipeline/2
        (Myapp) lib/Myapp/endpoint.ex:1: Myapp.Endpoint.instrument/4
        (Myapp) lib/phoenix/router.ex:261: Myapp.Router.dispatch/2
        (Myapp) web/router.ex:1: Myapp.Router.do_call/2
        (Myapp) lib/Myapp/endpoint.ex:1: Myapp.Endpoint.phoenix_pipeline/1
        (Myapp) lib/plug/debugger.ex:123: Myapp.Endpoint."call (overridable 3)"/2
        (Myapp) lib/Myapp/endpoint.ex:1: Myapp.Endpoint.call/2
        (plug) lib/plug/adapters/cowboy/handler.ex:15: Plug.Adapters.Cowboy.Handler.upgrade/4
        (cowboy) src/cowboy_protocol.erl:442: :cowboy_protocol.execute/4

I see by the response that the file upload begins. This mini returns the answer is not as expected exaws? or the problem is in something else?

Upd: after adding sweet_xml now get another error:

[error] GenServer #PID<0.593.0> terminating
** (MatchError) no match of right hand side value: nil
    (ex_aws) lib/ex_aws/s3/upload.ex:69: ExAws.S3.Upload.upload_chunk!/3
    lib/flow/materialize.ex:493: anonymous fn/4 in Experimental.Flow.Materialize.mapper/2
    lib/flow/materialize.ex:462: Experimental.Flow.Materialize."-mapper_ops/1-lists^foldl/2-1-"/3
    lib/flow/materialize.ex:462: anonymous fn/5 in Experimental.Flow.Materialize.mapper_ops/1
    lib/flow/map_reducer.ex:57: Experimental.Flow.MapReducer.handle_events/3
    lib/gen_stage.ex:2190: Experimental.GenStage.consumer_dispatch/7
    lib/gen_stage.ex:2313: Experimental.GenStage.take_pc_events/3
    (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:667: :gen_server.handle_msg/5
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: {:"$gen_consumer", {#PID<0.592.0>, #Reference<0.0.2.533>}, [{<<255, 216, 255, 224, 0, 16, 74, 70, 73, 70, 0, 1, 1, 1, 0, 72, 0, 72, 0, 0, 255, 219, 0, 67, 0, 2, 1, 1, 2, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 3, 5, 3, 3, 3, 3, 3, 6, ...>>, 1}]}
State: {%{#Reference<0.0.2.533> => nil}, %{consumers: [{#Reference<0.0.2.544>, #Reference<0.0.2.546>}], done: [#Reference<0.0.2.533>], done?: false, trigger: #Function<2.69990182/4 in Experimental.Flow.Window.Global.materialize/4>}, {0, 4}, [], #Function<29.77971684/4 in Experimental.Flow.Materialize.mapper_ops/1>}
[error] Ranch protocol #PID<0.587.0> (:cowboy_protocol) of listener Myapp.Endpoint.HTTP terminated
** (exit) an exception was raised:
    ** (MatchError) no match of right hand side value: nil
        (ex_aws) lib/ex_aws/s3/upload.ex:69: ExAws.S3.Upload.upload_chunk!/3
        lib/flow/materialize.ex:493: anonymous fn/4 in Experimental.Flow.Materialize.mapper/2
        lib/flow/materialize.ex:462: Experimental.Flow.Materialize."-mapper_ops/1-lists^foldl/2-1-"/3
        lib/flow/materialize.ex:462: anonymous fn/5 in Experimental.Flow.Materialize.mapper_ops/1
        lib/flow/map_reducer.ex:57: Experimental.Flow.MapReducer.handle_events/3
        lib/gen_stage.ex:2190: Experimental.GenStage.consumer_dispatch/7
        lib/gen_stage.ex:2313: Experimental.GenStage.take_pc_events/3
        (stdlib) gen_server.erl:601: :gen_server.try_dispatch/4
        (stdlib) gen_server.erl:667: :gen_server.handle_msg/5
        (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3

at the same time, I see that the download is in wireshark trace, but in the storage folder is empty

I looked more at the response from Minio, and I see that there is etag: Frame 1138: 344 bytes on wire (2752 bits), 344 bytes captured (2752 bits) on interface 0 Ethernet II, Src: 00:00:00_00:00:00 (00:00:00:00:00:00), Dst: 00:00:00_00:00:00 (00:00:00:00:00:00) Internet Protocol Version 4, Src: 127.0.0.1, Dst: 127.0.0.1 Transmission Control Protocol, Src Port: 9000 (9000), Dst Port: 42324 (42324), Seq: 515, Ack: 1539815, Len: 278 Hypertext Transfer Protocol HTTP/1.1 200 OK\r\n Accept-Ranges: bytes\r\n Etag: "2501de281894f7a1f79e8e0d6b8e7d85"\r\n Server: Minio/DEVELOPMENT.GOGET (linux; amd64)\r\n Vary: Origin\r\n X-Amz-Request-Id: M81ULLKCXKH0KRXD\r\n Date: Fri, 14 Oct 2016 15:00:16 GMT\r\n Content-Length: 0\r\n Content-Type: text/plain; charset=utf-8\r\n \r\n [HTTP response 2/2] [Time since request: 0.008850143 seconds] [Prev request in frame: 1016] [Prev response in frame: 1028] [Request in frame: 1032] UPD: the problem is solved by the author of the ExAws library - https://github.com/CargoSense/ex_aws/issues/253

Marsel.V
  • 1,015
  • 1
  • 12
  • 28
  • what configuration did you do to use Minio as cloud storage in you elixir application? I am not able to find the config for endpoint. `config :ex_aws, debug_requests: true, region: "eu-central-1"` – swapab Jul 28 '18 at 19:56
  • Never-mind. I got the configuration option: `s3: [ scheme: "http://", host: "localhost", region: "eu-west-1", port: 9000]` – swapab Jul 28 '18 at 20:57

1 Answers1

2

ExAWS upload is expecting a response of %{body: %{upload_id: upload_id}}, but you are getting unparsed XML as body. S3 require XML codec, so make sure you have XML parsing library sweet_xml added to your deps.

HIH

Ali Naqvi
  • 116
  • 3
  • Thank you! after adding sweet_xml I get another error. I add it to the question – Marsel.V Oct 13 '16 at 20:10
  • 2
    Uploading in chunks to AWS returns with ETag with each chunk request and library code is expecting that 'etag' in response, as it couldn't find that info in response from AWS, so match is failing. Please review the detailed HTTP response from AWS for each chunk upload and see if that contains the ETag field. ETag response from AWS is necessary to complete the multipart upload. – Ali Naqvi Oct 14 '16 at 05:50
  • etag is present in the response from the minio – Marsel.V Oct 14 '16 at 15:19
  • Seems like this issue is fixed via https://github.com/CargoSense/ex_aws/issues/253. @Marsel.V can you test and if it works as intended? – koolhead17 Apr 13 '17 at 06:34
  • @koolhead17Yes, I immediately added a response in which I indicated that the problem had been resolved. But my answer was deleted – Marsel.V Apr 13 '17 at 07:53