AWS Lambda replacing certain characters from attachments of multipart-form data with replacement character(U+FFFD)

Question

I am using AWS lambda function to receive a multipart request with attachments and upload them to S3.

But Lambda function replaces few characters with the replacement character and so the attachment getting corrupt.

I checked on a PNG file. sample content: \x89PNG\r\n\u001A\n\u0000\u0000\u0000

All the characters are received as they are but \x89 or in general \x** getting replaced by the replacement character(U+FFFD).

I am extracting the attachment file_str as a string and writing into a file and then uploading it to s3.

File.open(file_path, 'w') do |f|
  f << file_str
end

Thanks in advance.

My guess is the problem is not in writing the file but in the encoding of `file_str`. Whatever is doing the string encoding is forcing it to be valid UTF-8. You need to look earlier in the process — Max, Feb 27 '19 at 20:10
You are correct. I tried the same with rails and I got the error even before reaching the action in the controller. Error: "Encoding::UndefinedConversionError: "\x89" from ASCII-8BIT to UTF-8" — Avinash142857, Feb 28 '19 at 03:06

score 0 · Answer 1 · answered Feb 27 '19 at 11:56

0

You need the binary mode on to work with (to write) binary files.

#                      ⇓ THIS
File.open(file_path, 'wb') do |f|
  f << file_str
end

You try to store the content in UTF-8 and \x89 is not valid UTF-8.

answered Feb 27 '19 at 11:56

Tried this but it's still giving the same corrupt file. – Avinash142857 Feb 27 '19 at 13:27
`file_str` must also have `"BINARY"` encoding. – Aleksei Matiushkin Feb 27 '19 at 13:28
But how can I get BINARY encoded file_str in the first place?. The string I am receiving is already corrupt as Ruby couldn't understand some of the characters(So it replaces them with the replacement character ). – Avinash142857 Feb 28 '19 at 03:06
Should I encode the string before saving it to file? – Avinash142857 Mar 07 '19 at 08:49

1 Answers1