3

My API is receiving a call that contains windows-1252 encoded data in the POST body. In rails 6.0.3, this apparently worked. But rails 6.1.1 throws ActionController::BadRequest (Invalid request parameters: Invalid encoding for parameter.

Edit: I can not change the calls to UTF-8, those come from a 3rd party. So, my API must accept windows-1252 encoding.

Here is a snippet that emulates this call:

require 'net/http'
require 'uri'

uri = URI.parse("http://localhost:3030/dummy/create")
request = Net::HTTP::Post.new(uri)
request.content_type = "application/x-www-form-urlencoded"

# Rack::QueryParser::InvalidParameterError: Invalid encoding for parameter:
#G�bor
name = "Gábor".encode("windows-1252", "utf-8")
request.body = "last_name=#{name}&first_name=Sam&charset=windows-1252"


response = Net::HTTP.start(uri.hostname, uri.port, nil) do |http|
  http.request(request)
end

# 400
pp response.code

And on the server side, the dummy#create action is never reached, it stops with

Started POST "/dummy/create" for 127.0.0.1 at 2021-02-06 17:14:12 +0100
  
ActionController::BadRequest (Invalid request parameters: Invalid encoding for parameter: G�bor):
  
actionpack (6.1.1) lib/action_dispatch/request/utils.rb:39:in `check_param_encoding'
actionpack (6.1.1) lib/action_dispatch/request/utils.rb:34:in `block in check_param_encoding'
actionpack (6.1.1) lib/action_dispatch/request/utils.rb:34:in `each_value'
actionpack (6.1.1) lib/action_dispatch/request/utils.rb:34:in `check_param_encoding'
actionpack (6.1.1) lib/action_dispatch/http/request.rb:403:in `block in POST'
rack (2.2.3) lib/rack/request.rb:69:in `fetch'
rack (2.2.3) lib/rack/request.rb:69:in `fetch_header'
actionpack (6.1.1) lib/action_dispatch/http/request.rb:398:in `POST'
actionpack (6.1.1) lib/action_dispatch/http/parameters.rb:55:in `parameters'
...

I found that I could register a middleware to intercept this kind of call, as described in Rails/Rack: "ArgumentError: invalid %-encoding" for POST data

The middleware would then check for the 'charset' param in the POST, and this probably would work. However, this has the disadvantage that it would apply to all calls, not only the dummy#create endpoint.

Since this behavior was recently introduced, (in rail 6.1 I assume), is there a better way now to handle this?

Kjell
  • 492
  • 1
  • 7
  • 15
  • I have this same issue. What did you end up doing to solve the problem? – Eric Terry May 18 '21 at 17:35
  • 1
    Still unsolved. The middleware solution did not pass tests, so I did not yet deploy. Since then I did not retry to find a solution. It is blocking upgrade to rails 6.1 – Kjell May 19 '21 at 18:08
  • The bug has nothing to do with the `charset` parameter. I just finished the test setup and reproduced your bug, but it's almost morning here, will play with it tomorrow at night. Im interested in solving it. – Eugene Petrov Jul 15 '21 at 00:54
  • @Kjell did you end up solving this? Trying to bypass check_param_encoding for incoming emails from SendGrid Parse API which gets handled by Griddler's email_processing.rb file but struggling to override the POST method. – Paul Arterburn Jun 02 '23 at 13:11
  • @PaulArterburn The accepted answer was the solution. In my first comment to it, I misunderstood how it works, therefore the comment might be misleading. – Kjell Jun 03 '23 at 14:51

1 Answers1

1

The solution here, adding param_encoding for each parameter like document https://api.rubyonrails.org/classes/ActionController/ParameterEncoding/ClassMethods.html

param_encoding :create, :last_name, Encoding::ASCII_8BIT
param_encoding :create, :first_name, Encoding::ASCII_8BIT
Bình Trương
  • 380
  • 1
  • 13
  • The input simulates real world request that are made to my API. They come from a different company, and I can not change their requests. – Kjell Jul 11 '21 at 09:48
  • Sorry but your request always has `charset` type params right ? – Bình Trương Jul 11 '21 at 10:09
  • I don't have a guarantee, but yes, all requests in the logfiles have that charset param, and it is always windows-1252 . In case the param is not there, or it is not windows-1252, I think it would be fine to fallback to rails default behavior. – Kjell Jul 11 '21 at 10:48
  • I try one solution but it maybe have security problem,First I disable check params encoding on rack, and in action on controller I get last_name by `last_name = params[:last_name].encode('utf-8', 'windows-1252')`. It return `Gábor` – Bình Trương Jul 11 '21 at 11:13
  • Disabling check_param_encoding lets the G?bor value pass. I looked further in that direction and found https://github.com/rails/rails/commit/0cbc6114dc052faf66eab6a844964b93d1e7a4f6 – Kjell Jul 11 '21 at 12:29
  • @Kjell Do you resolve this problem, I come from upwork and see your post :D, it seem interesting problem. – Bình Trương Jul 11 '21 at 13:17
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/234745/discussion-between-kjell-and-binh-trng). – Kjell Jul 11 '21 at 15:33