How to verify request is coming from Twilio on Google Cloud Platform with http4k?

Question

I have a server using Kotlin 1.5, JDK 11, http4k v4.12, and I've got the Twilio Java SDK v8.19, hosted using Google Cloud Run.

I've created a predicate using Twilio's Java SDK RequestValidator.

import com.twilio.security.RequestValidator
import mu.KotlinLogging
import org.http4k.core.Filter
import org.http4k.core.HttpHandler
import org.http4k.core.Method
import org.http4k.core.Response
import org.http4k.core.Status
import org.http4k.core.body.form
import org.http4k.core.queries
import org.http4k.core.then
import org.http4k.core.toParametersMap
import org.http4k.filter.RequestPredicate
import org.http4k.filter.ServerFilters
import org.http4k.lens.Header

private val twilioAuthHeaderLens = Header.optional("X-Twilio-Signature")
/** Twilio's helper [RequestValidator]. */
private val twilioValidator = RequestValidator("my-auth-token")

/**
 * Use the Twilio helper validator, [RequestValidator]
 */
val twilioAuthPredicate: RequestPredicate = { request ->

  when (val requestSignature: String? = twilioAuthHeaderLens(request)) {
    null -> {
      logger.debug { "Request has no Twilio request header valid" }
      false
    }
    else -> {
      val uri: String = request.uri.toString()
      val paramMap: Map<String, String?> = request.form().toMap()

      logger.info { "Validating request with uri: $uri, paramMap: $paramMap, signature: $requestSignature" }
      val isTwilioSignatureValid = twilioValidator.validate(uri, paramMap, requestSignature)
      logger.info { "Request Twilio valid: $isTwilioSignatureValid" }
      isTwilioSignatureValid
    }
  }

}

This works using the example Twilio provide, as demonstrated with this Kotest unit test.

(the test and the example code are mismatched - but OperatorAuth is a class that applies the twilioAuthPredicate, and ApplicationProperties fetches the Twilio auth key from a .env file.)

test("demo https://www.twilio.com/docs/usage/security") {

  val twilioApiKey = "12345"
  val appProps = ApplicationProperties(
    TWILIO_API_AUTH_TOKEN(twilioApiKey, TEST_ENV)
  )

  // system-under-test
  val handler: HttpHandler = OperatorAuth(appProps).then { Response(OK) }

  // construct a GET request: https://mycompany.com/myapp.php?foo=1&bar=2
  val urlProto = "https"
  val urlBase = "mycompany.com"

  val requestSignature = "0/KCTR6DLpKmkAf8muzZqo1nDgQ="

  val request = Request(Method.GET, "$urlProto://$urlBase/myapp.php")

    .query("foo", "1")
    .query("bar", "2")

    .form("CallSid", "CA1234567890ABCDE")
    .form("Caller", "+12349013030")
    .form("Digits", "1234")
    .form("From", "+12349013030")
    .form("To", "+18005551212")

    .header("X-Twilio-Signature", requestSignature)
    .header("X-Forwarded-Proto", urlProto)
    .header("Host", urlBase)

  val response = handler(request)
  response shouldHaveStatus OK
}

However, aside from this simple example, no other requests work, either when creating a unit test, or when live. All Twilio requests fail validation and my server returns 401. The information in the Twilio website is completely opaque. It's incredibly frustrating. It doesn't tell me how it's calculated the hash so I can't tell what is going wrong.

Warning  15003
Message  Got HTTP 401 response to https://my-gcr-server.run.app/twilio

Here is an example test using real values gathered from a log (although I have edited the identifiers).

test("real request") {

  val appProps = ApplicationProperties() // this loads the Twilio Auth Key from my environment variables

  val handler: HttpHandler = OperatorAuth(appProps).then { Response(OK) }

  // construct a GET request
  val urlProto = "https"
  val urlBase = "my-gcr-server.run.app"

  val requestSignature = "GATG2313LSuCYRbPASD4axJ26XyTk="

  val request = Request(Method.GET, "$urlProto://$urlBase/voicemail/transcript")

    .query("ApplicationSid", "AP1234567890abcdefg")
    .query("ApiVersion", "2010-04-01")
    .query("Called", "")
    .query("Caller", "client:Anonymous")
    .query("CallStatus", "ringing")
    .query("CallSid", "CA1234567890abcdefg")
    .query("From", "client:Anonymous")
    .query("To", "")
    .query("Direction", "inbound")
    .query("AccountSid", "AC1234567890abcdefg")
    // note, changing these variables to be form parameters doesn't affect the result, Twilio's validator still says the request is invalid.

    .header("X-Twilio-Signature", requestSignature)
    .header("I-Twilio-Idempotency-Token", "337aaaa-1111-2222-3333-ffffb5333")
    .header("Content-Type", "text/html")
    .header("User-Agent: ", "TwilioProxy/1.1")
    .header("X-Forwarded-Proto", urlProto)
    .header("Host", urlBase)

  val response = handler(request)
  response shouldHaveStatus OK // this fails, Status: expected:<200 OK> but was:<401 Unauthorized>
}

Sometimes validation fails because of Google Cloud. I had previously hosted my server on Google Cloud Functions until I discovered there's an issue where GCF silently omits part of the URI https://github.com/GoogleCloudPlatform/functions-framework-java/issues/90

There is also an issue where if a request is 'modified', for example if I set a Twilio callback URL to include a query param, e.g. https://my-gcr-server.app.run/twilio/callback?type=recording, then the Twilio signature ignores this parameter, but when validating the auth it's impossible to know which parameters Twilio is ignoring. The same is true if the headers are altered.

Is there a working method of validating that a request originates from Twilio? Or an alternative validation solution?

Update

I've just found that Twilio's RequestValidator is really under-tested, there's only one example RequestValidatorTest

score 0 · Answer 1 · answered Sep 21 '21 at 03:01

Twilio developer evangelist here.

The documentation describes how the signature is created and that may show up some differences in the way you are testing. On your server, the algorithm to check the signature is:

Take the full URL of the request URL you specify for your phone number or app, from the protocol (https...) through the end of the query string (everything after the ?).

If the request is a POST, sort all of the POST parameters alphabetically (using Unix-style case-sensitive sorting order).

Iterate through the sorted list of POST parameters, and append the variable name and value (with no delimiters) to the end of the URL string.

Sign the resulting string with HMAC-SHA1 using your AuthToken as the key (remember, your AuthToken's case matters!).

Base64 encode the resulting hash value.

Compare your hash to ours, submitted in the X-Twilio-Signature header. If they match, then you're good to go.

You are using GET requests, so you can discard steps 2 and 3.

There are some things I can see from this algorithm that might cause differences in the way you are testing the validator.

Your error from testing in real life used the URL https://my-gcr-server.run.app/twilio, but your test script from a real request uses https://my-gcr-server.run.app/voicemail/transcript. The URL matters in the generation of the signature.

Your test also adds query parameters to the request, but it's hard to know what the order of those params will be. The order of the query params in the URL should be the exact same as the URL Twilio made the request to.

On the other hand, if the original Twilio request was a POST request, then those parameters should be added as form parameters, as the algorithm takes the form parameters, sorts them and appends them to the URL, with no delimiters.

You said:

There is also an issue where if a request is 'modified', for example if I set a Twilio callback URL to include a query param, e.g. https://my-gcr-server.app.run/twilio/callback?type=recording, then the Twilio signature ignores this parameter, but when validating the auth it's impossible to know which parameters Twilio is ignoring. The same is true if the headers are altered.

This is not true, a query param is part of the URL, as I've said above. Twilio does not ignore parameters, it deals with them according to the algorithm described above. As for headers, aside from the X-Twilio-Signature which is used to test the signature against, they do not come into play.

Having said all that, I am not sure why a real life request would fail the validator as it should be handling all of the things I have discussed above. You can inspect the code used to validate a request and get a signature.

In your code:

      val uri: String = request.uri.toString()
      val paramMap: Map<String, String?> = request.form().toMap()

      logger.info { "Validating request with uri: $uri, paramMap: $paramMap, signature: $requestSignature" }
      val isTwilioSignatureValid = twilioValidator.validate(uri, paramMap, requestSignature)
      logger.info { "Request Twilio valid: $isTwilioSignatureValid" }
      isTwilioSignatureValid

Can you guarantee that the uri is indeed the original URL that Twilio made the request to, not a URL that has been parsed into parts and put back together with the query parameters in a different order? In a GET request, does request.form().toMap() return an empty Map?

Sorry this isn't a full answer, I'm not much of a Java/Kotlin developer. I'm hoping this gives you a good idea of what to look into though.

score 0 · Answer 2 · answered Nov 23 '21 at 06:59

I was having trouble because I was testing with ngrok to route the request to my local server while developing. I had was I was running the algorithm (see above in philnash 's answer as per the Twilio docs )

However while I was setting the callback to the https ngrok enpoint which Twilio used to calculate the signature, the actual request that cam to me was the http endpoint, ngrok forwards the https to http on the free account.

so I was testing a http endpoint but Twilio was calculating against the https endpoint.

when I told Twilio to callback on the http endpoint there was no ngrok misdirection and the signature matched!

Also I notice from the "validate a request: and "get a signature" links above in philnash's answer that the code tried both with a port (eg 443 or 80) in the URL and without and accepts either signature as a match.

How to verify request is coming from Twilio on Google Cloud Platform with http4k?

Update

2 Answers2