Strange requests keep coming to my gateway API on AWS

Question

I have a simple HTTP service built with AWS lambda and API gateway. The Domain pointing to the gateway is hosted by Route53 and the gateway uses certificate from Certificate Manager. It's absolutely fresh with sole "service is running" text on index page. Pretty standard setup. All neatly connected using Terraform, works like a charm... except strange requests coming to the API every few seconds (!). I checked every possible probing, healthchecks available on AWS – everything is disabled, but the requests keep coming.

Dump of the request from the lambda:

{
    ...
    "headers": {
        "accept": "*/*", "accept-encoding": "gzip, deflate", "content-length": "0",
        "host": "sub-b.sub-a.my-domain.com", 
        "user-agent": "python-requests/2.26.0", 
        "x-amzn-trace-id": "Root=1-6158e0d5-0d266e5d0a84add227005a79", 
        "x-forwarded-for": "3.85.226.144", "x-forwarded-port": "443", 
        "x-forwarded-proto": "https"
    }, 
    ...
    "http": {
        "method": "GET", "path": "/", "protocol": "HTTP/1.1",
        "sourceIp": "3.85.226.144", "userAgent": "python-requests/2.26.0"
    }, 
    "requestId": "GmgRbgATFiAEMtQ=",
    "routeKey": "$default", 
    "stage": "$default", 
    "time": "02/Oct/2021:22:44:37 +0000"
    ...
}

The 3.85.226.144 IP is EC2 instance in North Virgina, so it's us-east-1 AWS region (mine is eu-central-1). And that agent python-requests/2.26.0. It looks like an AWS service. Especially that I have experimented with changing subdomain which does not alter the final result as the request start appearing instantly after setting up my service again with different subdomain. The volume is about 3000 requests per 1h.

I have already spent hours googling and checking anything that came to my mind, but even tough I work quite often with infrastructure I am literally helpless.

What the hell is responsible for those requests?!

All sorts of ideas and suggestions highly appreciated.

EDIT:

I conducted a test with a fresh new root domain to make sure it's something from AWS. The queries stopped (obviously) during outage but came back immediately (like seconds) after service was reachable again. terraform apply started at 9:52 and completed at 9:59. New domain was attached a little bit earlier than 9:59 (some irrelevant modifications were pending for few more minutes) and so requests started coming. As far as I can imagine there is no way anything could detect a new root domain so fast, so it must be related to AWS itself.

Answer 1

OK, so it turned out to be an issue related to a Route 53 route or Certificate Manger.

When I was applying my terraform-defined resources some kind of race condition must have been occurring and caused some kind of initialization probing to last forever instead of stop immediately after resources creation. So my conclusion is that when API gateway, DNS route and certificate are being created concurrently Route53 or CM tends to stay in some kind of weird state. Creating all services with terraform, then deleting route and then recreating solved the issue.

Answer 2

May I ask why do you think that's generated by AWS service itself? I don't know much about AWS, but the moment I see "python-requests/2.26.0" as UA, I immediately imagine someone has written a script to leverage/probe API for some reasons. What is actually a reply to these requests? Can any business information be parsed from it, like prices/rates etc.? If yes, then you might probably just have someone leasing a server in AWS and writing scripts to parse your data. Or else, they might be probing all possible hosts for all possible domains. Happens on the Internet)

Also, it looks like you're not the first person asking about or reporting that IP address: https://www.abuseipdb.com/check/3.85.226.144 https://github.com/photoprism/photoprism/issues/1458