Modify the policy that grants access to the principal that's signing the URL, adding this condition key test.
"Condition": {
"Bool": {
"aws:SecureTransport": "true"
}
}
If requests are made with those signed URLs using HTTP, they'll be denied. This change will be retroactive, impacting even signed URLs you already generated, because it's evaluated when the URL is actually used.
Alternately, create a bucket policy to deny all access to the bucket when the condition is false.
https://aws.amazon.com/blogs/security/how-to-use-bucket-policies-and-apply-defense-in-depth-to-help-secure-your-amazon-s3-data/