Ingress problem with prometheus-community helm chart

Question

I am deploying the prometheus-community helm chart to one of my EKS clusters and I'm having a problem with the ingress settings.

On the cluster I am using Amazon ALB controllers to provision a load balancer in AWS, I do this for many different apps without a problem, but here I am getting a 404 when hitting the ingress host address, despite the fact everything seems to be set up correctly.

Here's my values.yaml file with the ingress and service config: Service (note changed type to NodePort from default ClusterIP):

service:
annotations: {}
labels: {}
clusterIP: ""

externalIPs: []

loadBalancerIP: ""
loadBalancerSourceRanges: []
servicePort: 80
sessionAffinity: None
type: NodePort

Ingress:

ingress:
## If true, Prometheus server Ingress will be created
##
enabled: true

## Prometheus server Ingress annotations
##
annotations: { alb.ingress.kubernetes.io/scheme: "internet-facing",
                        kubernetes.io/ingress.class: "alb",
                alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]',
                alb.ingress.kubernetes.io/ssl-policy: "ELBSecurityPolicy-TLS-1-2-Ext-2018-06",
            alb.ingress.kubernetes.io/certificate-arn: "REDACTED: my-wildcard-arn",
            alb.ingress.kubernetes.io/inbound-cidrs: "REDACTED: my-inbound-cidrs" }
#   kubernetes.io/ingress.class: nginx
#   kubernetes.io/tls-acme: 'true'

## Prometheus server Ingress additional labels
##
extraLabels: {}

## Prometheus server Ingress hostnames with optional path
## Must be provided if Ingress is enabled
##
hosts:
   - prometheus.mydomain.com

## Extra paths to prepend to every host configuration. This is useful when working with annotation based services.
extraPaths: []
# - path: /*
#   backend:
#     serviceName: ssl-redirect
#     servicePort: use-annotation

## Prometheus server Ingress TLS configuration
## Secrets must be manually created in the namespace
##
tls: []
#   - secretName: prometheus-server-tls
#     hosts:
#       - prometheus.domain.com

So as you can see, all I've done there is change the service type to NodePort, and in the ingress settings I've added the host and the required annotations.

On the EKS cluster, the ingress is as follows:

NAME                CLASS    HOSTS                     ADDRESS                                                             PORTS   AGE
prometheus-server   <none>   prometheus.mydomain.com   my-aws-alb-address.region.elb.amazonaws.com   80      11m

The service is:

NAME                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
prometheus-server          NodePort    123.45.678.90   <none>        80:31986/TCP   14m

Navigating to the address gives a 404. The security group for the cluster is propogated with rules to allow access to HTTPS 443 from the Load Balancers. It looks like something is just not bound correctly and I can't see it for looking.

I have a similar problem with AlertManager, but I do get a route to that in browser, only I get a blank white screen. That's probably indicative of a different problem though.

Any ideas what I have missed here. I've looked through the github issues pages and not seen anything relevant.

UPDATE:

As an additional bit of info, here's how my service and ingress are templating in helm:

---
# Source: prometheus/templates/server/service.yaml
apiVersion: v1
kind: Service
metadata:
  labels:
    component: "server"
    app: prometheus
    release: prometheus
    chart: prometheus-12.0.0
    heritage: Tiller
  name: prometheus-server
  namespace: prometheus
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 9090
  selector:
    component: "server"
    app: prometheus
    release: prometheus
  sessionAffinity: None
  type: "NodePort"


---
# Source: prometheus/templates/server/ingress.yaml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/certificate-arn: my-wildcard-arn
    alb.ingress.kubernetes.io/inbound-cidrs: my-inbound-cidrs
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]'
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-Ext-2018-06
    kubernetes.io/ingress.class: alb
    
  labels:
    component: "server"
    app: prometheus
    release: prometheus
    chart: prometheus-12.0.0
    heritage: Tiller
  name: prometheus-server
  namespace: prometheus
spec:
  rules:
    - host: prometheus.mydomain.com
      http:
        paths:

          - path: /
            backend:
              serviceName: prometheus-server
              servicePort: 80