I am developing a chat app that has three main components (so far). I shall described the "architecture" of the working local development environment:
- a client in SolidJS running in vite, using
nice-grpc-web(a little TypeScript gRPC-web library) - an Envoy proxy running in a Docker container
- a Python gRPC server that streams responses from Haystack's
PromptNode, running directly on my machine but also tested and happy to run in a Docker container
This is all working fine locally
With a view to moving it into the cloud, I deployed the Python app in a Docker container as a Cloud Run service on GCP.
Now, this service is running happily and is looging that it is happy and is responding to health checks. So I think that is all good.
However, reconfiguring my Envoy proxy to point to this service, rather than my local Python gRPC server, causes the whole thing to break. I get messages from Envoy telling me that the endpoints can't be reached:
ClientError: /ChatBot/AskQuestion UNAVAILABLE: upstream connect error or disconnect/reset before headers. reset reason: connection failure
So it is either the Envoy config that is making it unhappy, or the Cloud Run service that is making it unhappy. I have done lots of reading around both, but can't work it out.
Things that I imagine might be causing it:
- the protocol is different. Now it is running over https, might I need to add more config to Envoy to support this?
- I had to override the port on Cloud Run (standard is 8080, I am using 50051). Although it is responding on this port to the health checks from GCP, is there some further config I need to do?
- perhaps a CORS issue? But I would expect the response to reflect that
- although I have checked to make sure it is available to the internet and not just internally, could this be causing an issue?
The only even vaguely helpful article I have come across is this one, but that doesn't work either.
Here is my envoy.yaml, actual service name elided:
admin:
access_log_path: /tmp/admin_access.log
address:
socket_address: { address: 0.0.0.0, port_value: 9901 }
static_resources:
listeners:
- name: listener_0
address:
socket_address: { address: 0.0.0.0, port_value: 8080 }
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
codec_type: auto
stat_prefix: ingress_http
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains: ["*"]
routes:
- match: { prefix: "/" }
route:
cluster: chat_service
timeout: 0s
max_stream_duration:
grpc_timeout_header_max: 0s
cors:
allow_origin_string_match:
- prefix: "*"
allow_methods: GET, PUT, DELETE, POST, OPTIONS
allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout
max_age: "1728000"
expose_headers: custom-header-1,grpc-status,grpc-message
http_filters:
- name: envoy.filters.http.grpc_web
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.grpc_web.v3.GrpcWeb
- name: envoy.filters.http.cors
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.cors.v3.Cors
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
clusters:
- name: chat_service
type: logical_dns
dns_lookup_family: V4_ONLY
connect_timeout: 20s
http2_protocol_options: {}
lb_policy: round_robin
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
"@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
explicit_http_config:
http2_protocol_options: { }
load_assignment:
cluster_name: cluster_0
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: myapp.a.run.app
port_value: 50051
Dockerfile
FROM envoyproxy/envoy:v1.26.1
COPY envoy.yaml /etc/envoy/envoy.yaml
EXPOSE 9901
EXPOSE 50051
EXPOSE 50052
EXPOSE 8080
ENTRYPOINT [ "/usr/local/bin/envoy" ]
CMD [ "-c /etc/envoy/envoy.yaml", "-l trace", "--log-path /tmp/envoy_info.log" ]
Suggestions please!
