Can't access Mongo from Node container running in Istio

Question

I'm trying to run a stateful Mongo inside Kubernetes and it works with these configurations outside of Istio.

  mongodb: {
    uri: "mongodb://mongo-0.mongo,mongo-1.mongo,mongo-2.mongo",
    dbName: "app"
  }

But when I run the node application inside Istio, it loses the ability to connect to the mongo. Is there something I'm missing or is it so that I can't use stateful-sets with Istio yet?

stateful mongo config below.

apiVersion: v1
kind: Service
metadata:
  labels:
    service: mongo
  name: mongo
spec:
  ports:
  - name: tcp
    port: 27017
    targetPort: 27017
  clusterIP: None
  selector:
    service: mongo
    role: mongo
***********
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: mongo
spec:
  serviceName: "mongo"
  replicas: 3
  template:
    metadata:
      labels:
        role: mongo
        environment: test
        service: mongo
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: mongo
          image: mongo:3.4.6
          resources:
            requests:
              cpu: "10m"
          command:
            - mongod
            - "--replSet"
            - rs0
            - "--smallfiles"
            - "--noprealloc"
          ports:
            - containerPort: 27017
          volumeMounts:
            - name: mongo-persistent-storage
              mountPath: /data/db
        - name: mongo-sidecar
          image: cvallance/mongo-k8s-sidecar
          resources:
            requests:
              cpu: "10m"
          env:
            - name: MONGO_SIDECAR_POD_LABELS
              value: "role=mongo,environment=test"

Error that I'm receiving is

[2017-07-25 12:01:11] ERROR Mongoose MonboDB connection error: {
  "message": "write EPIPE",
  "name": "MongoError",
  "stack": "Error: write EPIPE\n    at exports._errnoException (util.js:1024:11)\n    at WriteWrap.afterWrite [as oncomplete] (net.js:851:14)"
}
[2017-07-25 12:01:11] ERROR ... retrying createConnection in 5 seconds... 
[2017-07-25 12:01:16] ERROR Mongoose MonboDB connection error: {
  "message": "read ECONNRESET",
  "name": "MongoError",
  "stack": "Error: read ECONNRESET\n    at exports._errnoException (util.js:1024:11)\n    at TCP.onread (net.js:610:25)"
}

score 0 · Accepted Answer · answered Jul 25 '17 at 20:24

0

StatefulSet support has been recently added, but it's not in a release yet.

https://github.com/istio/pilot/pull/896

answered Jul 25 '17 at 20:24

Frank B

900
4
12

Thaks! Didn't notice that :) – jelums Jul 27 '17 at 05:24

nurgasemetey · Answer 2 · 2018-12-16T10:20:39.873

For ones who are receiving such error in istio with manual injection:

When you are injecting proxy to your deployment, network is dropped while istio-proxy is configured:

kubectl apply -f kubernetes-configs/
istioctl kube-inject -f kubernetes-configs/deployment.yml | kubectl -n foo apply -f -

In that time, you will receive:

Error: write EPIPE

I solved problem by adding pause before initialization like in this question: How Can I Wait In Node.js (Javascript), l need to pause for a period of time

UPDATE

Here is the another way. Just let nodejs app restart. By invoking process.exit(137) in any place where connection is not successful, Docker container will be killed and restarted. In that time, istio-proxy will finish network configuration and your app will successfully connect to DB.

Can't access Mongo from Node container running in Istio

2 Answers2