0

I get

All host(s) tried for query failed (tried: 10.244.0.72/10.244.0.72:9042 (com.datastax.driver.core.exceptions.TransportException: [10.244.0.72/10.244.0.72:9042] Channel has been closed))

when trying to access Cassandra within the same namespace. Although when I forward ports it works ok from localhost. keyspace is created successfully.

kubectl port-forward cassandra1-0 9042:9042

My yaml

apiVersion: v1
kind: Service
metadata:
  name: cassandra1
  labels:
    app: cassandra1
spec:
  ports:
  - name: "cql"
    protocol: "TCP"
    port: 9042
    targetPort: 9042
  - name: "thrift"
    protocol: "TCP"
    port: 9160
    targetPort: 9160
  selector:
    app: cassandra1
  type: NodePort
---  
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: cassandra1
  labels:
    app: cassandra1
spec:
  serviceName: cassandra1
  replicas: 1
  selector:
    matchLabels:
      app: cassandra1
  template:
    metadata:
      labels:
        app: cassandra1
    spec:
      terminationGracePeriodSeconds: 1800
      containers:
      - name: cassandra1
        image: gcr.io/google-samples/cassandra:v13
        imagePullPolicy: Always
        ports:
        - containerPort: 7000
          name: intra-node
        - containerPort: 7001
          name: tls-intra-node
        - containerPort: 7199
          name: jmx
        - containerPort: 9042
          name: cql
        - containerPort: 9160
          name: thrift  
        resources:
          limits:
            cpu: "500m"
            memory: 1Gi
          requests:
            cpu: "500m"
            memory: 1Gi
        securityContext:
          capabilities:
            add:
              - IPC_LOCK
        lifecycle:
          preStop:
            exec:
              command: 
              - /bin/sh
              - -c
              - nodetool drain
        env:
          - name: MAX_HEAP_SIZE
            value: 512M
          - name: HEAP_NEWSIZE
            value: 100M
          - name: CASSANDRA_SEEDS
            value: "cassandra1-0.cassandra1.default.svc.cluster.local"
          - name: CASSANDRA_CLUSTER_NAME
            value: "cassandra1"
          - name: CASSANDRA_DC
            value: "DC1-cassandra1"
          - name: CASSANDRA_RACK
            value: "Rack1-cassandra1"
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
        readinessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - /ready-probe.sh
          initialDelaySeconds: 15
          timeoutSeconds: 5
        # These volume mounts are persistent. They are like inline claims,
        # but not exactly because the names need to match exactly one of
        # the stateful pod volumes.
        volumeMounts:
        - name: cassandra1-data
          mountPath: /cassandra1_data
  volumeClaimTemplates:
  - metadata:
      name: cassandra1-data
      namespace: default
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi

Cassandra starts with following properties:

Starting Cassandra on 10.244.0.72
CASSANDRA_CONF_DIR /etc/cassandra
CASSANDRA_CFG /etc/cassandra/cassandra.yaml
CASSANDRA_AUTO_BOOTSTRAP true
CASSANDRA_BROADCAST_ADDRESS 10.244.0.72
CASSANDRA_BROADCAST_RPC_ADDRESS 10.244.0.72
CASSANDRA_CLUSTER_NAME cassandra1
CASSANDRA_COMPACTION_THROUGHPUT_MB_PER_SEC
CASSANDRA_CONCURRENT_COMPACTORS
CASSANDRA_CONCURRENT_READS
CASSANDRA_CONCURRENT_WRITES
CASSANDRA_COUNTER_CACHE_SIZE_IN_MB
CASSANDRA_DC DC1-cassandra1
CASSANDRA_DISK_OPTIMIZATION_STRATEGY ssd
CASSANDRA_ENDPOINT_SNITCH SimpleSnitch
CASSANDRA_GC_WARN_THRESHOLD_IN_MS
CASSANDRA_INTERNODE_COMPRESSION
CASSANDRA_KEY_CACHE_SIZE_IN_MB
CASSANDRA_LISTEN_ADDRESS 10.244.0.72
CASSANDRA_LISTEN_INTERFACE
CASSANDRA_MEMTABLE_ALLOCATION_TYPE
CASSANDRA_MEMTABLE_CLEANUP_THRESHOLD
CASSANDRA_MEMTABLE_FLUSH_WRITERS
CASSANDRA_MIGRATION_WAIT 1
CASSANDRA_NUM_TOKENS 32
CASSANDRA_RACK Rack1-cassandra1
CASSANDRA_RING_DELAY 30000
CASSANDRA_RPC_ADDRESS 0.0.0.0
CASSANDRA_RPC_INTERFACE
CASSANDRA_SEEDS cassandra1-0.cassandra1.default.svc.cluster.local
CASSANDRA_SEED_PROVIDER org.apache.cassandra.locator.SimpleSeedProvider
changed ownership of '/cassandra_data/data' from root to cassandra
changed ownership of '/cassandra_data' from root to cassandra

In my application that runs in the same namespace i tried setting cassandraport to 9042 and host to:

10.240.0.4 (hostIP)
10.244.0.72 (podIP)
cassandra1   (name of the service)
cassandra1.default
cassandra1.default.svc.cluster.local
cassandra1-0.cassandra1.default.svc.cluster.local
_cql._tcp.cassandra1.default.svc.cluster.local

I also tried different types of a service: headless, ClusterIP, NodePort

Does anybody has ANY ideas what is wrong or what else can i try to get this to work?

rigby
  • 1,280
  • 3
  • 13
  • 21
  • hi, which port are you trying to connect ? – Suresh Vishnoi May 02 '19 at 14:34
  • sorry i should have mentioned it, 9042 – rigby May 02 '19 at 14:51
  • Can you try connecting from ouside the cluster using a NodePort service? When you do `kubectl get svc cassandra1` in the column _PORT(S)_ you have the port of the pod (9042 in your case) and the port that is exposed in the node (if you use NodePort), for example "30588". So in the case of NodePort you should use the IP of a node as a host and the port you get with `kubectl get svc`. Besides that, what is the _CLUSTER-IP_ of you service? that is the IP you internal app should be using. Besides using DNS, you may also try using environment variables for service discovery. – victortv May 02 '19 at 15:38

0 Answers0