I'm attempting to configure the open telemetry collector in Kubernetes. I took the jaeger all in one deployment which is here: https://www.jaegertracing.io/docs/1.22/opentelemetry/ and ported it to kubernete running on my minikube.
The problem is I can't seem to get the open telemetry collector to receive the jaeger traces and send it to my proxy container. My jaeger all in one app seems to be working in my minikube instance. Traces are being sent through the hot rap app and I can view the traces in the jaeger UI.
My open telemetry collector looks like the following:
receivers:
jaeger:
protocols:
grpc:
endpoint: 0.0.0.0:14250
thrift_compact:
endpoint: 0.0.0.0:6831
thrift_http:
endpoint: 0.0.0.0:14268
logging:
loglevel: debug
exporters:
zipkin:
endpoint: "http://proxy.collector-agent.svc.cluster.local:80/v1/observations/api/v2/spans"
insecure: true
logging:
loglevel: debug
processors:
batch:
extensions:
health_check:
pprof:
endpoint: :1888
zpages:
endpoint: :55679
service:
extensions: [pprof, zpages, health_check]
pipelines:
traces:
receivers: [jaeger]
processors: [batch]
exporters: [zipkin]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [logging]
It doesn't seem that the open-tel collector is even receiving the jaeger traces. The logs from the container are below..
dev-MacBook-Pro otel-agent % kubectl logs otel-collector-6c4db7687c-h9pm9
2021-03-10T16:53:39.394Z info service/service.go:411 Starting OpenTelemetry Collector... {"Version": "v0.22.0-7-gc8bc12e3", "GitHash": "c8bc12e3", "NumCPU": 2}
2021-03-10T16:53:39.404Z info service/service.go:593 Using memory ballast {"MiBs": 683}
2021-03-10T16:53:39.404Z info service/service.go:255 Setting up own telemetry...
2021-03-10T16:53:39.406Z info service/telemetry.go:102 Serving Prometheus metrics {"address": ":8888", "level": 0, "service.instance.id": "85884852-3e34-4b13-b24e-03d7e9f49868"}
2021-03-10T16:53:39.406Z info service/service.go:292 Loading configuration...
2021-03-10T16:53:39.409Z info service/service.go:303 Applying configuration...
2021-03-10T16:53:39.409Z info service/service.go:324 Starting extensions...
2021-03-10T16:53:39.409Z info builder/extensions_builder.go:53 Extension is starting... {"component_kind": "extension", "component_type": "health_check", "component_name": "health_check"}
2021-03-10T16:53:39.409Z info healthcheckextension/healthcheckextension.go:40 Starting health_check extension {"component_kind": "extension", "component_type": "health_check", "component_name": "health_check", "config": {"TypeVal":"health_check","NameVal":"health_check","Port":13133}}
2021-03-10T16:53:39.410Z info builder/extensions_builder.go:59 Extension started. {"component_kind": "extension", "component_type": "health_check", "component_name": "health_check"}
2021-03-10T16:53:39.410Z info builder/extensions_builder.go:53 Extension is starting... {"component_kind": "extension", "component_type": "zpages", "component_name": "zpages"}
2021-03-10T16:53:39.410Z info zpagesextension/zpagesextension.go:42 Register Host's zPages {"component_kind": "extension", "component_type": "zpages", "component_name": "zpages"}
2021-03-10T16:53:39.413Z info zpagesextension/zpagesextension.go:55 Starting zPages extension {"component_kind": "extension", "component_type": "zpages", "component_name": "zpages", "config": {"TypeVal":"zpages","NameVal":"zpages","Endpoint":"localhost:55679"}}
2021-03-10T16:53:39.413Z info builder/extensions_builder.go:59 Extension started. {"component_kind": "extension", "component_type": "zpages", "component_name": "zpages"}
2021-03-10T16:53:39.414Z info builder/exporters_builder.go:302 Exporter is enabled. {"component_kind": "exporter", "exporter": "zipkin"}
2021-03-10T16:53:39.414Z info service/service.go:339 Starting exporters...
2021-03-10T16:53:39.414Z info builder/exporters_builder.go:92 Exporter is starting... {"component_kind": "exporter", "component_type": "zipkin", "component_name": "zipkin"}
2021-03-10T16:53:39.414Z info builder/exporters_builder.go:97 Exporter started. {"component_kind": "exporter", "component_type": "zipkin", "component_name": "zipkin"}
2021-03-10T16:53:39.414Z info memorylimiter/memorylimiter.go:108 Memory limiter configured {"component_kind": "processor", "component_type": "memory_limiter", "component_name": "memory_limiter", "limit_mib": 1572864000, "spike_limit_mib": 536870912, "check_interval": 5}
2021-03-10T16:53:39.414Z info builder/pipelines_builder.go:203 Pipeline is enabled. {"pipeline_name": "traces/1", "pipeline_datatype": "traces"}
2021-03-10T16:53:39.414Z info service/service.go:352 Starting processors...
2021-03-10T16:53:39.414Z info builder/pipelines_builder.go:51 Pipeline is starting... {"pipeline_name": "traces/1", "pipeline_datatype": "traces"}
2021-03-10T16:53:39.414Z info builder/pipelines_builder.go:61 Pipeline is started. {"pipeline_name": "traces/1", "pipeline_datatype": "traces"}
2021-03-10T16:53:39.414Z info builder/receivers_builder.go:230 Receiver is enabled. {"component_kind": "receiver", "component_type": "jaeger", "component_name": "jaeger", "datatype": "traces"}
2021-03-10T16:53:39.414Z info builder/receivers_builder.go:105 Ignoring receiver as it is not used by any pipeline {"component_kind": "receiver", "component_type": "zipkin", "component_name": "zipkin", "receiver": "zipkin"}
2021-03-10T16:53:39.414Z info service/service.go:364 Starting receivers...
2021-03-10T16:53:39.414Z info builder/receivers_builder.go:70 Receiver is starting... {"component_kind": "receiver", "component_type": "jaeger", "component_name": "jaeger"}
2021-03-10T16:53:39.415Z info static/strategy_store.go:201 No sampling strategies provided or URL is unavailable, using defaults {"component_kind": "receiver", "component_type": "jaeger", "component_name": "jaeger"}
2021-03-10T16:53:39.415Z info builder/receivers_builder.go:75 Receiver started. {"component_kind": "receiver", "component_type": "jaeger", "component_name": "jaeger"}
2021-03-10T16:53:39.415Z info healthcheck/handler.go:128 Health Check state change {"component_kind": "extension", "component_type": "health_check", "component_name": "health_check", "status": "ready"}
2021-03-10T16:53:39.415Z info service/service.go:267 Everything is ready. Begin running and processing data.
Even when I send a ton of jaeger traces nothing ever seems to be received by the collector. Is there a way to debug further or a configuration I'm missing? Any help would be greatly appreciated.
