Traces Gateway Collector not working with mTLS

[ssurbhi19]

Describe the bug
Traces Load Balancer Collector sends trace data to Traces Gateway Collector. When the Traces Gateway Collector was in strict mode, the Trace Load Balancer Collector was unable to send the trace data.

mTLS with ISITO service mesh is enabled and this is a headless service, and lb collector needs to use a headless service to route the traces based on traceID. With headless service and mTLS STIRCT, traces lb collector is not able to route the events to traces gateway collector.

Steps to reproduce
Create a Traces Load Balancer collector that collects traces from the application, then load balances it to the traces gateway collector based on trace id for sampling the trace. Enable mTLS in STRICT mode in both the places - Traces Load Balancer Collector & Traces Gateway Collector.

What did you expect to see?
Traces should flow from Traces Load Balancer Collector to Traces Gateway Collector.

What did you see instead?
Error in the Traces Load Balancer Collector:

2024-11-19T06:48:57.794Z info internal/retry_sender.go:118 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "traces", "name": "loadbalancing", "error": "rpc error: code = Unavailable desc = upstream connect error or disconnect/reset before headers. reset reason: connection termination", "interval": "28.149479787s"}

What version did you use?
Collector image: "open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib". Tag: "0.111.0"

What config did you use?
exporters:
debug:
verbosity: basic
loadbalancing:
protocol:
otlp:
retry_on_failure:
enabled: true
initial_interval: 5s
max_elapsed_time: 300s
max_interval: 30s
sending_queue:
enabled: true
num_consumers: 10
queue_size: 10000
tls:
insecure: true
resolver:
dns:
hostname: >-
collector-headless.ns.svc.cluster.local
port: '4317'
routing_key: traceID

Environment
MacOS - Sonoma 14.7

Additional context