Add support for specifying extra manifests via Helm

[oponomarov-tu]

Please describe your use case / problem.

It would be very beneficial to supply extraManifests value to the opentelemetry-collector Helm chart that would render extra manifests as a part of a single Helm chart deployment.

Describe the solution you'd like

An extra variable to hold a list of arbitrary Kubernetes manifests, e.g.

## Extra manifests to deploy as an array
extraManifests: []
  # - apiVersion: external-secrets.io/v1beta1
  #   kind: ExternalSecret
  #   metadata:
  #     name: "otel-external-secret"
  #   spec:
  #     secretStoreRef:
  #       name: opentelemetry-secret-store
  #       kind: SecretStore
  #     refreshInterval: "5m0s"
  #     target:
  #       name: opentelemetry-secrets
  #       creationPolicy: 'Owner'
  #     dataFrom:
  #       - extract:
  #           key: opentelemetry-secrets

Describe alternatives you've considered

1. Building a Helm chart wrapper to include opentelemetry-collector Helm chart as a subchart as suggested here.
2. Using Kustomize with overlays.
3. Deploying the extra manifests in a separate Helm chart / Kustomize overlay.

The (1) & (2) are hard to maintain and scale, the (3) introduces the dependency between deployments which requires putting the stuff in correct order.

Additional context

This has been a normal practise and a lot of OSS Helm charts support such options these days, e.g.

• https://github.com/prometheus-community/helm-charts/blob/ff10663d1c9fe327afccbf70698649621761079e/charts/kube-prometheus-stack/values.yaml#L4727
• https://github.com/external-secrets/external-secrets/blob/22c1af40e09d20b1bcedf2be28cb7022ba67a702/deploy/charts/external-secrets/values.yaml#L129

In a previous closed issue (#765), maintainers expressed reluctance about adding this feature, but without providing a clear explanation. The proposed alternatives are merely workarounds that require significant ongoing maintenance. Allowing end users of the Helm chart to specify custom resources would enable the collector to be deployed as a fully self-contained, independent unit. Moreover, this change would not introduce any breaking changes but would instead improve adoption and usability of the existing community-provided Helm chart.

I'm willing to implement this feature.

[crutonjohn]

i could see this being useful for sure, especially in light of #1417. if it is required to add RBAC in the event that you want to customize the processor, should a mechanism be provided to do so within the context of the chart?

on the other hand i can also see a scenario in which having this feature would have the potential to make troubleshooting more difficult. i imagine a scenario where someone is arbitrarily applying something that they may/may not be able to or willing to share, but it is that very thing that is breaking their deployment in some weird way.

[TylerHelmuth]

if it is required to add RBAC in the event that you want to customize the processor, should a mechanism be provided to do so within the context of the chart?

For the presets specifically if you further extend them, like by enabling k8s.pod.memory_limit_utilization metric on the kubeletstats receiver on top of enabling the preset, you can extend the clusterrole via .Values.clusterRole.

[TylerHelmuth]

@oponomarov-tu we do typically follow the patterns set by the prometheus chart. I am ok accepting this idea. @open-telemetry/helm-approvers please review.