Challenges Integrating MongoDB with Application Insights via OpenTelemetry

[yhsparrow]

Library name

Azure.Monitor.OpenTelemetry.Exporter

Please describe the feature.

Description:

I am facing significant challenges in configuring MongoDB dependency tracking with Application Insights using the OpenTelemetry approach that Microsoft is now recommending. While OpenTelemetry offers a standardized and vendor-agnostic telemetry collection framework, it currently lacks sufficient support for MongoDB or requires overly complex configurations to enable proper monitoring and observability.

I have followed Microsoft's guidelines to move towards OpenTelemetry for telemetry collection instead of the Application Insights SDK, given the clear messaging that this is the future direction for Application Insights integration. However, there are several issues when trying to track MongoDB dependencies and include them in Live Metrics or Application Insights Dependency Tracking.

Steps to Reproduce:

1. I configured OpenTelemetry with the AzureMonitorTraceExporter and AzureMonitorMetricExporter to send traces and metrics to Application Insights.
2. I added MongoDB diagnostics via DiagnosticsActivityEventSubscriber to capture MongoDB operations using OpenTelemetry.
3. Despite this setup, MongoDB dependency calls were not automatically appearing in Live Metrics or the Dependencies section of Application Insights without manual tracking using TelemetryClient.TrackDependency.
4. I also noticed that other databases (such as SQL Server) are automatically tracked with minimal configuration, whereas NoSQL databases like MongoDB require much more complex and manual instrumentation.

---

Current Workaround:

In order to get MongoDB dependency tracking working, I have reverted to using manual tracking in combination with the Application Insights SDK, specifically using the TelemetryClient.TrackDependency() method in each MongoDB operation. This approach is far from ideal:

• It introduces additional complexity and code overhead.
• It defeats the purpose of using OpenTelemetry, which should theoretically provide a standardized way to capture all telemetry (including dependencies) automatically.
• It makes the OpenTelemetry + Azure Monitor configuration inconsistent when compared to how SQL Server or HTTP dependencies are tracked.

---

Key Issues:

1. Lack of MongoDB Automatic Dependency Tracking:
   OpenTelemetry doesn't offer native support for tracking MongoDB dependencies in a seamless way when using Azure Monitor or Application Insights. This differs from the Application Insights SDK, which can automatically track SQL Server dependencies without any manual intervention.

2. Complexity for Non-SQL Databases:
   Other non-SQL databases (like Redis, Cassandra, or MySQL) face similar issues where dependency tracking isn't straightforward. This limits the flexibility of using OpenTelemetry in more complex environments where different databases are used.

3. Manual Instrumentation is Required:
   The current solution requires developers to manually track MongoDB dependency calls using TelemetryClient.TrackDependency(), which is both labor-intensive and error-prone. It also creates inconsistency in how telemetry is collected between different databases in the same application.

---

Feature Request / Change Proposal:

I would like to request that OpenTelemetry for Azure Monitor includes native support for MongoDB and other NoSQL database dependency tracking, similar to how SQL Server is tracked. Specifically:

1. Automatic MongoDB Instrumentation:
   OpenTelemetry should automatically capture and track MongoDB dependencies, making them appear in Live Metrics and the Application Insights Dependencies section without the need for manual tracking via TelemetryClient.

2. Consistency Across Database Types:
   OpenTelemetry should offer a consistent way to track all types of databases (both SQL and NoSQL) without requiring different approaches. This would simplify configuration and make telemetry collection more predictable for developers.

3. Enhanced Documentation:
   The current documentation for configuring OpenTelemetry with MongoDB is sparse and lacks clear examples. Providing better documentation and examples on how to achieve dependency tracking for MongoDB (and other databases) would significantly improve the developer experience.

---

Impact:

This issue impacts developers who are trying to follow Microsoft's recommended path of moving to OpenTelemetry for telemetry collection while using Application Insights. Without proper support for MongoDB or NoSQL databases, it's difficult to create consistent, automated telemetry pipelines across mixed environments, forcing developers to choose between OpenTelemetry's flexibility and the Application Insights SDK's ease of use.

This gap in functionality makes it challenging for those who rely on both SQL and NoSQL databases in the same application, ultimately requiring manual workarounds or dual configurations (OpenTelemetry and Application Insights SDK), leading to added complexity and potential bugs.

---

Summary:

To streamline the transition to OpenTelemetry and make it a viable alternative to the Application Insights SDK, especially for MongoDB and other NoSQL databases, we request that:

1. Native support for MongoDB and other databases is added in OpenTelemetry for Azure Monitor.
2. Documentation is improved to guide developers on integrating non-SQL databases like MongoDB seamlessly.

This would help create a consistent and efficient telemetry setup for applications with mixed database environments and make the move to OpenTelemetry more practical and developer-friendly.

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @cijothomas @rajkumar-rangaraj @reyang @TimothyMothra.

[TimothyMothra]

Hi @yhsparrow
Sorry you got bounced around on this issue.

Thanks for your feedback on the difficulty migrating. I'm going to share this with our PM.
cc @mattmccleary

You touched on several different issues.
I'm going to try to respond to all the issues you raised, but please let me know if I missed anything.

1. Application Insights SDK vs OpenTelemetry vs Azure Monitor.

First I want to set some context to help explain why you're seeing such different offerings.

Application Insights SDK

This is our classic SDK. My team was responsible for the end-to-end experience of generating telemetry from a large list of services and client libraries.
That list was growing faster than we could scale and we couldn't benefit from a broader community of contributions.

OpenTelemetry

This is fully opensource and has contributions from dozens of service providers.
Much of the base functionality has actually been built into the dotnet runtime.
Telemetry is generated from either an Instrumentation Library or from a native library.
Teams at Microsoft have contributed some Instrumentation Libraries for Microsoft owned technologies, which is why you're seeing the easy configuration for things like HTTP Client or MS SQL Server.

Azure Monitor (Exporter and Distro)

My team ships two libraries from THIS repo.
Our Azure Monitor Exporter plugs into the OpenTelemetry ecosystem and delivers telemetry to your Application Insights resource.
We also have an Azure Monitor Distro that includes some recommended Instrumentation Libraries for ASP.NET Core workflows. This tries to recreate that one-line enablement many users were familiar with in our classic SDK.

2. with regard to MongoDB

My team isn't planning on producing an Instrumentation library for MongoDB.

It seems like MongoDB doesn't officially support OpenTelemetry in their native libraries.
You should reach out to them so they can measure how many of their customers are asking for this.

In your original issue, someone shared a possible workaround (link).
I haven't personally tested this, but this is certainly something you could explore.

3. your current workaround

You can manually create dependency telemetry using OpenTelemetry. This would allow you to remove your dependency on the classic Application Insights SDK. There's an example you can follow here: https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-dotnet-migrate?tabs=aspnetcore#dependencytelemetry

[mattmccleary]

Hi @yhsparrow, thank you for taking the time to reach out and thank you @TimothyMothra for including me. I'm a product manager on the App Insights Team. I'm happy to chat more about your need for MongoDB dependency data collection. If we get enough interest, we could add it to our backlog. Feel free to reach out to me at [email protected].