Data Collection Basics of Azure Monitor Application Insights
Article
Before you can monitor your application, it needs to be instrumented.
In the following sections, we cover some data collection basics of Azure Monitor Application Insights.
Instrumentation Options
At a basic level, "instrumenting" is simply enabling an application to capture telemetry.
There are two methods to instrument your application:
Automatic instrumentation (autoinstrumentation)
Manual instrumentation
Autoinstrumentation enables telemetry collection through configuration without touching the application's code. Although it's more convenient, it tends to be less configurable. It's also not available in all languages. See Autoinstrumentation supported environments and languages. When autoinstrumentation is available, it's the easiest way to enable Azure Monitor Application Insights.
Manual instrumentation is coding against the Application Insights or OpenTelemetry API. In the context of a user, it typically refers to installing a language-specific SDK in an application. This means that you have to manage the updates to the latest package version by yourself. You can use this option if you need to make custom dependency calls or API calls that are not captured by default with autoinstrumentation. There are two options for manual instrumentation:
While we see OpenTelemetry as our future direction, we have no plans to stop collecting data from older SDKs. We still have a way to go before our Azure OpenTelemetry Distros reach feature parity with our Application Insights SDKs. In many cases, customers continue to choose to use Application Insights SDKs for quite some time.
Important
"Manual" doesn't mean you'll be required to write complex code to define spans for distributed traces, although it remains an option. Instrumentation Libraries packaged into our Distros enable you to effortlessly capture telemetry signals across common frameworks and libraries. We're actively working to instrument the most popular Azure Service SDKs using OpenTelemetry so these signals are available to customers who use the Azure Monitor OpenTelemetry Distro.
Telemetry Types
Telemetry, the data collected to observe your application, can be broken into three types or "pillars":
Distributed Tracing
Metrics
Logs
A complete observability story includes all three pillars, and Application Insights further breaks down these pillars into tables based on our data model. Our Application Insights SDKs or Azure Monitor OpenTelemetry Distros include everything you need to power Application Performance Monitoring on Azure. The package itself is free to install, and you only pay for the data you ingest in Azure Monitor.
There are two ways to send your data to Azure Monitor (or any vendor):
Via a direct exporter
Via an agent
A direct exporter sends telemetry in-process (from the application's code) directly to the Azure Monitor ingestion endpoint. The main advantage of this approach is onboarding simplicity.
The currently available Application Insights SDKs and Azure Monitor OpenTelemetry Distros rely on a direct exporter.
Note
For Azure Monitor's position on the OpenTelemetry-Collector, see the OpenTelemetry FAQ.
Tip
If you are planning to use OpenTelemetry-Collector for sampling or additional data processing, you may be able to get these same capabilities built-in to Azure Monitor. Customers who have migrated to Workspace-based Application Insights can benefit from Ingestion-time Transformations. To enable, follow the details in the tutorial, skipping the step that shows how to set up a diagnostic setting since with Workspace-centric Application Insights this is already configured. If you’re filtering less than 50% of the overall volume, it’s no additional cost. After 50%, there is a cost but much less than the standard per GB charge.
OpenTelemetry
Microsoft is excited to embrace OpenTelemetry as the future of telemetry instrumentation. You, our customers, asked for vendor-neutral instrumentation, and we're pleased to partner with the OpenTelemetry community to create consistent APIs and SDKs across languages.
Microsoft worked with project stakeholders from two previously popular open-source telemetry projects, OpenCensus and OpenTracing. Together, we helped to create a single project, OpenTelemetry. OpenTelemetry includes contributions from all major cloud and Application Performance Management (APM) vendors and lives within the Cloud Native Computing Foundation (CNCF). Microsoft is a Platinum Member of the CNCF.
For terminology, see the glossary in the OpenTelemetry specifications.
Some legacy terms in Application Insights are confusing because of the industry convergence on OpenTelemetry. The following table highlights these differences. OpenTelemetry terms are replacing Application Insights terms.
The Azure Monitor Exporter uses EventSource for its internal logging. The exporter logs are available to any EventListener by opting in to the source that's named OpenTelemetry-AzureMonitor-Exporter. For troubleshooting steps, see OpenTelemetry Troubleshooting on GitHub.
Step 2: Test connectivity between your application host and the ingestion service
Application Insights SDKs and agents send telemetry to get ingested as REST calls at our ingestion endpoints. To test connectivity from your web server or application host computer to the ingestion service endpoints, use cURL commands or raw REST requests from PowerShell. For more information, see Troubleshoot missing application telemetry in Azure Monitor Application Insights.
Known issues
The following items are known issues for the Azure Monitor OpenTelemetry Exporters:
The operation name is missing from dependency telemetry. The missing operation name causes failures and adversely affects performance tab experience.
The device model is missing from request and dependency telemetry. The missing device model adversely affects device cohort analysis.
Step 1: Enable diagnostic logging
The Azure Monitor Exporter uses EventSource for its internal logging. The exporter logs are available to any EventListener by opting in to the source that's named OpenTelemetry-AzureMonitor-Exporter. For troubleshooting steps, see OpenTelemetry Troubleshooting on GitHub.
Step 2: Test connectivity between your application host and the ingestion service
Application Insights SDKs and agents send telemetry to get ingested as REST calls at our ingestion endpoints. To test connectivity from your web server or application host computer to the ingestion service endpoints, use cURL commands or raw REST requests from PowerShell. For more information, see Troubleshoot missing application telemetry in Azure Monitor Application Insights.
Known issues
The following items are known issues for the Azure Monitor OpenTelemetry Exporters:
The operation name is missing from dependency telemetry. The missing operation name causes failures and adversely affects performance tab experience.
The device model is missing from request and dependency telemetry. The missing device model adversely affects device cohort analysis.
Step 2: Test connectivity between your application host and the ingestion service
Application Insights SDKs and agents send telemetry to get ingested as REST calls at our ingestion endpoints. To test connectivity from your web server or application host computer to the ingestion service endpoints, use cURL commands or raw REST requests from PowerShell. For more information, see Troubleshoot missing application telemetry in Azure Monitor Application Insights.
Known issues
If you download the Application Insights client library for installation from a browser, sometimes the downloaded JAR file is corrupted and is about half the size of the source file. If you experience this problem, download the JAR file by running the curl or wget command, as shown in the following example command calls:
The example command calls apply to Application Insights for Java version 3.4.11. To find the version number and URL address of the current release of Application Insights for Java, see https://github.com/microsoft/ApplicationInsights-Java/releases.
The following steps are applicable to Spring Boot native applications.
Step 1: Verify the OpenTelemetry version
You might notice the following message during the application startup:
WARN c.a.m.a.s.OpenTelemetryVersionCheckRunner - The OpenTelemetry version is not compatible with the spring-cloud-azure-starter-monitor dependency.
The OpenTelemetry version should be <version>
In this case, you have to import the OpenTelemetry Bills of Materials
by following the OpenTelemetry documentation in the Spring Boot starter.
Step 2: Enable self-diagnostics
If something doesn't work as expected, you can enable self-diagnostics at the DEBUG level to get some insights. To do so, set the self-diagnostics level to ERROR, WARN, INFO, DEBUG, or TRACE by using the APPLICATIONINSIGHTS_SELF_DIAGNOSTICS_LEVEL environment variable.
To enable self-diagnostics at the DEBUG level when running a docker container, run the following command:
docker run -e APPLICATIONINSIGHTS_SELF_DIAGNOSTICS_LEVEL=DEBUG <image-name>
Note
Replace <image-name> with the docker image name accordingly.
Third-party information disclaimer
The third-party products that this article discusses are manufactured by companies that are independent of Microsoft. Microsoft makes no warranty, implied or otherwise, about the performance or reliability of these products.
Step 1: Enable diagnostic logging
Azure Monitor Exporter uses the OpenTelemetry API logger for internal logs. To enable the logger, run the following code snippet:
Step 2: Test connectivity between your application host and the ingestion service
Application Insights SDKs and agents send telemetry to get ingested as REST calls at our ingestion endpoints. To test connectivity from your web server or application host computer to the ingestion service endpoints, use cURL commands or raw REST requests from PowerShell. For more information, see Troubleshoot missing application telemetry in Azure Monitor Application Insights.
Known issues
The following items are known issues for the Azure Monitor OpenTelemetry Exporters:
The operation name is missing from dependency telemetry. The missing operation name causes failures and adversely affects performance tab experience.
The device model is missing from request and dependency telemetry. The missing device model adversely affects device cohort analysis.
The database server name is missing from the dependency name. Because the database server name isn't included, OpenTelemetry Exporters incorrectly aggregate tables that have the same name onto different servers.
Step 1: Enable diagnostic logging
The Microsoft Azure Monitor Exporter uses the Python standard logging library for its internal logging. OpenTelemetry API and Azure Monitor Exporter logs are assigned a severity level of WARNING or ERROR for irregular activity. The INFO severity level is used for regular or successful activity.
By default, the Python logging library sets the severity level to WARNING. Therefore, you must change the severity level to see logs under this severity setting. The following example code shows how to output logs of all severity levels to the console and a file:
Step 2: Test connectivity between your application host and the ingestion service
Application Insights SDKs and agents send telemetry to get ingested as REST calls at our ingestion endpoints. To test connectivity from your web server or application host computer to the ingestion service endpoints, use cURL commands or raw REST requests from PowerShell. For more information, see Troubleshoot missing application telemetry in Azure Monitor Application Insights.
Step 3: Avoid duplicate telemetry
Duplicate telemetry is often caused if you create multiple instances of processors or exporters. Make sure that you run only one exporter and processor at a time for each telemetry pillar (logs, metrics, and distributed tracing).
The following sections describe scenarios that can cause duplicate telemetry.
Duplicate trace logs in Azure Functions
If you see a pair of entries for each trace log within Application Insights, you probably enabled the following types of logging instrumentation:
The native logging instrumentation in Azure Functions
The azure-monitor-opentelemetry logging instrumentation within the distribution
To prevent duplication, you can disable the distribution's logging, but leave the native logging instrumentation in Azure Functions enabled. To do this, set the OTEL_LOGS_EXPORTER environment variable to None.
Duplicate telemetry in "Always On" Azure Functions
If the Always On setting in Azure Functions is set to On, Azure Functions keeps some processes running in the background after each run is complete. For instance, suppose that you have a five-minute timer function that calls configure_azure_monitor each time. After 20 minutes, you then might have four metric exporters that are running at the same time. This situation might be the source of your duplicate metrics telemetry.
In this situation, either set the Always On setting to Off, or try manually shutting down the providers between each configure_azure_monitor call. To shut down each provider, run shutdown calls for each current meter, tracer, and logger provider, as shown in the following code:
Azure Workbooks and Jupyter Notebooks might keep exporter processes running in the background. To prevent duplicate telemetry, clear the cache before you make more calls to configure_azure_monitor.
Step 4: Make sure that Flask request data is collected
If you implement a Flask application, you might find that you can't collect Requests table data from Application Insights while you use the Azure Monitor OpenTelemetry Distro client library for Python. This issue could occur if you don't structure your import declarations correctly. You might be importing the flask.Flask web application framework before you call the configure_azure_monitor function to instrument the Flask library. For example, the following code doesn't successfully instrument the Flask app:
from azure.monitor.opentelemetry import configure_azure_monitor
from flask import Flask
configure_azure_monitor()
app = Flask(__name__)
Instead, we recommend that you import the flask module as a whole, and then call configure_azure_monitor to configure OpenTelemetry to use Azure Monitor before you access flask.Flask:
from azure.monitor.opentelemetry import configure_azure_monitor
import flask
configure_azure_monitor()
app = flask.Flask(__name__)
Alternatively, you can call configure_azure_monitor before you import flask.Flask:
from azure.monitor.opentelemetry import configure_azure_monitor
configure_azure_monitor()
from flask import Flask
app = Flask(__name__)
Support
Select a tab for the language of your choice to discover support options.
Coming soon: Throughout 2024 we will be phasing out GitHub Issues as the feedback mechanism for content and replacing it with a new feedback system. For more information see: https://aka.ms/ContentUserFeedback.