One of the main challenges that developers, SREs, and DevOps professionals face is the absence of an extensive tool that provides them with visibility to their application stack. Many of the APM solutions out on the market do provide methods to monitor applications that were built on languages and frameworks (i.e., .NET, Java, Python, etc.) but fall short when it comes to C++ applications. Luckily, Elastic has been one of the leading solutions in observability space and a contributor to the OpenTelemetry project. Elastic’s unique position and its extensive observability capabilities allows end-users to monitor applications built with object-oriented programming languages & Framework in a variety of ways. In this blog we will explore using Elastic APM to investigate C++ traces with the OpenTelemetry client. We will be providing a comprehensive guide on how to implement the OpenTelemetry client for C++ applications and connecting to Elastic APM solutions. While OTel has its libraries, and this blog reviews how to use the OTel CPP library, Elastic also has its own Elastic Distributions of OpenTelemetry, which were developed to provide commercial support, and are completely upstreamed regularly. Here are some resources to help get you started: - [Use OpenTelemetry with APM](https://www.elastic.co/guide/en/observability/current/apm-open-telemetry.html) - [The OpenTelemetry C++ Client](https://github.com/open-telemetry/opentelemetry-cpp) - [OpenTelemetry C++ Docs](https://opentelemetry.io/docs/languages/cpp/) ## Step by Step Guide ## Prerequisites - ### Environment Choosing an environment is quite important as there is limited support for the OTEL client. We have experimented with using multiple Operating Systems and here are the suggestions: - Ubuntu 22.04 - Debian 11 Bullseye - For this guide we are focusing on Ubuntu 22.04. - Machine: 2 vCPU, 4GB is sufficient. - Image: Ubuntu 22.04 LTS (x86_64). - Disk: \~30 GB is enough. ## Implementation method We have experimented with multiple methods but we found that the most suitable approach is to use a package manager. After extensive testing, It appears that trying to run otel-cpp client could be quite challenging to the users. If practitioners desire to build with tools such as CMake and Bazel that is a viable solution. With that, as we tested both methods it became obvious that we were spending most of our time and effort fixing compatibility and dependencies’ issues for the OS Vs. Focusing on sending data to our APM. Hence we decided to move to a different method. The main issues that we kept running into as we test are: - Compatibility of packages. - Availability of packages. - Dependencies of libraries and packages. In this guide we will use vcpkg since it allows us to bring in all the dependencies required to run the Opentelemetry C++ client. ## Installing required OS tools ### Update package lists ``` sudo apt-get update ``` Install build essentials, cmake, git, and sqlite dev library ``` sudo apt-get install -y build-essential cmake git curl zip unzip sqlite3 libsqlite3-dev ``` sqlite3 and libsqlite3-dev allow us to build/run SQLite queries in our C++ code. ### Set Up vcpkg vcpkg is the C++ package manager that we’ll use to install opentelemetry-cpp client. ``` # Clone vcpkg cd ~ git clone https://github.com/microsoft/vcpkg.git ``` ``` # Bootstrap cd ~/vcpkg ./bootstrap-vcpkg.sh ``` ### Install OpenTelemetry C++ with OTLP gRPC In this guide we focus on trace export to Elastic. At time of writing, vcpkg’s opentelemetry-cpp version 1.18.0 fully supports traces but has limited direct metrics exporting. ### Install the package ``` cd ~/vcpkg ./vcpkg install opentelemetry-cpp[otlp-grpc]:x64-linux ``` **Note** Sometimes when installing opentelemetry-cpp on linux it doesn't install all the required packages. As a workaround if you run into that case, try running again but pass a flag to allow-unsupported: ``` ./vcpkg install opentelemetry-cpp[*]:x64-linux --allow-unsupported ``` ### Verify ``` ./vcpkg list | grep opentelemetry-cpp ``` The output thould be something like this: ``` opentelemetry-cpp:x64-linux 1.18.0 ``` ## Create the C++ Project with Database Spans We’ll build a sample in \~/otel-app that: - Uses SQLite to do basic CREATE/INSERT/SELECT queries. This is helpful to showcase capturing transactions for apps that use databases on Elastic APM. - Generate random traces to showcase how they are captured on Elastic APM. This app is going to generate random queries where some will contain database transactions and some are just application traces. Each query is contained in a child span, so they appear in APM as separate database transactions. # Below is the structure of our project ``` otel-app/ ├── main.cpp └── CMakeLists.txt ``` ### Create App Project ``` cd ~ mkdir otel-app cd otel-app ``` Inside this project we will create two files - main.cpp - CMakeLists.txt Keep in mind that main.cpp is where you are going to pass the otel exporters that are going to send data to the Elastic cluster. So for your tech stack it would be your application's source code. #### Sample application code ``` main.cpp // Below we declare required libraries that we will be using to ship // traces to Elastic APM #include #include #include #include #include #include #include #include #include // for rand(), srand() #include // for time() // Namespace aliases namespace trace_api = opentelemetry::trace; namespace sdktrace = opentelemetry::sdk::trace; namespace otlp = opentelemetry::exporter::otlp; // Below we are using a helper function to run SQLITE statement inside // child span bool ExecuteSql(sqlite3 *db, const std::string &sql, trace_api::Tracer &tracer, const std::string &span_name) { // Starting the child span auto db_span = tracer.StartSpan(span_name); { auto scope = tracer.WithActiveSpan(db_span); // Here we mark Database attributes for clarity in APM db_span->SetAttribute("db.system", "sqlite"); db_span->SetAttribute("db.statement", sql); char *errMsg = nullptr; int rc = sqlite3_exec(db, sql.c_str(), nullptr, nullptr, &errMsg); if (rc != SQLITE_OK) { db_span->AddEvent("SQLite error: " + std::string(errMsg ? errMsg : "unknown")); sqlite3_free(errMsg); db_span->End(); return false; } db_span->AddEvent("Query OK"); } db_span->End(); return true; } /** * DoNonDbWork - Simulate some other operation */ void DoNonDbWork(trace_api::Tracer &tracer, const std::string &span_name) { auto child_span = tracer.StartSpan(span_name); { auto scope = tracer.WithActiveSpan(child_span); // Just sleep or do some "fake" work std::cout << "[TRACE] Doing non-DB work for " << span_name << "...\n"; std::this_thread::sleep_for(std::chrono::milliseconds(200 + rand() % 300)); child_span->AddEvent("Finished non-DB work"); } child_span->End(); } int main() { // Seed random generator for example srand(static_cast(time(nullptr))); // 1) Create OTLP exporter for traces otlp::OtlpGrpcExporterOptions opts; auto exporter = std::make_unique(opts); // 2) Simple Span Processor auto processor = std::make_unique(std::move(exporter)); // 3) Tracer Provider auto sdk_tracer_provider = std::make_shared(std::move(processor)); auto tracer = sdk_tracer_provider->GetTracer("my-cpp-multi-app"); // Prepare an in-memory SQLite DB (for random DB usage) sqlite3 *db = nullptr; int rc = sqlite3_open(":memory:", &db); if (rc == SQLITE_OK) { // Create a table so we can do inserts/reads ExecuteSql(db, "CREATE TABLE IF NOT EXISTS items (id INTEGER PRIMARY KEY, info TEXT);", *tracer.get(), "db_create_table"); } // Create the following loop to generate multiple transactions int num_transactions = 5; // Change this variable to the desired number of transaction for (int i = 1; i <= num_transactions; i++) { // Each iteration is a top-level transaction std::string transaction_name = "transaction_" + std::to_string(i); auto parent_span = tracer->StartSpan(transaction_name); { auto scope = tracer->WithActiveSpan(parent_span); std::cout << "\n=== Starting " << transaction_name << " ===\n"; // Randomly select whether a transaction will interact with the database or not. bool doDb = (rand() % 2 == 0); // 50% chance if (doDb && db) { // Insert random data std::string insert_sql = "INSERT INTO items (info) VALUES ('Item " + std::to_string(i) + "');"; ExecuteSql(db, insert_sql, *tracer.get(), "db_insert_item"); // Select from DB ExecuteSql(db, "SELECT * FROM items;", *tracer.get(), "db_select_items"); } else { // Do some random non-DB tasks DoNonDbWork(*tracer.get(), "non_db_task_1"); DoNonDbWork(*tracer.get(), "non_db_task_2"); } // Sleep a little to simulate transaction time std::this_thread::sleep_for(std::chrono::milliseconds(200)); } parent_span->End(); } // Close DB sqlite3_close(db); // Extra sleep to ensure final flush std::cout << "\n[INFO] Sleeping 5 seconds to allow flush...\n"; std::this_thread::sleep_for(std::chrono::seconds(5)); std::cout << "[INFO] Exiting.\n"; return 0; } ``` ##### What does the code do? We create 5 top-level “transaction_i” spans. For each transaction, we randomly choose to do DB or non-DB work ``` - If DB: Insert a row, then select. Each is a child span. - If non-DB: We do two “fake tasks” (child spans). ``` Once we finish, we close the database connection and wait 5 seconds for data flush. #### Sample instruction file CMakeLists.txt : This file contains instructions describing the source files and targets. ``` cmake_minimum_required(VERSION 3.10) project(OtelApp VERSION 1.0) set(CMAKE_CXX_STANDARD 11) set(CMAKE_CXX_STANDARD_REQUIRED ON) # Here we are pointing to use the vcpkg toolchain set(CMAKE_TOOLCHAIN_FILE "PATH-TO/vcpkg.cmake" CACHE STRING "Vcpkg toolchain file") find_package(opentelemetry-cpp CONFIG REQUIRED) add_executable(otel_app main.cpp) # Below we are linking the OTLP gRPC exporter, trace library, and sqlite3 target_link_libraries(otel_app PRIVATE opentelemetry-cpp::otlp_grpc_exporter opentelemetry-cpp::trace sqlite3 ) ``` #### Declare Environmental Variables Here we are going to export our Elastic Cloud endpoints as environmental variables You can get that information by doing the following: 1. Login into your elastic cloud 2. Go into your deployment 3. On the Left hand side, click on the hamburger menu and scroll down to “Integrations” 4. Go on the search bar inside the integration and type “APM” ![](https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt45627cb3f8bd9eec/6a7f19071967ea7d50330b4a/APM-Search.png) 5. Click on the APM integration 6. Scroll down and click on the OpenTelemetry Option on the far left side ![](https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt5eb12d359510901a/6a7f190afc63ab721064d05a/highlighted.png) 7. You should be able to see values similar to the screenshot below. Once you copy the values to export, click on launch APM. ![](https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt991641419ccdcfb4/6a7f190d3cab1c7d650e4c51/highlighted2.png) As you copy the required values, go ahead and export them. ``` export OTEL_EXPORTER_OTLP_ENDPOINT="APM-ENDPOINT" export OTEL_EXPORTER_OTLP_HEADERS="KEY" export OTEL_RESOURCE_ATTRIBUTES="service.name=my-app,service.version=1.0.0,deployment.environment=dev" ``` Note that the elastic OTEL_EXPORTER_OTLP_HEADERS value usually starts with “Authorization=Bearer” make sure that you convert the upper case “A” in authorization to a lower case “a”. This is due to the fact that the otel header exporter expects a lower case “a” for authorization. ### Build and Run Once we create the two files we then move to building the application. cd ~/otel-app mkdir -p build cd build cmake -DCMAKE_TOOLCHAIN_FILE=~/vcpkg/scripts/buildsystems/vcpkg.cmake \ -DCMAKE_PREFIX_PATH=~/vcpkg/installed/x64-linux/share \ .. make Once make is successful run the the application ./otel-app You should be able to see the script execute with a similar console output ``` Console outcome: === Starting transaction_1 === [TRACE] Doing non-DB work for non_db_task_1... [TRACE] Doing non-DB work for non_db_task_2... === Starting transaction_2 === [TRACE] Doing DB work for doDb_task_1... [TRACE] Doing DB work for doDb_task_2... === Starting transaction_3 === [TRACE] Doing non-DB work for non_db_task_1... [TRACE] Doing non-DB work for non_db_task_2... === Starting transaction_4 === [TRACE] Doing non-DB work for non_db_task_1... [TRACE] Doing non-DB work for non_db_task_2... === Starting transaction_5 === [TRACE] Doing non-DB work for non_db_task_1... [TRACE] Doing non-DB work for non_db_task_2... [INFO] Sleeping 5 seconds to allow flush... [INFO] Exiting. ``` Once the script executes you should be able to observe those traces on Elastic APM similar to the screenshots below. ### Observe in Elastic APM Go to Elastic Cloud, open your deployment, and navigate to Observability > APM. Look for the app name in the service list (as defined by OTEL_RESOURCE_ATTRIBUTES). Inside that service’s Traces tab, you’ll find multiple transactions like “transaction_1”, “transaction_2”, etc. Expanding each transaction shows child spans: ``` - Possibly db_insert_item and db_select_items if random DB path was taken. - Otherwise, non_db_task_1 and non_db_task_2. ``` You can see how some transactions do DB calls, some do not, each with different spans. This variety demonstrates how your real application might produce multiple different “routes” or “operations.” #### Service Map If everything runs correctly, you should be able to view your services and see service maps for your application. ![](https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt2b097f6854507a5b/6a7f1910b6b7346a13e491a0/Service-Map.png) #### Services ![](https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd03422687896996d/6a7f19121967ea32f7330b4e/Services.png) #### My Elastic App ![](https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt674df3a1b282a2af/6a7f191633fa8a5c28202b60/Overview-transactions.png) #### App Transactions ![](https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltdaa6307b2d585f39/6a7f191842a117df4295c2db/Transactions2.png) ![](https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt77f910a5f0aa9405/6a7f191b5967e583f55dd68d/Trace-db.png) #### Dependencies ![](https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltd25a359057e1f0a3/6a7f191d6693f845a8664357/Dependecies.png) #### Logs Navigate to your logs window/Discover to see the incoming application logs ![](https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt9807252308e58d22/6a7f19202f00b234edefef0f/Logs.png) ![](https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltcf99221e3f2250aa/6a7f1923448e4e993f5c0b3e/Logs2.png) #### Patterns Log pattern analysis helps you to find patterns in unstructured log messages and makes it easier to examine your data. ![](https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt99d01a0dc50b84b8/6a7f1926e02fac4be75d6989/patt2.png) ## Final Recap Here is a quick summary of what we did: - Provisioned an Ubuntu 22.04 machine. - Installed build tools for SQLite, dev libs, and vcpkg. - Installed the client for opentelemetry-cpp via vcpkg. - Created a minimal C++ project that executes app traces and captures database operations. - Connected database sqlite3 in CMakeLists.txt. - Exported the Elastic OTLP endpoint & token as environment variables (with a lowercase authorization=Bearer key!). - Ran the application and observed DB interactions and app traces in Elastic APM. - Observed application logs and patterns on Elastic logs and Discover. ## FAQ & Common Issues - Getting “Could not find package configuration file provided by opentelemetry-cpp”? Make sure you pass ``` -DCMAKE_TOOLCHAIN_FILE=... and -DCMAKE_PREFIX_PATH=... ``` to cmake, or embed them in CMakeLists.txt. - Crash: “validate_metadata: INTERNAL:Illegal header key”? Use all-lowercase in ``` OTEL_EXPORTER_OTLP_HEADERS, e.g. authorization=Bearer \. ``` - Missing otlp_grpc_metrics_exporter.h? Your vcpkg version of opentelemetry-cpp (1.18.0) lacks a direct metrics exporter for OTLP. For metrics, either upgrade the library or consider an OpenTelemetry Collector approach. - No data in Elastic APM? Double-check your endpoint URL, Bearer token, firewall rules, or service name in the APM ## Additional Resources: - [Sign up for Elastic Cloud free trial](https://cloud.elastic.co/registration) - [More Elastic OpenTelemetry Topics](https://www.elastic.co/observability-labs/blog/tag/opentelemetry) - [Introducing Elastic Distributions of OpenTelemetry](https://www.elastic.co/observability-labs/blog/elastic-distributions-opentelemetry) - [Introducing Elastic Distribution of OpenTelemetry Collector](https://www.elastic.co/observability-labs/blog/elastic-distribution-opentelemetry-collector) - [Instrumenting your OpenAI- powered Python, Node.js, and Java Applications with EDOT](https://www.elastic.co/observability-labs/blog/elastic-opentelemetry-openai)

Context engineering

Vector database

Search powered applications

Logs

Threat protection

Workflows

Elasticsearch

Kibana (Discover, Dashboards)

Elastic Agent Builder

AutoOps

Piped query language

Jina AI search models

Elastic Cloud Serverless

Elastic Cloud Hosted

Self-managed Elasticsearch

Ecommerce search

Customer support search

Search-driven apps

Log analytics

Infrastructure monitoring

Digital experience monitoring

App performance monitoring

AIOps

LLM observability

Next-gen SIEM

Agentic SOAR

XDR and endpoint security

AI for security

10x your data's value

Cloud providers

Elastic AI Ecosystem

Search AI Partner Program

AV-Comparatives

Forrester Wave™ XDR

Gartner Magic Quadrant Leader

IDC MarketScape

Search

Security

Observability

Get started

Demo gallery

Downloads

Integrations

Docs

Elasticsearch Labs

Elastic Security Labs

Elastic Observability Labs

Blog

Community

Events

Webinars

Discuss

Training

Support

Consulting

Haidar Braimaanie

Monitor your C++ Applications with Elastic APM