Why efficiency matters — for you and the environment

Data centers are estimated to consume ~3% of global electricity consumption, and their usage is expected to double by 2030.* The cost of a digital service is a close proxy to its computing efficiency, and thus, being more efficient is a win-win: less energy consumed, smaller bill.

In the same scenario, companies want the ability to scale to more users while spending less for each user and are effectively looking into methods of reducing their energy consumption.

In this spirit, Universal Profiling comes equipped with data and visualizations to help determine where efficiency improvement efforts are worth the most.

Energy efficiency measures how much a digital service consumes to produce an output given an input. It can be measured in multiple ways, and we at Elastic Observability chose CO₂ emissions and annualized CO₂ emissions (more details on them later).

Let’s take the example of an e-commerce website: the energy efficiency of the “search inventory” process could be calculated as the average CPU time needed to serve a user request. Once the baseline for this value is determined, changes to the software delivering the search process may result in more or less CPU time consumed for the same feature, resulting in less or more efficient code.

How to set up and configure wattage and CO2

You can find a “Settings” button in the top-right corner of the Universal Profiling views. From there, you can customize the coefficient used to calculate CO₂ emissions tied to profiling data.

The values set here will be used only when the profiles gathered from host agents are not already associated with publicly known data certified by cloud providers. For example, suppose you have a hybrid cloud deployment with a portion of your workload running on-premise and a portion running in GCP. In that case, the values set here will only be used to calculate the CO₂ emissions for the on-premise machines; we already use all the coefficients as declared by GCP to calculate the emissions of those machines.

Python vs. Go

Our first blog post implemented a solution to read PGN chess games, a text representation in Python. It showed how Universal Profiler can be leveraged to identify slow functions and help you rewrite your code faster and more efficiently. At the end of it, we were happy with the Python version. It is still used today to grab the monthly updates from the Lichess database and ingest them into Elasticsearch®. I always wanted a reason to work more with Go, and we rewrote Python to Go. We leveraged goroutines and channels to send data through message passing. You can see more about it in our GitHub repository.

Rewriting in Go also means switching from an interpreted language to a compiled one. As with everything in IT, this has benefits as well as disadvantages. One disadvantage is that we must ship debug symbols for the compiled binary. When we build the binary, we can use the symbtool program to ship the debug symbols. Without debug symbols, we see uninterpretable information as frames will be labeled with hexadecimal addresses in the flame graph rather than source code annotations.

First, make sure that your executable includes debug symbols. Go per default builds with debug symbols. You can check this by using file yourbinary. The important part is that it is not stripped.

file lichess
lichess: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, Go BuildID=gufIkqA61WnCh8haeW-2/lfn3ne3U_y8MGoFD4AvT/QJEykzbacbYEmEQpXH6U/MqVbk-402n1k3B8yPB6I, with debug_info, not stripped

Now we need to push the symbols using symbtool. You must create an Elasticsearch API key as the authentication method. In the Universal Profiler UI in Kibana®, an Add Data button in the top right corner will tell you exactly what to do. The command is like this. The -e is the part where you pass through the path of your executable file. In our case, this is lichess as above.

symbtool push-symbols executable -t "ApiKey" -u "elasticsearch-url" -e "lichess"

Now that debug symbols are available inside the cluster, we can run both implementations with the same file simultaneously and see what Universal Profiler can tell us about it.

Identifying CO2 and energy efficiency savings

Python is more frequently scheduled on the CPU. Thus, it runs more often on the hardware and contributes more to the machines’ resource usage.

We use the differential flame graph to identify and automatically calculate the difference in the following comparison. You need to filter on process.thread.name: “python3.11” in the baseline, and for the comparison, filter for lichess.

Looking at the impact of annualized CO₂ emissions, we see a decrease from 65.32kg of CO₂ from the Python solution to 16.78kg. That is a difference of 48.54kg CO₂ savings over a year.

If we take a step back, we’ll want to figure out why Python produces many more emissions. In the flamegraph view, we filter down to just showing Python, and we can click on the first frame called python3.11. A little popup tells us that it caused 32.95kg of emissions. That is nearly 50% of all emissions caused by the runtime. Our program itself caused the other ~32kg of CO₂. We immediately reduced 32kg of annual emissions by cutting out the Python interpreter with Go.

We can lock that box using a right click and click Show more information.

The Show more information link displays detailed information about the frame, like sample count, total CPU, core seconds, and dollar costs. We won’t go into more detail in this blog.

Reduce your carbon footprint today with Universal Profiling

This blog post demonstrates that rewriting your code base can reduce your carbon footprint immensely. Using Universal Profiler, you could do a quick PoC to showcase how much carbon resources can be spared.

Learn how you can get started with Elastic Universal Profiling today.

• Cluster for storing the data where three nodes, each 64GB RAM and 32 CPU cores, are running GCP on Elastic Cloud.
• The machine for sending the data is a GCP e2-standard-32, thus 128GB RAM and 32 CPU cores with a 500GB balanced disk to read the games from.
• The file used for the games is this Lichess database containing 96,909,211 games. The extracted file size is 211GB.

Source:

*https://media.ccc.de/v/camp2023-57070-energy_consumption_of_data_centers

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

First, we need an example application that allows us to monitor infrastructure changes effectively. We use a Python Flask application with the Elastic Python APM agent. The application is a simple calculator taking the numbers as a REST request. We utilize Locust — a simple load-testing tool to evaluate performance under varying workloads.

The next step includes obtaining the pricing information associated with the cloud services. Every cloud provider is different. Most of them offer an option to retrieve pricing through an API. But today, we will focus on Google Cloud and will leverage their pricing calculator to retrieve relevant cost information.

The calculator and Google Cloud pricing

To perform a cost analysis, we need to know the cost of the machines in use. Google provides a billing API and Client Library to fetch the necessary data programmatically. In this blog, we are not covering the API approach. Instead, the Google Cloud Pricing Calculator is enough. Select the machine type and region in the calculator and set the count 1 instance. It will then report the total estimated cost for this machine. Doing this for an e2-standard-4 machine type results in 107.7071784 US$ for a runtime of 730 hours.

Now, let’s go to our Kibana® where we will create a new index inside Dev Tools. Since we don’t want to analyze text, we will tell Elasticsearch® to treat every text as a keyword. The index name is cloud-billing. I might want to do the same for Azure and AWS, then I can append it to the same index.

PUT cloud-billing
{
  "mappings": {
    "dynamic_templates": [
      {
        "stringsaskeywords": {
          "match": "*",
          "match_mapping_type": "string",
          "mapping": {
            "type": "keyword"
          }
        }
      }
    ]
  }
}

Next up is crafting our billing document:

POST cloud-billing/_doc/e2-standard-4_europe-west4
{
  "machine": {
    "enrichment": "e2-standard-4_europe-west4"
  },
  "cloud": {
    "machine": {
       "type": "e2-standard-4"
    },
    "region": "europe-west4",
    "provider": "google"
  },
  "stats": {
    "cpu": 4,
    "memory": 8
  },
  "price": {
    "minute": 0.002459068,
    "hour": 0.14754408,
    "month": 107.7071784
  }
}

We create a document and set a custom ID. This ID matches the instance name and the region since the machines' costs may differ in each region. Automatic IDs could be problematic because I might want to update what a machine costs regularly. I could use a timestamped index for that and only ever use the latest document matching. But this way, I can update and don’t have to worry about it. I calculated the price down to minute and hour prices as well. The most important thing is the machine.enrichment field, which is the same as the ID. The same instance type can exist in multiple regions, but our enrichment processor is limited to match or range. We create a matching name that can explicitly match as in e2-standard-4_europe-west4. It’s up to you to decide whether you want the cloud provider in there and make it google_e2-standard-4_europ-west-4.

Calculating the cost

There are multiple ways of achieving this in the Elastic Stack. In this case, we will use an enrich policy, ingest pipeline, and transform.

The enrich policy is rather easy to setup:

PUT _enrich/policy/cloud-billing
{
  "match": {
    "indices": "cloud-billing",
    "match_field": "machine.enrichment",
    "enrich_fields": ["price.minute", "price.hour", "price.month"]
  }
}

POST _enrich/policy/cloud-billing/_execute

Don’t forget to run the _execute at the end of it. This is necessary to make the internal indices used by the enrichment in the ingest pipeline. The ingest pipeline is rather minimalistic — it calls the enrichment and renames a field. This is where our machine.enrichment field comes in. One caveat around enrichment is that when you add new documents to the cloud-billing index, you need to rerun the _execute statement. The last bit calculates the total cost with the count of unique machines seen.

PUT _ingest/pipeline/cloud-billing
{
  "processors": [
    {
      "set": {
        "field": "_temp.machine_type",
        "value": "{{cloud.machine.type}}_{{cloud.region}}"
      }
    },
    {
      "enrich": {
        "policy_name": "cloud-billing",
        "field": "_temp.machine_type",
        "target_field": "enrichment"
      }
    },
    {
      "rename": {
        "field": "enrichment.price",
        "target_field": "price"
      }
    },
    {
      "remove": {
        "field": [
          "_temp",
          "enrichment"
        ]
      }
    },
    {
      "script": {
        "source": "ctx.total_price=ctx.count_machines*ctx.price.hour"
      }
    }
  ]
}

Since this is all configured now, we are ready for our Transform. For this, we need a data view that matches the APM data_streams. This is traces-apm*, metrics-apm.*, logs-apm.*. For the Transform, go to the Transform UI in Kibana and configure it in the following way:

We are doing an hourly breakdown, therefore, I get a document per service, per hour, per machine type. The interesting bit is the aggregations. I want to see the average CPU usage and the 75,95,99 percentile, to view the CPU usage on an hourly basis. Allowing me to identify the CPU usage across an hour. At the bottom, give the transform a name and select an index cloud-costs and select the cloud-billing ingest pipeline.

Here is the entire transform as a JSON document:

PUT _transform/cloud-billing
{
  "source": {
    "index": [
      "traces-apm*",
      "metrics-apm.*",
      "logs-apm.*"
    ],
    "query": {
      "bool": {
        "filter": [
          {
            "bool": {
              "should": [
                {
                  "exists": {
                    "field": "cloud.provider"
                  }
                }
              ],
              "minimum_should_match": 1
            }
          }
        ]
      }
    }
  },
  "pivot": {
    "group_by": {
      "@timestamp": {
        "date_histogram": {
          "field": "@timestamp",
          "calendar_interval": "1h"
        }
      },
      "cloud.provider": {
        "terms": {
          "field": "cloud.provider"
        }
      },
      "cloud.region": {
        "terms": {
          "field": "cloud.region"
        }
      },
      "cloud.machine.type": {
        "terms": {
          "field": "cloud.machine.type"
        }
      },
      "service.name": {
        "terms": {
          "field": "service.name"
        }
      }
    },
    "aggregations": {
      "avg_cpu": {
        "avg": {
          "field": "system.cpu.total.norm.pct"
        }
      },
      "percentiles_cpu": {
        "percentiles": {
          "field": "system.cpu.total.norm.pct",
          "percents": [
            75,
            95,
            99
          ]
        }
      },
      "avg_transaction_duration": {
        "avg": {
          "field": "transaction.duration.us"
        }
      },
      "percentiles_transaction_duration": {
        "percentiles": {
          "field": "transaction.duration.us",
          "percents": [
            75,
            95,
            99
          ]
        }
      },
      "count_machines": {
        "cardinality": {
          "field": "cloud.instance.id"
        }
      }
    }
  },
  "dest": {
    "index": "cloud-costs",
    "pipeline": "cloud-costs"
  },
  "sync": {
    "time": {
      "delay": "120s",
      "field": "@timestamp"
    }
  },
  "settings": {
    "max_page_search_size": 1000
  }
}

Once the transform is created and running, we need a Kibana Data View for the index: cloud-costs. For the transaction, use the custom formatter inside Kibana and set its format to “Duration” in “microseconds.”

With that, everything is arranged and ready to go.

Observing infrastructure changes

Below I created a dashboard that allows us to identify:

• How much costs a certain service creates
• CPU usage
• Memory usage
• Transaction duration
• Identify cost-saving potential

From left to right, we want to focus on the very first chart. We have the bars representing the CPU as average in green and 95th percentile in blue on top. It goes from 0 to 100% and is normalized, meaning that even with 8 CPU cores, it will still read 100% usage and not 800%. The line graph represents the transaction duration, the average being in red, and the 95th percentile in purple. Last, we have the orange area at the bottom, which is the average memory usage on that host.

We immediately realize that our calculator does not need a lot of memory. Hovering over the graph reveals 2.89% memory usage. The e2-standard-8 machine that we are using has 32 GB of memory. We occasionally spike to 100% CPU in the 95th percentile. When this happens, we see that the average transaction duration spikes to 2.5 milliseconds. However, every hour this machine costs us a rounded 30 cents. Using this information, we can now downsize to a better fit. The average CPU usage is around 11-13%, and the 95th percentile is not that far away.

Because we are using 8 CPUs, one could now say that 12.5% represents a full core, but that is just an assumption on a piece of paper. Nonetheless, we know there is a lot of headroom, and we can downscale quite a bit. In this case, I decided to go to 2 CPUs and 2 GB of RAM, known as e2-highcpu2. This should fit my calculator application better. We barely touched the RAM, 2.89% out of 32GB are roughly 1GB of use. After the change and reboot of the calculator machine, I started the same Locust test to identify my CPU usage and, more importantly, if my transactions get slower, and if so, by how much. Ultimately, I want to decide whether 1 millisecond more latency is worth 10 more cents per hour. I added the change as an annotation in Lens.

After letting it run for a bit, we can now identify the smaller hosts' impact. In this case, we can see that the average did not change. However, the 95th percentile — as in 95% of all transactions are below this value — did spike up. Again, it looks bad at first, but checking in, it went from ~1.5 milliseconds to ~2.10 milliseconds, a ~0.6 millisecond increase. Now, you can decide whether that 0.6 millisecond increase is worth paying ~180$ more per month or if the current latency is good enough.

Conclusion

Observability is more than just collecting logs, metrics, and traces. Linking user experience to cloud costs allows your business to identify areas where you can save money. Having the right tools at your disposal will help you generate those insights quickly. Making informed decisions about how to optimize your cloud cost and ultimately improve the user experience is the bottom-line goal.

The dashboard and data view can be found in my GitHub repository. You can download the .ndjson file and import it using the Saved Objects inside Stack Management in Kibana.

Caveats

Pricing is only for base machines without any disk information, static public IP addresses, and any other additional cost, such as licenses for operating systems. Furthermore, it excludes spot pricing, discounts, or free credits. Additionally, data transfer costs between services are also not included. We only calculate it based on the minute rate of the service running — we are not checking billing intervals from Google Cloud. In our case, we would bill per minute, regardless of what Google Cloud has. Using the count for unique instance.ids work as intended. However, if a machine is only running for one minute, we calculate it based on the hourly rate. So, a machine running for one minute, will cost the same as running for 50 minutes — at least how we calculate it. The transform uses calendar hour intervals; therefore, it's 8 am-9 am, 9 am-10 am, and so on.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.