banner-1183445_1920.jpg

banner-1183445_1280.jpg

Changes to Elastic Machine Learning Anomaly Scoring in 6.5

outliers.jpg

temporal-distance-917364_640.jpg

Temporal vs. Population Analysis in Elastic Machine Learning

pexels-photo-712786.jpeg

adult-blur-bokeh-712786.jpg

Filtering Input Data to Refine Machine Learning Jobs

Recently I had a question that came up asking how to manage a derivative aggregation in a Machine Learning job. Custom aggregations are supported as part of the Machine Learning job's data feed config, and <a href="https://www.elastic.co/guide/en/x-pack/current/ml-configuring-aggregation.html">there is documentation</a> on how to do this. There are a few specific rules that should be followed so let's go through an example from start to finish so that you can see how it is done.
For this particular example, we'll use the following data set.&nbsp;
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt814a8cedaadd92d3/5e346ace0aff5474f7624f90/blog-custom-elasticsearch-aggregations-01.png">
 
Under normal circumstances, just analyzing the data using ML's <code>low_sum</code> function is good enough to detect the anomaly of the "brown-out" on the last day of the data viewed above. But, for the sake of argument, let's imagine that what we're really interested is in the rate of change (the derivative) of the data set. In other words, the "recovery" coming out of the brown-out is a sharp, positive, rate of change that should have a high derivative at that moment. So, using the derivative aggregation in an Elasticsearch query, we can calculate the instantaneous rate of change and then use ML to see if the derivative is extreme at any point in time.&nbsp;
<h2>Defining The Job</h2>Let's create a job config that will do this for us. I'll use the <a href="https://www.elastic.co/guide/en/x-pack/current/ml-api-quickref.html">ML API</a> (invoked via the dev Console in Kibana) to do the creation:
<pre>PUT _xpack/ml/anomaly_detectors/orders_deriv
{
 "description": "Derivative of Order Volume",
 "analysis_config": {
 "bucket_span": "5m",
 "detectors": [
 {
 "detector_description": "sum(orders_deriv)",
 "function": "sum",
 "field_name": "orders_deriv"
 }
 ],
 "influencers": [],
 "summary_count_field_name": "doc_count"
 },
 "model_plot_config": {
 "enabled": "true"
 },
 "data_description": {
 "time_field": "@timestamp"
 }
}
</pre>Notice the usage of <code>summary_count_field_name</code>, this is required when using aggregated queries in Machine Learning. Also note that the <code>field_name</code> of "orders_deriv" in the detector doesn’t exist in the raw data, it will instead come from Elasticsearch's derivative aggregation in the datafeed’s query, outlined below.
<h3>Defining the Datafeed</h3>Let's create the datafeed which defines the&nbsp;source of the data to feed to the ML job:
<pre>PUT _xpack/ml/datafeeds/datafeed-orders_deriv/
{
 "job_id": "orders_deriv",
 "indices": [
 "it_ops_kpi-2017"
 ],
 "types": [
 "doc"
 ],
 "aggregations": {
 "buckets": {
 "date_histogram": {
 "field": "@timestamp",
 "interval": "5m",
 "time_zone": "UTC"
 },
 "aggregations": {
 "@timestamp": {
 "max": {
 "field": "@timestamp"
 }
 },
 "orders": {
 "sum": {
 "field": "events_per_min"
 }
 },
 "orders_deriv": {
 "derivative": {
 "buckets_path": "orders"
 }
 }
 }
 }
 }
}
</pre>
	As shown in the&nbsp;<a href="https://www.elastic.co/guide/en/x-pack/current/ml-configuring-aggregation.html">documentation</a>, there first must be a&nbsp;buckets aggregation, which in turn must contain a date histogram aggregation. This requirement ensures that the aggregated data is a time series. After that, however, we do a sum aggregation (to mimic what we were doing before with ML's <code>low_sum</code> function) and then a derivative aggregation on that summed value. When all is said and done, Elasticsearch is creating and returning a "new" field called "orders_deriv" to the ML job. 
One can test the datafeed's query and see if the outcome is as expected - just use the <code>_preview</code> endpoint:
<pre>GET _xpack/ml/datafeeds/datafeed-orders_deriv/_preview
</pre>The return may look something like:
<pre>[
 {
 "@timestamp": 1485643149000,
 "doc_count": 4
 },
 {
 "@timestamp": 1485643449000,
 "orders_deriv": 66,
 "doc_count": 5
 },
 {
 "@timestamp": 1485643749000,
 "orders_deriv": 20,
 "doc_count": 5
 },
 {
 "@timestamp": 1485644049000,
 "orders_deriv": -3,
 "doc_count": 5
 },
 {
 "@timestamp": 1485644349000,
 "orders_deriv": -42,
 "doc_count": 5
 },
...
</pre> 
<h3>The Outcome</h3>The result of running the job (either via the UI or the API) indeed shows the large, positive, rate of change at 11:10AM:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt12d926435b5ffefd/5e346ace7e155302ca0c8f9e/blog-custom-elasticsearch-aggregations-02.png"> 
 
We can also "overlay" this job with a straight-up analysis of the raw data and see them together and confirm that the high derivative is when the orders sharply recover after the brown-out:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt0a4b697d1c40c2c2/5e346ace7ca70442647a6221/blog-custom-elasticsearch-aggregations-03.png"> 
However, notice in this case, the little thumbnail graph on the left (for the derivative analysis) does not show the plot of the derivative over time like the Single Metric View does. Why is that, you might ask?
Well, it is because in the Single Metric View,&nbsp; the chart drawn shows the user the "actual" values from the ML model if <code>model_plot_config</code> is enabled. If <code>model_plot_config</code> is not enabled, then the "actual" values will come from a query to ES for the raw data that is executed when the view is loaded. As such, if <code>model_plot_config</code> is not enabled, then the Single Metric Viewer will look the same as the thumbnail chart in the Anomaly Explorer:
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt68d1a92c0e16a341/5e346acde147ae4537d946d1/blog-custom-elasticsearch-aggregations-04.png">
This is currently a limitation of the UI - it is unable to, on-the-fly, reverse engineer the complex Elasticsearch query that was used as part of the datafeed in order to query the raw data to draw the line. We hope to be able to support this in the future though.
You can find more about X-Pack and Machine Learning <a href="https://www.elastic.co/products/x-pack/">here</a>. If you are already using Machine Learning then make sure you check out our <a href="https://www.elastic.co/products/x-pack/machine-learning/recipes">prebuilt recipes</a> to help bootstrap your efforts.

custom-aggregations-elasticsearch-fullbleed.jpg

pea-1205673_640.jpg

Custom Elasticsearch Aggregations for Machine Learning Jobs

Machine Learning Anomaly Scoring and Elasticsearch - How it Works

ML_detail.png

ML_anomalies.png

Alerting on Machine Learning Jobs in Elasticsearch v5.5

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

headshot-rich-collier-300x300.jpg

Solutions Architect, Machine Learning Specialist

Rich Collier

Надсилаючи цю форму ви погоджуєтеся з <a href="/legal/terms-of-use">Умовами надання послуг Elastic</a>. Ваші особисті дані будуть опрацьовані відповідно з <a href="/legal/privacy-statement">Положеннями про конфіденційність Elastic</a>.

Newsletter GDPR Text - Ukrainian

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Intentionally Blank

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

提交即表示您確認已詳閱並同意我們的<a href="/legal/elastic-cloud-account-terms" target="_blank">服務條款</a>，且同意 Elastic 使用您在上方提供的個人資料<a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">與您聯絡</a>，為您提供相關產品和服務的資訊。如需詳細說明或隨時想停止收到產品和服務資訊，請參閱 <a href="/legal/privacy-statement/" target="_blank">Elastic 的隱私聲明</a>。

Newsletter GDPR Text - TW

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.

GDPR Text + Google Cloud

Inviando la registrazione (1) esprimi il consenso ai <a href="/legal/terms-of-use">termini e condizioni di Elastic</a> e <a href="/legal/privacy-statement">all’informativa sulla privacy</a>, nonché (2) a ricevere saltuarie e-mail.

Newsletter GDPR Text - Italian

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Load more

Overview

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Author

Articles by Rich Collier

Solutions Architect, Machine Learning Specialist, Elastic

Changes to Elastic Machine Learning Anomaly Scoring in 6.5

Temporal vs. Population Analysis in Elastic Machine Learning

Filtering Input Data to Refine Machine Learning Jobs

Custom Elasticsearch Aggregations for Machine Learning Jobs

Machine Learning Anomaly Scoring and Elasticsearch - How it Works

Alerting on Machine Learning Jobs in Elasticsearch v5.5

Follow us

About us

Join us

Partners

Trust & Security

Investor relations

Excellence Awards