logstash_nagios_banner.png

logstash_nagios_thumb.png

Integrating Nagios Checks with Logstash

In this integration, we explore NHL Play-by-Play data in Elasticsearch & Kibana, then dive into creating a custom WMS tile server for a hockey rink backdrop.

<table><tbody><tr><td>Note:&nbsp;For a more detailed and flexible approach&nbsp;using geo-referenced vectors, the reader should follow the steps outlined <a href="https://www.elastic.co/blog/operational-analytics-with-elasticsearch-at-elasticon-2017-part-2">in this blog post</a>.&nbsp;</td></tr></tbody></table>The National Hockey League (NHL) provides public access to each game’s play-by-play data, in convenient JSON format (see <a href="https://assets.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt59f1e2e912ab70b1/5f516abcb553152466d1c1c0/PlayByPlay.json">sample doc</a>* for 1 game), which we can ingest into Elasticsearch with minimal effort. &nbsp;In this blog post, we will explore approximately 73 thousand plays made (so far) in the 2016-2017 NHL Season.Each play contains the following information: &nbsp;<ul><li dir="ltr">Time of Play</li><li dir="ltr">Type of Play (Goal, Shot, Hit, Penalty, Fight)</li><li dir="ltr">Who made the play</li><li dir="ltr">Who else was involved in the play</li><li dir="ltr">Who was on the ice at the time of the play</li><li dir="ltr">X,Y coordinates of play location on the ice.</li><li dir="ltr">A text description of the play.</li></ul>Using Kibana dashboard with some basic visualizations, we can drill down on a specific game. &nbsp;&nbsp;Let’s see a timeline of plays, colour coded by type and split by team, then some pie charts of each play, also split by team. &nbsp;&nbsp;The first pie shows us that only the Ducks scored in this game. &nbsp;Gogliano and Silfverberg made this a 2-0 win over the Stars.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt15712bc88f5738df/5f8861f11529bb3239a1f1bd/kibana-nhl-1.png"/> A hockey rink is 200 Feet by 85 Feet, which conveniently equates closely to longitude and latitude (180 by 90). &nbsp;&nbsp;In the case that your data does not equate naturally, you could use Logstash to translate the coordinates to a compatible longitude and latitude range. &nbsp;The following visualization shows the location of all goals scored, but something’s not right - Latitude and Longitude is a nice coordinate hack, but we need the correct backdrop.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt6b48cc6ffe994e1e/5f8861f12f59ae27f3d60e40/kibana-nhl-2.png" style="max-width: 100%;"/> As of <a href="https://www.elastic.co/blog/kibana_4.2_beta2">Kibana 4.2, we added support for custom map tiles</a>, so we can turn our world into a hockey rink (for those of us who haven’t already). Sidney Crosby’s Shot Locations<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltebfafefc8f3d23cb/5f8861f14671ee30c689e83a/kibana-nhl-3.png" style="max-width: 100%;"/> P.K. Subban’s Shot Locations<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltb6807bdb86d1eb91/5f8861f1f6c586323f8b34d4/kibana-nhl-4.png" style="max-width: 100%;"/> <h2 dir="ltr" style="line-height: 1.38;margin-top: 10pt;margin-bottom: 0pt;">Setting up a custom Tile Server</h2>Let’s now outline the process used to load this custom map into the Kibana visualization. &nbsp;&nbsp; Step 1) Create a GeoTIFF version of the Arena image <pre class="prettyprint">gdal_translate -of GTiff -a_srs EPSG:4326 -a_ullr -105 45 105 -45 nhl_rink.gif nhl_rink_gtiff.tiff gdal_warp -t_srs EPSG:4326 nhl_rink_gtiff.tiff nhl_rink.tiff </pre>The <a href="http://www.gdal.org/gdal_translate.html">gdal_translate</a> and <a href="http://www.gdal.org/gdalwarp.html">gdalwarp</a>&nbsp;commands are used in conjunction to encode longitude/latitude points into the image, and to “reproject”, or set control points. Step 2) Set up GeoServer <a href="http://geoserver.org/">GeoServer</a> is a free and open-source <a href="https://en.wikipedia.org/wiki/Web_Map_Service">WMS</a> server, which makes it compatible with Kibana. &nbsp;&nbsp;&nbsp;&nbsp;Downloading and installing this software onto your favourite platform is easy, and results in a Web Front end that you can log into to manage your tile service. &nbsp;&nbsp;Once the server is running, you can log into via http://&lt;hostname&gt;:8080/geoserver.&nbsp; &nbsp;&nbsp;In the screenshots, we are working with GeoServer 2.7.2. Step 3) Create a new Workspace From the Data-&gt;Workspaces menu, click “Add new Workspace”. &nbsp;&nbsp;In this example, we name it “NHL”, and also set the Namespace URI to “NHL” &nbsp;&nbsp;&nbsp; Step 4) Import the GeoTIFF into Data Stores. From the Data-&gt;Stores menu, click “Add new Store”. &nbsp;&nbsp;&nbsp;Select GeoTIFF. &nbsp;&nbsp;Select the Workspace you just created, and set a datasource name. &nbsp;For this example, we use the DS name “nhl_rink_ds”. &nbsp;&nbsp;&nbsp;Browse to select your nhl_rink.tiff file, and press Save. <img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blte2afbd188247962c/5f8861ee271ecf4a1a644e09/kibana-nhl-5.png" style="max-width: 100%;"/> Step 5) Create the WMS Layer From the Data-&gt;Layers menu, click “Add new Resource”. &nbsp;&nbsp;&nbsp;Select your newly created data source (NHL:nhl_rink_ds). &nbsp;&nbsp;By default, the Layer will be named the same as the store image name, without the file extension. (nhl_rink). &nbsp;Click Publish (which then opens up the Layer Settings) then Click Save.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt233dbd7b50c72fb1/5f8861f03db5122648cbccfb/kibana-nhl-6.png" style="max-width: 100%;"/> That’s it, your tile service is ready to go! Step 6) Configure Kibana to use the new tile server Finally, create a tile map visualization in Kibana and configure a custom WMS compliant map server.<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt1e10ee39fdbd4513/5f8861ef1f5f6d4173b501b4/kibana-nhl-7.png" alt="Screen Shot 2017-01-11 at 1.37.32 PM.png" style="max-width: 100%;"/><table style="font-size: 14px;"><tbody><tr style="height: 0px;"><td style="border-color: rgb(0, 0, 0);padding: 7px;">WMS url</td><td style="border-color: rgb(0, 0, 0);padding: 7px;">http://&lt;hostname&gt;:8080/geoserver/NHL/wms</td></tr><tr style="height: 0px;"><td style="border-color: rgb(0, 0, 0);padding: 7px;">WMS layers</td><td style="border-color: rgb(0, 0, 0);padding: 7px;">nhl_rink</td></tr><tr style="height: 0px;"><td style="border-color: rgb(0, 0, 0);padding: 7px;">WMS version</td><td style="border-color: rgb(0, 0, 0);padding: 7px;">1.1.0</td></tr><tr style="height: 0px;"><td style="border-color: rgb(0, 0, 0);padding: 7px;">WMS format</td><td style="border-color: rgb(0, 0, 0);padding: 7px;">image/png</td></tr><tr style="height: 0px;"><td style="border-color: rgb(0, 0, 0);padding: 7px;">WMS attribution</td><td style="border-color: rgb(0, 0, 0);padding: 7px;">My NHL Data</td></tr><tr style="height: 0px;"><td style="border-color: rgb(0, 0, 0);padding: 7px;">WMS styles</td><td style="border-color: rgb(0, 0, 0);padding: 7px;">raster&nbsp; &nbsp;&nbsp;</td></tr></tbody></table><h2 dir="ltr" style="line-height: 1.38;margin-top: 10pt;margin-bottom: 0pt;">Exploring Further</h2>Consider the possibilities for drawing conclusions about the data set: <ul><li>Which goalies are vulnerable to shots from certain players shooting from certain locations?</li><li>Who is drawing the most penalties (Aggregating by p2name.raw where type.raw:penalty)?</li><li>Time Series Analysis in&nbsp;<a href="https://www.elastic.co/blog/timelion-timeline">Timelion</a> to show relationships between stats</li></ul>Try Importing the data into Elasticsearch yourself: &nbsp;<a href="https://github.com/elastic/examples/tree/master/Exploring%20Public%20Datasets/nhl">Find the source code with instructions on GitHub.</a>As always, we appreciate your feedback, so feel free to ping us on <a href="https://discuss.elastic.co/">Discuss</a>&nbsp;or&nbsp;<a href="https://twitter.com/elastic">Twitter</a>. &nbsp;Or, if you want to kick it up a notch and meet us in person, please come to our big user conference Elastic{ON} in March 2017! <hr/>* - The game data was previously available from&nbsp;http://live.nhl.com/GameData/20142015/2014021136/PlayByPlay.json

Screen Shot 2017-01-18 at 10.22.50 AM.png

Screen Shot 2017-01-18 at 10.24.17 AM.png

Kibana and a Custom Tile Server for NHL Data

<h2>Theory</h2>Network infrastructure has evolved in recent years. &nbsp;Systems reside in a combination of the corporate office, datacentre co-locations, and the cloud. &nbsp;Trends in automation and configuration management have simplified massive scaling.A single security domain interconnected by private lines is no longer a luxury that engineers can expect. &nbsp; Ad-hoc measures must be implemented such as virtual private networks and least privilege firewall permissions, on a case-by-case basis, requiring extensive architecture and maintenance. &nbsp;System logging and monitoring are essential components in stable and intelligent business operations - but too often, security is complicated - or worse, compromised. &nbsp; A smart engineer will usually choose the simplest approach, as it is often the most stable and secure.In most of today’s networks, logging and monitoring are mutually exclusive, one pushing back to a central location, and the other polling from a central location. &nbsp; The polling piece of this dual-architecture has several problems:<ul><li>Polling requires either individual firewall rules, or unrestricted access from an operations network. &nbsp;These rules must be manually maintained in each target datacentre, or cloud domain.</li><li>Each polled system and each monitored service must be configured and maintained, apart from existing automation or configuration management systems.</li></ul>Theoretically, it is now possible to simplify the traditional architecture, streamlining logging, monitoring, performance statistics and business reporting into a single platform - Elasticsearch.<h2>Proof of Concept</h2><h3>Beats</h3>In order for a solution to be viable, it must ultimately address any need that arises, on any platform. &nbsp;The <a href="https://www.elastic.co/products/beats">Beats</a> platform is a lightweight, open source data shipper, and runs on most <a href="https://www.elastic.co/support/matrix">common operating systems</a>. &nbsp;The community has already begun <a href="https://www.elastic.co/guide/en/beats/libbeat/master/community-beats.html">contributing Beats</a> to monitor their favourite services.A wealth of Nagios Checks already exist (currently over 5000 in the <a href="https://exchange.nagios.org/">official community directory</a>), and can be integrated with Beats via <a href="https://github.com/PhaedrusTheGreek/nagioscheckbeat">Nagioscheckbeat</a>. &nbsp; Here is an example of how we would configure nagioscheckbeat to check the redis service every 10 seconds:<pre class="prettyprint lang-js">period: "10s" name: "redis" cmd: "/usr/lib/nagios/plugins/check_redis.pl" args: "-H 127.0.0.1 -R -m -T -f -A" </pre><h3 dir="ltr">Elasticsearch</h3>Beats must have an output, such as Logstash or an Elasticsearch cluster. &nbsp; Self-hosted, or <a href="https://www.elastic.co/found">as a cloud service</a>, the organization’s cluster must be secure and&nbsp;accessible by all end systems. &nbsp;<h3 dir="ltr">Kibana</h3>Elasticsearch and <a href="https://www.elastic.co/products/kibana">Kibana</a> serve nicely for reporting performance metrics:<img src="https://api.contentstack.io/v2/assets/575e4d1b9e7a83165490d882/download?uid=blt58eb7126c65867ca?uid=blt58eb7126c65867ca" data-sys-asset-uid="blt05db4a5c70078c4d" alt="Screen Shot 2016-01-13 at 4.53.17 PM.png"/>Apache Workers fit perfectly into a percentage visualization:<img src="https://api.contentstack.io/v2/assets/575e4d1a3dc542cb38c09a9e/download?uid=blt4af5089d36b924c1?uid=blt4af5089d36b924c1" data-sys-asset-uid="blt7083bf3b7b7e3685" alt="Screen Shot 2016-01-13 at 4.25.54 PM.png"/>Kibana’s new <a href="https://www.elastic.co/blog/timelion-timeline">Timelion</a> plugin allows for more detailed time series analysis. &nbsp;For example, we could overlay arbitrary metrics in a single visualization, for custom correlations during an RCA. <img src="https://api.contentstack.io/v2/assets/575e4d1a3dc542cb38c09ab2/download?uid=bltc92cd4a616c54d37?uid=bltc92cd4a616c54d37" data-sys-asset-uid="blta9d6974b8601a6ca" alt="Screen Shot 2016-01-13 at 4.21.00 PM.png"/><h3></h3><h3>Alerting on Crossed Thresholds or Unresponsive Services</h3>Nagioscheckbeat does two things. &nbsp;It publishes performance metrics — for making pretty graphs — but also reports the status of each check against warning &amp; critical thresholds defined in the configuration. &nbsp;Those checks are published separately from the metrics, so we can watch the results directly. &nbsp; Elastic’s <a href="https://www.elastic.co/products/watcher">Watcher</a> plugin can be configured to alert with little effort, and even supports <a href="https://www.pagerduty.com/">PagerDuty</a> notifications.This Watcher configuration will send an email when any nagioscheck has reported a&nbsp;CRITICAL status&nbsp;in the last 30 minutes:<pre class="prettyprint lang-js">PUT _watcher/watch/critical_watch { "trigger" : { "schedule" : { "interval" : "1m" } }, "input" : { "search" : { "request" : { "indices" : [ "nagioscheckbeat*" ], "body" : { "query":{ "filtered":{ "query" : { "bool" : { "must" : [ { "term" : {"_type": "nagioscheck"} }, { "range" : {"@timestamp" : {"gte" : "now-30m"}} }, { "term" : {"status" : "CRITICAL" } } ] } } } } } } } }, "condition" : { "compare" : { "ctx.payload.hits.total" : { "gt" : 0 }} }, "actions" : { "send_email" : { "throttle_period": "30m", "email" : { "to" : "me@elastic.co", "subject" : "Alert from Watcher - Service(s) Critical", "body" : "One or more services reported a CRITICAL state within the last 30 minutes. See the attached file for details. You will not receive another notification for 30 minutes.", "attach_data" : true } } } } </pre>Notice the throttle_period directive, which ensures that alerts are only sent periodically until the condition is resolved. &nbsp;&nbsp; <h3></h3><h3></h3><h3>Alerting on Lost Heartbeats</h3>No monitoring system would be complete without knowing if a host is down. &nbsp;Consider this experimental approach: &nbsp;On each host, a special “heartbeat” is published every several seconds.<pre class="prettyprint lang-js">name: "heartbeat" cmd: "/usr/lib64/nagios/plugins/check_dummy" args: "0 Hello" period: "10s" </pre>The watch works like this: Any host that sent a heartbeat in the last 1 day, and whom we have not heard from in the last 5 minutes is considered “down”. &nbsp;The watch uses an Elasticsearch <a href="https://www.elastic.co/blog/top-hits-aggregation">Top Hits</a> aggregation to show us the most recent heartbeat from each host.&nbsp;<pre class="prettyprint lang-js">... "condition" : { "script" : { "file" : "monitor-hosts", "params" : { "threshold" : 0 } } } ... </pre>(<a href="https://github.com/PhaedrusTheGreek/nagioscheckbeat/blob/master/blog/watch-heartbeat.sense">see the entire watch here</a>)Notice that our condition in the watch is a script, which looks like this:<pre class="prettyprint lang-js">def minutes = 5 def now = DateTime.now().getMillis() ctx.vars.hosts = [ up: [], down: [] ] ctx.payload.aggregations.hosts.buckets.each { def last_heartbeat = it.latest_heartbeat.hits.hits[0].sort[0]; def ms_ago = now - last_heartbeat if (ms_ago &gt; (minutes * 1000) ){ ctx.vars.hosts.down.add( [ hostname: it.key, last_heartbeat: new Date(last_heartbeat) ]) } } return ctx.vars.hosts.down.size() &gt; 0 </pre> <h3>A Complete Open Source solution</h3>As an alternative to Watcher, Integrating Elasticsearch with any existing monitoring system would be trivial - one could simply set up a check that alerts and escalates under similar conditions. &nbsp;A monitoring system communicating with a single Elasticsearch cluster is more secure and scalable than reaching out to every service on each end system.Also, in Kibana, we can easily report on services that are running in a CRITICAL threshold:<img src="https://api.contentstack.io/v2/assets/575e4d1a0342dfd7382649a7/download?uid=blte5b6b2a544d84fb0?uid=blte5b6b2a544d84fb0" data-sys-asset-uid="blt698a8385e33751a1" alt="Screen Shot 2016-01-15 at 1.17.38 PM.png"/><h2>Conclusion</h2>When various Beats are used in combination, we have a unified solution for exporting system logs, application data, and performance metrics. &nbsp;&nbsp;In the changing landscape of infrastructure operations, networks are relying on cloud services such as <a href="https://www.elastic.co/found">Found</a> and Software as a Service such as <a href="https://www.pagerduty.com/">PagerDuty</a>. &nbsp;Leveraging these services is trending because it simplifies architecture and operations.Modern automation technologies have simplified the host-configuration process. &nbsp;Cohesively, when a host’s monitoring configuration is applied via automation, that configuration is spun up and torn down, autoscaling with each host.Please see the entire <a href="https://github.com/PhaedrusTheGreek/nagioscheckbeat/tree/master/blog">Proof of Concept on Github</a>, and feel free to clone and run it yourself.<div> </div>

full-bleed.png

Image-For-Blog.png

A case for self-monitoring systems

<html><head></head><body>A load balancer often serves as a gateway into your production network. In this position, it has a unique perspective of seeing all traffic, while also being aware of the mapping between web services and servers. A great deal of high level information can be learned from the load balancer with minimal effort, and it is independent of web server, application, and platform. This flow of information is highly consistent, and serves well for baselines. Naturally, we can log from this point (as an alternative to processing web server logs) to gain visibility in web traffic patterns and application behavior. 
In this blog, we will take a look at how to pair the Elastic Stack&#xA0;with an F5 <a href="https://f5.com/products/modules/local-traffic-manager">Local Traffic Manager</a>, which is great at quickly balancing load across massive amounts of traffic. This is not unlike Elasticsearch &amp; Kibana that gives us &#x2018;wire speed&#x2019; access to our data for analysis, allowing us to answer important questions in near real time.
Vertical Bar Chart: &#x201C;Am I seeing an increase in HTTP 4xx errors?&#x201D;
<img src="https://api.contentstack.io/v2/assets/575e4cca9e7a83165490cb47/download?uid=blt7946f3224082a3ef" data-sys-asset-uid="blt7946f3224082a3ef" alt="Bar Chart.png">
Tile Map: &#x201C;Where is my Traffic Coming from?&#x201D;
<img src="https://api.contentstack.io/v2/assets/575e4cca9e7a83165490cb4f/download?uid=bltca5a0cdb7e5ceacf" data-sys-asset-uid="bltca5a0cdb7e5ceacf" alt="Geo-Heatmap.png">
Line Chart: &#x201C;What are the response times of my Applications?&#x201D;&#xA0;
<img src="https://api.contentstack.io/v2/assets/575e4ccc3dc542cb38c08d28/download?uid=blt61389db170fd10b3" data-sys-asset-uid="blt61389db170fd10b3" alt="Response Times.png">
From there you can drill down on these answers by time period, or other HTTP metric , and even compare them side-by-side with other services.
This information is of course useful - and the F5 and Elastic architectures mix particularly well making it quick and easy to get to these answers. The F5 distributes logging traffic across a pool of Logstash Servers, conveniently including information about the Virtual Service.

<h2>HSL Pools for Logstash</h2><img src="https://api.contentstack.io/v2/assets/575e4cca58208ba076e28abb/download?uid=blt5cb94b9da5d28242" data-sys-asset-uid="blt5cb94b9da5d28242" alt="HSL-ELK-Diagram.png" style="max-width:50%; float:right; padding:0 0 10px 10px;">F5&#x2019;s High Speed Logging (HSL) mechanism is designed to pump out as much data as can be readily consumed, with the least amount of overhead, to a pool of syslog listeners. As it happens, Elastic Stack is designed to consume data in high volume.
The HSL template packs the information into a parsable string, perfect for Logstash to interpret.
Consult F5&#x2019;s documentation on Request Logging for the official (and more detailed) procedure, but in brief, one must follow these steps:&#xA0;
<ol>
	<li>Create a Server Pool, pointing to the syslog port on each Logstash instance&#xA0;</li>
	<li>Create a Request Logging Profile, specifying the Logstash Server Pool&#xA0;</li>
	<li>Assign the Request Logging Profile to Virtual Servers as desired</li>
</ol><h2>GROK &amp; The HSL Template</h2>Out of the box, we have accomplished a difficult task easily - we have our logs flowing into Logstash. Next, we need to tell logstash how to parse the messages before indexing them into Elasticsearch. Just like the Logstash cluster is analogous to the HSL Pool, the GROK pattern is analogous to the HSL Request Logging Template. The two can be thought of as a CODEC, converting HTTP request details to a parsable string, that can be subsequently parsed into JSON for output into Elasticsearch.
Here is an example of a request logging template which will give us information about the load balancer&#x2019;s Virtual Server, Pool Name, Server IP, Response Time, in addition to other common attributes.
<pre class="prettyprint">$CLIENT_IP $DATE_NCSA $VIRTUAL_IP $VIRTUAL_NAME $VIRTUAL_POOL_NAME $SERVER_IP $SERVER_PORT &quot;$HTTP_PATH&quot; &quot;$HTTP_REQUEST&quot; $HTTP_STATCODE $RESPONSE_SIZE $RESPONSE_MSECS &quot;$Referer&quot; &quot;${User-agent}&quot;
</pre>NOTE: The F5 Logging Profile requires configuration of Request and/or Response Logging. The HTTP Response Time variable is naturally only available in the response event.
Now on the other side, in our Logstash cluster, we decode and process the log entry generated by the F5. Logstash has 3 primary phases: Input, Filter, and Output, known as the <a href="https://www.elastic.co/guide/en/logstash/current/pipeline.html">Logstash Processing Pipeline</a>
Logstash has a built-in syslog input. But because you are decoding the data manually, you should just use raw TCP and/or UDP input plugins.
The core of the filter phase is GROK - an unbiased parsing language - perfect for decomposing the log entry exactly as it was put together. Here is the GROK pattern used to decode the above template.
<pre class="prettyprint">&quot;%{IP:clientip} \[%{HTTPDATE:timestamp}\] %{IP:virtual_ip} %{DATA:virtual_name} %{DATA:virtual_pool_name} %{DATA:server} %{NUMBER:server_port} \&quot;%{DATA:path}\&quot; \&quot;(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\&quot; %{NUMBER:response:int} %{NUMBER:bytes:int} %{NUMBER:response_ms:int} %{QS:referrer} %{QS:agent}&quot;
</pre>After parsing, the filter phase can continue in any way you would like to enrich your data, such as IP address GeoLocation. DNS resolution, or even drop to Ruby for custom manipulation. Finally, the newly created event object is indexed into Elasticsearch via the Logstash Elasticsearch output plugin.
<h2>Setting up Health Checks</h2>You will want the F5 to periodically check the health of your Logstash instances, to insure that they are alive and ready to accept logs.
When setting up a health check, use caution with UDP. F5&#x2019;s UDP health check sends an arbitrary string (of your choice), and fails if it receives back an ICMP unreachable response. The caveat is that Logstash will try to process the arbitrary string, so be sure to filter it out!
Logstash also ships with a <a href="https://www.elastic.co/blog/how-to-check-logstashs-pulse">heartbeat plugin</a> that periodically says &#x2018;I&#x2019;m OK&#x2019;. This information can be output to Elasticsearch and checked by the load balancer. To set this up, you can use the following configuration example:
First create your heartbeat index in Elasticsearch, and put this mapping:
<pre class="prettyprint">PUT /heartbeats/
PUT /heartbeats/_mapping/heartbeat
{
 &quot;properties&quot;: {
 &quot;host&quot;: {
 &quot;type&quot;:&quot;string&quot;, &quot;index&quot;:&quot;not_analyzed&quot;
 }
 }
}
</pre>Then configure logstash to send a heartbeat every 5 seconds:
<pre class="prettyprint">input {
 heartbeat {
 interval =&gt; 5
 type =&gt; &quot;heartbeat&quot;
 }
}
output {
 if [type] == &quot;heartbeat&quot; {
 elasticsearch {
 protocol =&gt; &quot;transport&quot;
 index =&gt; &quot;heartbeats&quot;
 document_id =&gt; &quot;%{host}&quot;
 }
 }
}
</pre>Finally, configure the load balancer to check Elasticsearch for a heartbeat within the last 15 seconds:
<pre class="prettyprint">GET http://es1:9200/heartbeats/heartbeat/_search?q=host:logstash1 AND @timestamp:&gt;now-15s
</pre>You can test with curl:
<pre class="prettyprint">curl -s http://es1:9200/heartbeats/heartbeat/_search?q=host:&apos;logstash1&apos;%20AND%20@timestamp:%3Enow-15s
</pre><h2>UDP vs TCP for HSL</h2>UDP or TCP can be used with HSL, and there are pros and cons to each. One important thing to note is that existing TCP connections are reused for long periods of time, in order to reduce what would be unrealistic overhead. While helpful, it does cause imperfect load balancing between logstash instances. Consider using a Least Connection Algorithm over Round Robin with TCP Based HSL.
<h2>Alternatives for measuring HTTP Response Time</h2>If you don&#x2019;t have an F5, or if you are more comfortable working within the bounds of your automation infrastructure, glean the information from web servers via Logstash or Beats.
Apache&#x2019;s Default Format, with corresponding GROK:
<pre class="prettyprint">LogFormat &quot;%h %l %u %t \&quot;%r\&quot; %&gt;s %b \&quot;%{Referer}i\&quot; \&quot;%{User-Agent}i\&quot;&quot;
&quot;%{COMBINEDAPACHELOG}&quot;
</pre>Modified for Response Time:
<pre class="prettyprint">LogFormat &quot;%h %l %u %t \&quot;%r\&quot; %&gt;s %b \&quot;%{Referer}i\&quot; \&quot;%{User-Agent}i\&quot; %D&quot;
&quot;%{COMBINEDAPACHELOG} %{NUMBER:response_microseconds}&quot;
</pre><h2>See Also</h2><a href="https://www.elastic.co/guide/en/logstash/current/deploying-and-scaling.html">Deploying and Scaling Logstash</a>
</body></html>

F5 High Speed Logging with Elastic Stack

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Jay Greenberg

Надсилаючи цю форму ви погоджуєтеся з <a href="/legal/terms-of-use">Умовами надання послуг Elastic</a>. Ваші особисті дані будуть опрацьовані відповідно з <a href="/legal/privacy-statement">Положеннями про конфіденційність Elastic</a>.

Newsletter GDPR Text - Ukrainian

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Intentionally Blank

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

提交即表示您確認已詳閱並同意我們的<a href="/legal/elastic-cloud-account-terms" target="_blank">服務條款</a>，且同意 Elastic 使用您在上方提供的個人資料<a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">與您聯絡</a>，為您提供相關產品和服務的資訊。如需詳細說明或隨時想停止收到產品和服務資訊，請參閱 <a href="/legal/privacy-statement/" target="_blank">Elastic 的隱私聲明</a>。

Newsletter GDPR Text - TW

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.

GDPR Text + Google Cloud

Inviando la registrazione (1) esprimi il consenso ai <a href="/legal/terms-of-use">termini e condizioni di Elastic</a> e <a href="/legal/privacy-statement">all’informativa sulla privacy</a>, nonché (2) a ricevere saltuarie e-mail.

Newsletter GDPR Text - Italian

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Load more

Overview

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Author

Articles by Jay Greenberg

Integrating Nagios Checks with Logstash

Kibana and a Custom Tile Server for NHL Data

A case for self-monitoring systems

F5 High Speed Logging with Elastic Stack

Follow us

About us

Join us

Partners

Trust & Security

Investor relations

Excellence Awards