This post is a recap of a community talk given at a recent <a href="/elasticon">Elastic{ON} Tour event</a>. Interested in seeing more talks like this? Check out the conference archive or find out when the <a href="/elasticon">Elastic{ON} Tour</a> is coming to a city near you.
The fastest supercomputer in the world, <a href="https://www.olcf.ornl.gov/summit/">Summit</a>, is housed at Oak Ridge National Laboratory (ORNL) within the U.S. Department of Energy (DOE). Researchers at ORNL rely on Summit, which is capable of delivering 200 petaflops of computing power, to facilitate scientific discovery across a range of fields, such as materials science, neutron science, energy, national security, and high-performance computing.
To safeguard the digital information generated by all of this research, ORNL’s cybersecurity team uses <a href="/products/elasticsearch">Elasticsearch</a> and the <a href="/products/stack">Elastic Stack</a> for security information and event management (SIEM). But it wasn’t always that way. At <a href="/elasticon/tour/2018/washington-dc/log-monitoring-and-anomaly-detection-at-scale-at-ornl">Elastic{ON} Tour in Washington, D.C.</a>, Larry Nichols, Cyber Security Engineer & SIEM Admin at ORNL, explained why ORNL transitioned from Splunk to Elasticsearch to improve their ability to manage security for roughly 20,000 endpoints through log monitoring and anomaly detection at scale.
For six years, ORNL’s cybersecurity team used Splunk as their SIEM. But as ORNL’s needs evolved, requiring them to ingest more and more data and run queries on massive indices of over 30 billion documents, they began to run into challenges that led them to explore other solutions. Their Splunk license, which was based on the amount of data they ingested each day, curbed the lab’s ability to add more data. Speed was another barrier. A key goal at ORNL is to facilitate research — yet some searches on their Splunk clusters were taking as long as 15 minutes, diverting valuable time away from data analysis. “Splunk was taking more time than we wanted to spend,” Nichols said. “[T]ime that the analysts could be analyzing data instead of waiting to collect it.”
The decision to switch to Elasticsearch made sense for ORNL: they were no longer limited in the amount of data they could ingest and searches that once took minutes were down to just seconds. “Elastic obviously was appealing because we had the hardware and the resources to expand out, we just didn’t want to spend more money on the ingestion costs,” said Nichols, adding that “[t]he speed was obviously significantly better with Elasticsearch.”
With Elasticsearch, ORNL was able to deploy a SIEM that increased speed and security. Today, the lab’s production architecture runs 25 Elasticsearch nodes, all within Docker, across 25 virtual machines. Their system ingests over two billion documents each day (roughly 1.5 terabytes (TB) of data) and maintains 180 days’ worth of data (over 300 billion documents) across 10 hot nodes and 7 warm nodes. They run three machine learning nodes, three master nodes, and two coordination nodes, with a total disk usage of about 120TB of data. Their development architecture is almost identical, although they maintain fewer logs (about 30 days’ worth). ORNL has a third cluster devoted to research and testing. This cluster, which ingests about 1.5TB of data per day, features six Elasticsearch nodes on physical servers. Their final cluster, a single node, is used to monitor the other three clusters.
In addition to Elasticsearch, other parts of the Elastic Stack have enhanced the lab’s ability to identify and address security issues. With <a href="/products/kibana">Kibana</a>, the team is able to see at a glance what’s going on with their users in general and focus on specific users as needed. <a href="/products/stack/graph">Graph</a> enables them to visualize relational data based off different indices. With this information, if a machine is known to be infected, they can quickly identify any other machines that have interacted with the infected device in order to contain and resolve the issue. More recently, the team has begun using <a href="/products/stack/canvas">Canvas</a> to create dynamic, infographic-style dashboards to provide their management with high-level views of activity at the laboratory.
To learn more about how ORNL is using Elasticsearch to power research and manage security across the world’s fastest supercomputer, <a href="/elasticon/tour/2018/washington-dc/log-monitoring-and-anomaly-detection-at-scale-at-ornl">watch Nichols’s full session at Elastic{ON} Tour</a>.
<a href="/elasticon/tour/2018/washington-dc/log-monitoring-and-anomaly-detection-at-scale-at-ornl"><img src="https://images.contentstack.io/v3/assets/bltf7afce26b89a5b33/bltd89de5b6cba45c11/5c7803f0f6d704c65fb2c527/download" data-sys-asset-uid="bltd89de5b6cba45c11" alt="Learn how ORNL is using Elasticsearch to power research and manage security for the world's fastest supercomputer, Summit." "="" style="display: block; margin: auto;"></a>

ORNL_Banner.jpg

ORNL_thumb.jpg

Elasticsearch cybersecurity at the home of the world’s fastest supercomputer

Learn how John Deere is using the Elastic Stack to scale its mobile applications and deliver valuable agricultural data to farmers.

When it comes to farming, most people’s conceptions are outdated. “[T]hey think of some guy in bib overalls driving a tractor,” said Tim Arp, Senior Operations Engineer in the Intelligent Solutions Group (ISG) at John Deere. “But really, it’s all about the data now.” And that data — whether it’s GPS information powering self-driving equipment, metrics about machine performance, or even data about how crops are being planted — represents an immense amount of information pouring into John Deere’s systems.
While many people associate the company with its ubiquitous “green machines,” John Deere’s data-based applications are just as crucial to agricultural professionals. Today, John Deere supports farming operations with machine and agronomic telematics applications that help farmers maximize yield and manage costs.
At <a href="/elasticon/tour/2018/chicago/elastic-at-john-deere">Elastic{ON} Tour Chicago</a>, Arp shared how the company began this work with JDLink, their first mobile application. Originally built on 25 servers, JDLink provided farmers with an easy way of collecting data — including geographic location and machine health metrics — on farm equipment such as combines and harvesters. But as the application grew, requiring the deployment of more and more servers, the ISG began to encounter difficulty scaling, ultimately resulting in outages. It quickly became clear that to ensure that the application worked for the farmers who relied on it, the ISG needed a way to aggregate logs and consolidate data so they could visualize the system holistically.
In 2013, the ISG found a solution: the Elastic Stack. Arp and a colleague began experimenting with Elasticsearch, putting together a working demo using version 0.98. Before long, they were able to build their first production cluster on version 1.4, using 4 nodes.
<blockquote>“We started pulling in all kinds of logs: web server logs, application logs, system logs, load balancer logs. What we soon realized was all the analytics that we could get with this. No longer did we look at servers individually, but we looked at the system.” Tim Arp, Senior Operations Engineer, Intelligent Solutions Group (ISG), John Deere
</blockquote>Arp and team also used Kibana to create useful dashboards for monitoring errors and Apache web logs, visualizing response codes and hits by server.
Over the years, John Deere’s application platform has expanded to help farmers in a range of crucial areas. No longer focused solely on data management, the company offers applications to support remote management, guidance, variable rate application, and field and water management. As its offerings have grown, so has the organization’s use of Elasticsearch. &nbsp;
Their current environment, Logcentral@Deere, scaled up from 4 nodes to 25 to provide a logging framework for all of the ISG application development teams. Running Elastic Stack 6.3, the system handles 18 billion documents across 11 terabytes of storage. Roughly 14 different applications log to it at any one time, amounting to 20,000 events being streamed in every second. John Deere has also expanded Logcentral into the cloud, with 8 Elasticsearch clusters. The largest cluster, which has more than 100 applications logging to it, contains 51 nodes.
Today, John Deere relies on <a href="/ece">Elastic Cloud Enterprise</a> (ECE) for features such as support, integrated authentication and authorization, monitoring and alerts through the Elastic Stack’s alerting features (Watcher), cross cluster search, seamless upgrades, and machine learning. In the future, John Deere looks forward to making use of new capabilities such as rollups and data aggregations to maximize results for the farmers who count on them.
 <a href="/elasticon/tour/2018/chicago/elastic-at-john-deere">Watch Tim Arp’s full session at Elastic{ON} Tour</a> to learn more about how John Deere is using Elasticsearch to put critical data in the hands of agricultural professionals across the world.<a href="/elasticon/tour/2018/chicago/elastic-at-john-deere"><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltec6e28f437d97375/5c98d7b3fc9053fa5ecd78fd/john-deere-vid.jpg" data-sys-asset-uid="bltec6e28f437d97375" alt="Learn how John Deere uses the Elastic Stack to help farmers." style="display: block; margin: auto;"></a>

john-deere-banner.jpg

john-deere-thumb2.jpg

From ELK Stack to Elastic Cloud Enterprise: Scaling Up Capabilities at John Deere

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Samantha Dodge

Надсилаючи цю форму ви погоджуєтеся з <a href="/legal/terms-of-use">Умовами надання послуг Elastic</a>. Ваші особисті дані будуть опрацьовані відповідно з <a href="/legal/privacy-statement">Положеннями про конфіденційність Elastic</a>.

Newsletter GDPR Text - Ukrainian

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Intentionally Blank

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

提交即表示您確認已詳閱並同意我們的<a href="/legal/elastic-cloud-account-terms" target="_blank">服務條款</a>，且同意 Elastic 使用您在上方提供的個人資料<a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">與您聯絡</a>，為您提供相關產品和服務的資訊。如需詳細說明或隨時想停止收到產品和服務資訊，請參閱 <a href="/legal/privacy-statement/" target="_blank">Elastic 的隱私聲明</a>。

Newsletter GDPR Text - TW

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.

GDPR Text + Google Cloud

Inviando la registrazione (1) esprimi il consenso ai <a href="/legal/terms-of-use">termini e condizioni di Elastic</a> e <a href="/legal/privacy-statement">all’informativa sulla privacy</a>, nonché (2) a ricevere saltuarie e-mail.

Newsletter GDPR Text - Italian

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Load more

Overview

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Author

Articles by Samantha Dodge

Elasticsearch cybersecurity at the home of the world’s fastest supercomputer

From ELK Stack to Elastic Cloud Enterprise: Scaling Up Capabilities at John Deere

Follow us

About us

Join us

Partners

Trust & Security

Investor relations

Excellence Awards