Read about the updates and bug fixes that have been included in this release.

blog-banner-release-patch.png

blog-thumb-release-patch.png

Elastic Stack 6.8.19 released

Indexing the web is hard. As we build the new web crawler for Elastic Enterprise Search, see how we contend with the challenge of a nearly infinite supply of misbehaving sites, misapplied (or ignored) standards, duplicate content, and corner cases.

Indexing the web is hard. There’s a nearly infinite supply of misbehaving sites, misapplied (or ignored) standards, duplicate content, and corner cases to contend with. It’s a big task to create an easy-to-use web crawler that’s thorough and flexible enough to account for all the different content it encounters. In the spirit of our open source roots, we wanted to share a bit about our approach to these challenges as we build this powerful new ingestion mechanism for <a href="https://www.elastic.co/enterprise-search">Elastic Enterprise Search</a>.&nbsp; <h2>A bit of history, and indexing challenges we’ve learned from along the way</h2> We’ve come a long way since we built the first iteration of our web crawler about eight years ago (for the Swiftype Site Search product — Swiftype and Elastic joined forces in 2017). A revamped version of that web crawler has been in production on swiftype.com for the past five years, and it now handles more than a billion web pages per month. Now we’re using all that experience operating at scale to add a powerful content ingestion mechanism for the Elastic Enterprise Search solution. This new scalable and easy-to-use web crawler will allow our users to index content from any external sources, further enhancing the content ingestion picture for Elastic Enterprise Search. The following are just a few of the interesting challenges we’ve encountered — some obvious, some not as much — while working on what seems like a simple task of indexing a bunch of HTML pages into a search index. <h3>Deduplicating content</h3> One of the key problems for a web crawler is the range of user expectations regarding the process of eliminating duplicate copies of data. On one hand, there are users who follow best practices around content presentation and URL structure. They don't have any duplicate content and, when it is needed, they rely on canonical URL, robots.txt rules, and redirects to help web crawlers correctly handle their sites. Those sites tend to work with any web crawler software out of the box. On the other hand are sites that abuse standards and don't follow the best practices. A few large, popular content management systems and forum hosting software tools are especially problematic. For reasons unknown, they abuse URL and content formatting rules to a point where a small site could easily generate millions (!!!) of seemingly unique pages. Organizations using those tools tend to require powerful content deduplication and URL filtering functionality. To allow for such a wide range of different website structures and expectations towards the quality of content deduplication functionality, we’ve developed a set of features that allow users to control what content gets indexed and how they want to define content uniqueness. Our new web crawler is going to have a powerful ability to specify the rules for identifying unique pages and will ensure that duplicate pages are treated correctly for each use case. <h3>Identifying content</h3> What should we use to uniquely identify a piece of content on a website? It seems like a trivial question. Most people (and Internet bodies that create standards) tend to agree that a URL is a unique identifier of a web page. Unfortunately, the messy reality of the Internet in action breaks down this assumption rather quickly. For example: <ul> <li>Pages get moved without proper redirects — and users expect the indexed content to be updated accordingly without having duplicate pieces of content in their index.</li> <li>Multiple duplicate copies of the same page get created without any canonical URL tags — and users expect correct deduplication.</li> </ul> We want to make sure our web crawler covers the widest range of possible content movement and update scenarios. It will use sophisticated URL and content hashing techniques to maintain the search index content even when content and URLs get updated in the presence of multiple duplicate copies of the data. <h3>Dealing with standards noncompliance</h3> Unfortunately, the Internet is ultimately a nearly limitless collection of broken HTML pages and faulty HTTP protocol implementations and web servers. Any web crawler worth its weight has to account for a variety of broken elements, like some of these memorable examples we’ve encountered: <ul> <li>Huge auto-generated HTML pages with millions of links, where simply trying to parse such a page inevitably leads to a web crawler running out of memory</li> <li>Rendering of different content based on the User Agent value (aka "content cloaking"), which makes it difficult to debug and understand what is going on</li> <li>HTTP 200 response codes used for "Page not found" responses combined with a ton of broken links on the site</li> <li>100% of the page content being encoded in a JavaScript string and rendered with a <code>document.write</code> (and no other JavaScript on the page!)</li> </ul> The experience of troubleshooting and working around hundreds of cases like these over the years has led us to build a set of defensive mechanisms and content parsing features. These features will allow the web crawler to handle almost any kind of content as long as it is accessible via HTTP. <h3>Managing crawl lifecycles</h3> Over the years of crawling customer content, we’ve realized that it’s nearly impossible to predict the duration for a given web site crawl. Even for small, simple sites we had cases when a crawl would take many months and have to be manually stopped. Frequently, a random combination of issues with the site leads to an explosion of unique site URLs due to a bug in their implementation of standard protocols. Here’s a few of the “greatest hits” we’ve encountered that led to crawl lifecycle issues: <ul> <li>Sites using URLs to store the HTTP session state (e.g., old forums and a number of old Java web frameworks). Every time a web crawler would visit a page, it'd get redirected to a unique URL containing a session identifier (e.g., /some/forum?sid=123456789), and each subsequent visit would present the same page with a different unique URL.</li> <li>A relative HTML link on the "Page not found" error page combined with the site using HTTP 200 response codes for all error pages. Each incorrect URL on the site would turn into a large number of unique pages that look like /foo/foo/foo/foo....</li> <li>Websites that use URL components to control/configure content filtering/sorting in their product catalogs (e.g., /dresses/color/red/size/small), causing an explosion of seemingly unique URLs generated by combining different filters together (/small/color/red, etc.).</li> </ul> As a result of such complexity, clearly defining a finished crawl is tricky. Our new web crawler is going to include a number of built-in heuristics, allowing it to handle a wide range of situations when websites take a long time to index. Included will be URL length and depth limits, crawl duration limits, support for content-based deduplication, and a number of rules that focus on the key content while allowing deep-linked broken pages to be ignored. <h3>Adding observability</h3> An important aspect of any crawl operation is understanding what happened, what decisions were made by the system, and why. The goal is to see how each page got discovered, fetched from the web server, parsed and ingested into an engine, or, when issues occur, what failed and why. Since the new web crawler is based on Elasticsearch (the most popular platform for indexing and analysis of logging data in the world), we have made a decision early on to build as much observability as possible into the system. Every single web crawler action and decision is captured and indexed in Elasticsearch. The logs use a standard <a href="https://www.elastic.co/guide/en/ecs/1.6/ecs-reference.html">Elastic Common Schema</a> structured logging format to allow for easier correlation between different logs and very detailed analysis of the events. All of the indexed events are available for analysis using all of the powerful capabilities of <a href="https://www.elastic.co/kibana">Kibana</a> and will be documented from day one. <h2>What’s next</h2> Be on the lookout for web crawler-related product announcements and updates. If you’d like to hear more about the web crawler, check out the <a href="https://www.elastic.co/elasticon/archive/2020/global/sprinting-to-a-crawl-building-an-effective-web-crawler">Sprinting to a crawl</a> presentation from our recent ElasticON Global virtual event (along with <a href="https://www.elastic.co/elasticon/global/agenda">dozens of other presentations</a> across all Elastic solutions). And, as always, you can add search experiences to your workplace, websites, and apps with Elastic Enterprise Search, with a <a href="http://cloud.elastic.co/registration?elektra=blog-building-scalable-web-crawler">free 14-day trial of Elastic Cloud</a> (or a free <a href="https://www.elastic.co/downloads/enterprise-search">download</a>).

Building a scalable, easy-to-use web crawler for Elastic Enterprise Search

We highly recommend that all App Search users keep their deployments up to date with the latest available version to have access to new features, security updates, and performance improvements. This guide is designed to help customers through the upgrade process, to minimize the impact of an upgrade on production environments, and to ensure data safety during an upgrade. Finally, the guide helps App Search users troubleshoot any issues that may occur during an upgrade.
Pssst... Just looking for instructions to upgrade from 7.5 to 7.6? <a href="https://swiftype.com/documentation/app-search/self-managed/upgrading-to-7-6">Read the guide!</a>
 <h2>Before you begin</h2>Before attempting an upgrade of an App Search cluster to a newer version, you need to take a few steps to guarantee your data safety and increase the probability of a successful upgrade:
<ol>
	<li>Before you upgrade production servers, test the upgrades in a development environment to familiarize yourself with the process.</li>
	<ul>
		<li>Using an <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html">Elasticsearch snapshot</a> created from a production deployment may be the best option to completely test the upgrade process without risking your service availability or data consistency.</li>
	</ul>
	<li>Stop writing to your Elasticsearch cluster:</li>
	<ul>
		<li>If running App Search version 7.6 or later, enable <a href="https://swiftype.com/documentation/app-search/self-managed/read-only-mode">read-only mode</a> to guarantee a consistent snapshot of your data.</li>
		<li>For versions before 7.6, manually stop all write/indexing operations to your cluster at the source.</li>
	</ul>
	<li>Back up your data with <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html">Elasticsearch snapshots</a>. To roll back to an earlier version of App Search, you must have a backup of your data stored in Elasticsearch.</li>
</ol> <h2>Upgrade process overview</h2>How you upgrade App Search is going to depend on many factors:
<ul>
	<li>The mode of deployment, either self-managed or Elastic Cloud</li>
	<li>Your ability to create Elasticsearch snapshots and recover from them</li>
	<li>Available resources. Can you spin up a new Elasticsearch cluster and/or new App Search instances while the old infrastructure is running?</li>
</ul>We will outline different scenarios for the upgrade process below, but first, here is an overview of the App Search upgrade process to help you navigate the different approaches outlined further in this guide:
<ul>
	<li>App Search uses Elasticsearch as its only data store. The Elasticsearch cluster is used for control layer data (engines, settings, etc.), documents, search indexes, API logs, and analytics events.</li>
	<li>Every time a new instance of App Search starts, it checks every index within the Elasticsearch cluster to see if it needs upgrading to a newer version. When its structure changes, we have to reindex the data stored in an index.&nbsp;</li>
	<li>All upgrades are done in a nondestructive way and are performed automatically. Existing indexes are never deleted and we only create new indexes with a copy of the data.</li>
	<li>If there are multiple App Search instances starting against a single Elasticsearch cluster, they will use a distributed locking mechanism to coordinate the upgrade process and ensure it is done in a consistent way. We create an installation lock record in Elasticsearch to do this.</li>
	<li>Until the upgrade process is complete, the new version of App Search will not begin to serve API requests.</li>
</ul>Given the details of the upgrade process described above, there are a few different options available for performing an upgrade of an App Search deployment based on your ability to handle App Search downtime during an upgrade and to change App Search client configuration after an upgrade:
<ol>
	<li>Simple in-place upgrade (with downtime, no client-side changes) – replace a set of App Search instances with a new set running a newer version. In other words, you would shut down the instance, upgrade the packages, then start it back up and let the instances migrate the data if needed.&nbsp;</li>
	<li>Snapshot-based upgrades (no downtime, client-side changes needed) – create a snapshot of an Elasticsearch cluster used by App Search, restore the snapshot to a new Elasticsearch cluster, and then start App Search on the new cluster.</li>
	<li>In-place upgrades in <a href="https://swiftype.com/documentation/app-search/self-managed/read-only-mode">read-only mode</a> (write downtime, no client-side changes) – switch a cluster to read-only mode, start new App Search instances to perform the upgrade, shut down old instances, and remove read-only mode.</li>
</ol>Unfortunately, we cannot prescribe the best upgrade path for your specific situation. It will depend on your requirements for App Search service availability, on the capabilities of the platform you are using to manage the deployment, and on available resources, etc. See the details below to better understand the pros and cons of each upgrade path.
 <h3>In-place upgrade with downtime</h3>If you are able to handle downtime of the App Search service, the easiest upgrade method is an in-place upgrade. We recommend this method to all customers who are able to schedule a maintenance window for their service or for non-mission-critical applications based on App Search.
This upgrade method has a few characteristics that make it uniquely suitable for many deployments:
<ul>
	<li>No need for new infrastructure – everything is done in-place and you do not need to provision any new instances of Elasticsearch or App Search.</li>
	<li>No need to change client configurations – all of your API clients can use the same API endpoints before and after the upgrade since new App Search instances will simply replace the existing ones.</li>
</ul>The disadvantages of this method:
<ul>
	<li>Downtime required – you have to shut down all of your App Search instances to perform the upgrade and your service will not be available until the upgrade is complete. With proper planning, the downtime period can be reduced significantly, but downtime is still unavoidable.</li>
	<li>Harder to roll back – if your upgrade fails for any reason, we do not recommend rolling back to the older version of App Search due to potential issues with the partially migrated dataset in Elasticsearch. You would have to restore from a backup to get back to your original state, which would prolong the downtime.</li>
</ul>Here are the simple steps you need to take:
<ol>
	<li>Stop ALL of your App Search instances.</li>
	<li>Back up your data from Elasticsearch, using <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html">snapshots</a>.</li>
	<li>Upgrade App Search packages on your servers, or change your container image tags to point to the latest version if you use Docker or Kubernetes.</li>
	<li>Start up the new version of App Search, and new instances will take care of coordinating and performing the upgrade before starting up and accepting your API traffic.</li>
</ol> <h3>Snapshot-based upgrades</h3>For situations where App Search downtime is not acceptable or in cases where you want to ensure a rollback is possible – no matter what happens during an upgrade – the safest way to perform the upgrade is through snapshot-based cloning of a deployment.
As you can see, this process is a lot more involved, but it does guarantee data consistency and allows you to perform a migration without a downtime of the Search API. Here are the advantages of this method:
<ul>
	<li>No Search API downtime required – both the old App Search deployment and the new one are able to handle search traffic throughout the upgrade process, meaning your clients should not notice the migration at all.</li>
	<li>Easy to rollback – if you notice any issues with the new deployment (B) of App Search, you can retry the migration as many times as you need to, since your original deployment is still functional.</li>
</ul>The method has a few important disadvantages:
<ul>
	<li>The need for additional infrastructure – you need to provision a new Elasticsearch cluster and deploy a set of new App Search instances during the upgrade. This requires some coordination and additional compute resources during the upgrade process.</li>
	<ul>
		<li>For some deployments it may be possible to scale down Elasticsearch and App Search clusters before using the freed-up hardware to provision new instances, but the details of that operation are beyond the scope of this guide.</li>
	</ul>
	<li>The need to change the client configuration – your API clients need to be switched to the new deployment endpoint after you perform the upgrade.&nbsp;</li>
	<ul>
		<li>Please note: You could proxy your traffic through a load-balancer (ELB, ALB, etc) or a CDN to keep your API endpoint stable while you replace the App Search cluster behind the proxy.</li>
	</ul>
</ul>Here are the steps you need to take:
<ol>
	<li>Stop writes into your App Search deployment (A):</li>
	<ol>
		<li style="list-style-type: lower-alpha;">Starting with 7.6.0, you can use the App Search <a href="https://swiftype.com/documentation/app-search/self-managed/read-only-mode">read-only</a> feature to block the deployment and ensure a consistent migration.</li>
		<li style="list-style-type: lower-alpha;">For all versions below 7.6.0, you should either stop the writes manually (disable your indexing jobs, etc.) or use the Elasticsearch API to put a write lock on all App Search indexes.</li>
	</ol>
	<li>Create a backup of your Elasticsearch cluster (A) using snapshots.</li>
	<li>Create a new Elasticsearch cluster (B) and restore data from the latest snapshot.</li>
	<li>Deploy a new set of App Search instances (B) using the the new Elasticsearch cluster (B) as the data store.</li>
	<li>At this point you should have two separate App Search deployments both serving the same data. You should spot-check to make sure the new deployment (B) looks correct to you.</li>
	<li>Switch API traffic from the original App Search deployment (A) to the new one (B).</li>
	<li>Shut down the old App Search and Elasticsearch clusters.</li>
</ol> <h2>In-place upgrade with read-only mode (7.6-only)</h2>Finally, for situations where you do not have the ability to provision new infrastructure during an upgrade, but scheduling App Search downtime is not an option, starting with App Search 7.6.1 you can perform an in-place upgrade using App Search’s <a href="https://swiftype.com/documentation/app-search/self-managed/read-only-mode">read-only mode</a>.
This method has the following advantages:
<ul>
	<li>No Search API downtime required – both old and new App Search instances are able to handle search traffic throughout the upgrade process, meaning clients should be unaffected by the migration.</li>
	<li>No need to change client configurations – all of your API clients can use the same API endpoints before and after the upgrade, since new App Search instances will simply replace the existing ones.</li>
	<li>No need for new infrastructure – everything can be done in-place and you do not need to provision any new instances of Elasticsearch or App Search.</li>
</ul>The only comparative disadvantage of this upgrade method is that it makes it harder to roll back if your upgrade fails for any reason. If you experience any issues during the upgrade, we do not recommend enabling writes on the older version of App Search due to potential issues with the partially migrated dataset in Elasticsearch. You would have to restore from a backup to return to your original state, which may require downtime.
Here are the steps you need to take:
<ol>
	<li>Stop writes to your App Search deployment:</li>
	<ol>
		<li style="list-style-type: lower-alpha;">If possible, stop your indexing requests at the source (disable indexing jobs, etc.).</li>
		<li style="list-style-type: lower-alpha;">Enable <a href="https://swiftype.com/documentation/app-search/self-managed/read-only-mode">read-only mode</a> on your App Search cluster.</li>
	</ol>
	<li>Back up your data from Elasticsearch (using snapshots).</li>
	<li>Start a new set of App Search instances running the latest version of the product:</li>
	<ol>
		<li style="list-style-type: lower-alpha;">You can do a rolling upgrade – stopping, upgrading, and starting one instance at a time.</li>
		<li style="list-style-type: lower-alpha;">Or you could provision a full new fleet of App Search instances.</li>
	</ol>
	<li>New App Search instances will perform an upgrade and then become available to serve your Search API traffic.</li>
	<li>Once new App Search instances are running, stop your old App Search instances.</li>
	<li>Finally, remove <a href="https://swiftype.com/documentation/app-search/self-managed/read-only-mode">read-only mode</a> from the cluster and re-enable your indexing jobs. You may need to retry any indexing requests that may have failed during the upgrade.</li>
</ol> <h2>Upgrades on Elastic Cloud</h2>When deploying App Search on Elastic Cloud, you have access to two separate upgrade paths depending on your tolerance for App Search downtime:
<ol>
	<li>The default upgrade path (via the Cloud Console) is performed in-place via a full cluster restart, meaning App Search will be unavailable for the duration of the upgrade. In future versions of App Search, this mode will allow for in-place upgrades via <a href="https://swiftype.com/documentation/app-search/self-managed/read-only-mode">read-only mode</a>, but as of 7.6.0 downtime is required.</li>
</ol><ol start="2">
	<li>If downtime is unacceptable for your use case, we recommend a snapshot-based upgrade process, which involves taking a snapshot of your deployment (ideally in <a href="https://swiftype.com/documentation/app-search/self-managed/read-only-mode">read-only mode</a>) and then creating a new App Search deployment from the snapshot using a newer App Search version. Please note: The snapshot-based upgrade process requires client-side changes to switch users to another deployment, but guarantees availability of the service during the upgrade process. If you want to keep your client configuration stable, you can proxy your API traffic through a load balancer (ELB, ALB, etc.) or a CDN endpoint.</li>
</ol>Please note: If you are using Elastic Cloud or ECE versions 2.4.0 to 2.4.4, there is a known problem with App Search upgrades from versions 7.5 to 7.6 (rolling upgrades could lead to limited data loss if you are actively modifying new engines during the upgrade process). As a workaround, before upgrading the 7.5 App Search instances to 7.6, you can stop the traffic towards those instances by clicking the Stop routing button in the UI. Once all 7.5 instances are no longer routing traffic, it is safe to perform the upgrade by clicking the Upgrade button.
 <h2>Troubleshooting App Search upgrades</h2>As with any software upgrades, there is always a chance that things will not go according to plan. This means you need to plan and prepare for the possibility of your App Search upgrade failing. You can review the comparison of different upgrade methods above and ensure you are comfortable with the risks associated with your preferred upgrade method before starting the upgrade process.
If you experience an upgrade failure, the information below should help you identify the causes of the issue and allow you to retry the upgrade if needed. App Search should be able to recover from many potential failures, meaning you should be able to retry an upgrade if it fails.
If your App Search upgrade fails, please do the following:
<ol>
	<li>Check the App Search application logs of the new deployment – log/app-search.log or /var/log/app-search/app-search.log – to see the details of what happened.</li>
	<li>If the upgrade failed due to an issue with Elasticsearch that you can fix, do that and then re-attempt the upgrade by starting a new App Search instance again.</li>
	<li>If the process consistently fails, roll back using a method specific to your preferred upgrade path.</li>
	<li>Capture the app-search.log file and file a support case with Elastic if you need further help troubleshooting the upgrade process.</li>
</ol>

blog-thumb-app-search-service.png

How to upgrade Elastic App Search

Elastic App Search provides you with a robust set of API endpoints to build any type of search experience. While modern tools and a blank canvas can be inspiring, we want you to craft functional and valuable things as quickly as possible. And so, we bring you the App Search Reference UI. 
<h2>Open Sourcery
</h2>The <a href="https://github.com/swiftype/app-search-reference-ui-react">Elastic App Search Reference UI</a> is written in&nbsp;React. It is an open source developer tool that has been woven into the App Search dashboard. Its goal is twofold:
<ol>
<li>Provide a clean and dynamic search preview to experiment with features like facets, pagination, and sorting.</li>
<li>Generate a quality design and developmental foundation for new search experiences. &nbsp;
</li>
</ol>You can see a live Reference UI by visiting: <a href="https://parks.swiftype.info">https://parks.swiftype.info</a>.
<h2>Field Day
</h2>Before we dig deeper into the Reference UI, we should consider a key goal of App Search: provide relevant search. Relevance is the key to the value of search, for both your end-user and your business goals. But what does it mean, to be relevant?
Relevance is how well a result can fulfill the needs of the user or system that requested it. In technical terms, a query is analyzed against document fields and the most relevant document is returned given how well the field values match the query.
Code and national parks feel like a nice pair, and so we will take a practical look at the Reference UI through the eyes of the national parks data set. It has a schema containing the following fields:
<pre class="prettyprint">{
 "description": "text",
 "nps_link": "text",
 "states": "text",
 "title": "text",
 "visitors": "number",
 "world_heritage_site": "text",
 "location": "geolocation",
 "acres": "number",
 "square_km": "number",
 "date_established": "date"
}
</pre>Within this data set, a park is less vast and wood-y, like this…
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt5594e4870938fc30/5c304c8fc0508d076e1712e1/mosquito-creek.jpg" data-sys-asset-uid="blt5594e4870938fc30" alt="" "="">
… And more concise and document-y, like this…
<pre class="prettyprint">{
 "nps_link": "https://www.nps.gov/cong/index.htm",
 "title": "Congaree",
 "date_established": "2003-11-10T06:00:00+00:00",
 "world_heritage_site": "false",
 "states": "South Carolina",
 "description": "On the Congaree River, this park is the largest portion of old-growth floodplain forest left in North America. Some of the trees are the tallest in the eastern United States. An elevated walkway called the Boardwalk Loop guides visitors through the swamp.",
 "visitors": 143843,
 "id": "park_congaree",
 "location": "33.78,-80.78",
 "square_km": 106.3,
 "acres": 26275.82
}
</pre>While not nearly as lush, the JSON-ified version is more searchable. We want to match queries against field values with great accuracy, which is why we need structure.
If we search for “old growth” within our search engine, Congaree National Park will be our first result. The text “old-growth” is included as part of the value of the `description` field. But we also return 10 other parks which describe themselves as “old-growth”: Yellowstone, Yosemite, Petrified Forest...
The results are relevant&nbsp;—&nbsp;we searched for “old growth” and we returned 10 parks which contain old-growth forests. But, what now? We have 10 results and we would like 1.Relevance can be fine tuned by the search provider or by the search user.
A search provider can leverage a feature like App Search Curations to promote, rearrange, and hide documents for a given query. If Yellowstone park would make a better first result for “old growth” forests, then we can curate the result set to place it on top. If Congaree were to close its doors to visitors, we could hide it from our results.
A search user can use the interactive tools provided within the search experience. Two such tools are present within the Reference UI: facets and sorting. A user can choose to narrow down the parks by state and whether or not they are a world heritage site. They can sort by the number of acres or the number of visitors.
An excellent search experience will not just provide relevant and tuned results by default, but allow searchers to further explore into the result data.
<h2>Back to the Lab </h2>After our trip to the woods, we are now back within the App Search dashboard. We have an Engine filled with national parks and we have clicked the Reference UI button within Engine navigation.
Configuring the Reference UI is a matter of defining fields...
<ul>
	<li>Title Field: The prominent name or title to display within search results. We will use the <code>title</code> field.</li>
	<li>Facet Field: A facet is a count based on a field. It is a way to refine search results using a shared field value as a grouping point. We will facet using the <code>state</code> and <code>world_heritage_site</code> fields.</li>
	<li>Sort Field: Ascending, or descending? And based on which field? We will choose the <code>acres</code> and <code>visitors</code> fields, so that people can find parks which are large or small, busy or quiet.</li>
	<li>URL Field: Search results will often link back to something else within your website or application: a product, an article, whatever it may be. We have the <code>nps_link</code> field.</li>
</ul>As we fill in these fields, the image to the left will illuminate which function you are tuning. Give it a try using your own Engine, or your national parks demo Engine.
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt85998fee2ed05d5e/5c304c8825a401fc6fe4f49b/reference-ui-with-fields.png" data-sys-asset-uid="blt85998fee2ed05d5e" alt="" "="">
When we click Generate a Preview, it appears just like our sample application. And when we search for “old growth"...
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltbbd9c245556d3fe8/5c304c84ebb7714d66027022/reference-ui-search-demo.png" data-sys-asset-uid="bltbbd9c245556d3fe8" alt="" "="">
We get parks. Old growth parks! Our search term is highlighted, we have a Relevance dropdown to change our sort order and field, a Show dropdown to change the size of our result set, and a sidebar with our faceting parameters: states and world heritage sites. But we still have 10 parks… No problem, the user will use the faceting, sorting, and pagination tools available and pick the best one.
<h2>Growing the Seed </h2>If you were just looking to experiment with relevance tuning or seeking inspiration, then we might be finished. If we want to start building, then we can click Download.
Clicking download retrieves a .zip file containing the Reference UI repository pre-populated with your configuration data. The repository will also contain a unique <code>config/engine.json</code> generated by App Search based on your account and field information.
It looks like this:
<pre class="prettyprint">{
 "engineName": "sample-engine",
 "hostIdentifier": "host-2376rb",
 "searchKey": "search-soaewu2ye6uc45dr8mcd54v8", 
 "fields": [
 "description",
 "nps_link",
 "location",
 "states",
 "title",
 "visitors",
 "world_heritage_site",
 "acres",
 "square_km",
 "date_established"
 ],
 "sortFields": [
 "visitors",
 "acres"
 ],
 "facets": [
 "states",
 "world_heritage_site"
 ],
 "titleField": "title",
 "urlField": "nps_link"
}
</pre>Now, within the unzipped Reference UI repository…
<pre class="prettyprint">npm install && npm start
</pre>Your Reference UI will run locally and provide search out of the box. The UI is yours to fork, customize, and build upon however you wish. The repository is maintained should you want to report issues or contribute changes upstream.
The Reference UI is not far from a viable search implementation. Styling can be changed within <code>src/styles</code> and the HTML structure can be altered within <code>src/components</code>. The README included within the repository will guide you through the relationship between the React components and the App Search APIs which perform the heavy lifting of search.
As with any open source initiative, we are excited to see what sort of UIs the community will create. Perhaps your next search experience will not only provide relevant and valuable search to your users, but inspire others to push the state of search to exciting new places.
<h2>Summary </h2>The Elastic App Search Reference UI is a foundation to help you build or test great search experiences. Whether you are looking to explore your relevance tuning in a closer-to-production sandbox or you are building a brand new experience, you have the tools you need and the seed to get started. You can read the Reference UI guide in the <a href="https://swiftype.com/documentation/app-search/guides/reference-ui">App Search documentation</a> for more information.
Enjoy the Reference UI and Elastic App Search with a <a href="/cloud/app-search-service">14-day free trial</a> — no credit card required.

plants-growin-nicely.jpg

plant-yaaaas.jpg

Kickstart Search with the Elastic App Search Reference UI

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

headshot-oleksiy-kovyrin-300x300.jpg

Tech Lead

Oleksiy Kovyrin

Надсилаючи цю форму ви погоджуєтеся з <a href="/legal/terms-of-use">Умовами надання послуг Elastic</a>. Ваші особисті дані будуть опрацьовані відповідно з <a href="/legal/privacy-statement">Положеннями про конфіденційність Elastic</a>.

Newsletter GDPR Text - Ukrainian

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Intentionally Blank

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

提交即表示您確認已詳閱並同意我們的<a href="/legal/elastic-cloud-account-terms" target="_blank">服務條款</a>，且同意 Elastic 使用您在上方提供的個人資料<a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">與您聯絡</a>，為您提供相關產品和服務的資訊。如需詳細說明或隨時想停止收到產品和服務資訊，請參閱 <a href="/legal/privacy-statement/" target="_blank">Elastic 的隱私聲明</a>。

Newsletter GDPR Text - TW

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.

GDPR Text + Google Cloud

Inviando la registrazione (1) esprimi il consenso ai <a href="/legal/terms-of-use">termini e condizioni di Elastic</a> e <a href="/legal/privacy-statement">all’informativa sulla privacy</a>, nonché (2) a ricevere saltuarie e-mail.

Newsletter GDPR Text - Italian

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Load more

Overview

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Author

Articles by Oleksiy Kovyrin

Tech Lead, Elastic

Elastic Stack 6.8.19 released

Building a scalable, easy-to-use web crawler for Elastic Enterprise Search

How to upgrade Elastic App Search

Kickstart Search with the Elastic App Search Reference UI

Follow us

About us

Join us

Partners

Trust & Security

Investor relations

Excellence Awards