Elastic Common Schema .NET library and integrations released

2000-scaffold.jpg

720-scaffold.jpg

NEST and Elasticsearch.Net: upgrading your Elasticsearch server and codebase

2000x415.jpg

720x480.jpg

Building real-time address search with the Australian G-NAF dataset

The NEST and Elasticsearch.Net 7.0 clients are now GA. Learn more about performance improvements, a simplified codebase, and parity with Elasticsearch 7.0+.

blog-banner-release-elasticsearch.png

blog-thumb-release-elasticsearch.png

NEST and Elasticsearch.Net 7.0 now GA

<h2>Introduction</h2>
There are a number of ways you can index documents into Elasticsearch using the NEST Elasticsearch .NET client.
This blog post will demonstrate some of the simple methods, from indexing a single document at a time, to more advanced methods using the BulkObservable helper.
<h2>Single documents</h2>
Within NEST, a document is modelled as POCO (plain old CLR object), an example is given below:<pre class="prettyprint">public class Person { public int Id { get; set; } public string FirstName { get; set; } public string LastName { get; set; } } </pre>
An instance of this object, which represents a single document in Elasticsearch, can then be indexed using a few different methods. Let's use the following instance as an example:<pre class="prettyprint">var person = new Person { Id = 1, FirstName = "Martijn", LastName = "Laarman" }; </pre>
The IndexDocument&lt;T&gt; and IndexDocumentAsync&lt;T&gt; methods provide a simple way to index a single document of type T, using default parameters. The result of this method call can be inspected to determine if the indexing operation was successful.<pre class="prettyprint">// synchronous method that returns an IIndexResponse object var indexResponse = client.IndexDocument(person); //asynchronous method that returns a Task&lt;IIndexResponse&gt; that can be awaited var indexResponseAsync = await client.IndexDocumentAsync(person); // Inspect the result of the synchronous operation if (!indexResponse.IsValid) { // If the request isn't valid, we can take action here } </pre>
The IsValid property can be used to check if a response is functionally valid or not. This is a NEST abstraction to have a single point to check whether something wrong happened with the request.
If you need to set additional parameters when indexing a document, you can use the fluent or object initializer syntax. This will give you finer control over the indexing process. In the example below we will index the document into an index named “people”.<pre class="prettyprint">// fluent syntax var fluentIndexResponse = client.Index(person, i =&gt; i.Index("people")); // object initializer syntax var initializerIndexResponse = client.Index(new IndexRequest&lt;Person&gt;(person, "people")); </pre>
A naïve approach to indexing multiple documents would be to simply create a loop to index a single document on each iteration; however, this is a very inefficient approach that will not scale well for large document collections.
<h2>Multiple documents</h2>
The bulk API can be used for indexing multiple documents. First, let’s create a collection of documents to index:<pre class="prettyprint">var people = new [] { new Person { Id = 1, FirstName = "Martijn", LastName = "Laarman" }, new Person { Id = 2, FirstName = "Stuart", LastName = "Cam" }, new Person { Id = 3, FirstName = "Russ", LastName = "Cam" } // snip }; </pre>
Multiple documents can be indexed using the IndexMany and IndexManyAsync methods, either synchronously or asynchronously, respectively. These methods are specific to the NEST client and wrap calls to the client's Bulk method and bulk API, providing a convenient shortcut for indexing many documents.
Note that these methods index all documents in a single HTTP request, so for very large document collections, you need to partition the collection into many smaller batches and issue multiple Bulk calls. When you find yourself needing to do this, consider using the BulkAllObservable&lt;T&gt; helper instead, described later in the post.<pre class="prettyprint">// synchronous method that returns an IBulkResponse var indexManyResponse = client.IndexMany(people); if (indexManyResponse.Errors) { // the response can be inspected for errors foreach (var itemWithError in indexManyResponse.ItemsWithErrors) { // if there are errors, they can be enumerated and inspected Console.WriteLine("Failed to index document {0}: {1}", itemWithError.Id, itemWithError.Error); } } // alternatively, documents can be indexed asynchronously var indexManyAsyncResponse = await client.IndexManyAsync(people); </pre>
If you require finer-grained control over indexing many documents you can use the Bulk and BulkAsync methods and use the descriptors to customise the bulk calls.
As with the IndexMany methods above, documents are sent to the _bulk endpoint in a single HTTP request. This does mean that consideration will need to be given to the overall size of the HTTP request. For indexing large numbers of documents you'll likely want to use the BulkAllObservable&lt;T&gt; helper.<pre class="prettyprint">// returns an IBulkResponse that can be inspected for errors var bulkIndexResponse = client.Bulk(b =&gt; b .Index("people") .IndexMany(people) ); // asynchronous version var asyncBulkIndexResponse = await client.BulkAsync(b =&gt; b .Index("people") .IndexMany(people) ); </pre>
<h2>BulkAllObservable&lt;T&gt; helper</h2>
Using the BulkAllObservable&lt;T&gt; helper allows you to focus on the overall objective of indexing a collection of documents, without having to concern yourself with retry, backoff or batching mechanics.
Multiple documents can be indexed using the BulkAll method and BlockingSubscribeExtensions Wait() extension method. This helper exposes functionality to automatically retry / backoff in the event of an indexing failure, and to control the number of documents indexed in a single HTTP request.
In the following example, each request indexes 1000 documents, batched from the original input. In the event of a large number of documents this could result in many HTTP requests, each containing 1000 documents (the last request may contain less, depending on the total number).
The helper lazily enumerates an IEnumerable&lt;T&gt; collection, allowing you to index a large number of documents easily, such as documents materialized from paginated database records.<pre class="prettyprint">var bulkAllObservable = client.BulkAll(people, b =&gt; b .Index("people") // how long to wait between retries .BackOffTime("30s") // how many retries are attempted if a failure occurs .BackOffRetries(2) // refresh the index once the bulk operation completes .RefreshOnCompleted() // how many concurrent bulk requests to make .MaxDegreeOfParallelism(Environment.ProcessorCount) // number of items per bulk request .Size(1000) ) // Perform the indexing, waiting up to 15 minutes. // Whilst the BulkAll calls are asynchronous this is a blocking operation .Wait(TimeSpan.FromMinutes(15), next =&gt; { // do something on each response e.g. write number of batches indexed to console }); </pre>
The BulkAllObservable&lt;T&gt; helper exposes a number of advanced features.
<ol>
 <li>BufferToBulk allows for the customisation of individual operations within the bulk request before it is dispatched to the server.</li>
 <li>RetryDocumentPredicate enables fine control on deciding if a document that failed to be indexed should be retried.</li>
 <li>DroppedDocumentCallback in the event a document is not indexed, even after retrying, this delegate is called.</li>
</ol><pre class="prettyprint">client.BulkAll(people, b =&gt; b .BufferToBulk((descriptor, list) =&gt; { // customise the individual operations in the bulk // request before it is dispatched foreach (var item in list) { // index each document into either even-index or odd-index descriptor.Index&lt;Person&gt;(bi =&gt; bi .Index(item.Id % 2 == 0 ? "even-index" : "odd-index") .Document(item) ); } }) .RetryDocumentPredicate((item, person) =&gt; { // decide if a document should be retried in the event of a failure return item.Error.Index == "even-index" &amp;&amp; person.FirstName == "Martijn"; }) .DroppedDocumentCallback((item, person) =&gt; { // if a document cannot be indexed this delegate is called Console.WriteLine($"Unable to index: {item} {person}"); }) ); </pre>
<h2>Ingest nodes</h2>
Since Elasticsearch will automatically reroute ingest requests to ingest nodes, you don't have to specify or configure any routing information. However, if you're doing heavy ingestion and have dedicated ingest nodes, it makes sense to send index requests to these nodes directly, to avoid any extra hops in the cluster.
The simplest way to achieve this is to create a dedicated "indexing" client instance, and use it for indexing requests.<pre class="prettyprint">// list of ingest nodes var pool = new StaticConnectionPool(new [] { new Uri("http://ingestnode1:9200"), new Uri("http://ingestnode2:9200"), new Uri("http://ingestnode3:9200") }); var settings = new ConnectionSettings(pool); var indexingClient = new ElasticClient(settings); </pre>
In complex cluster configurations it can be easier to use a sniffing connection pool along with a node predicate to filter out the nodes that have ingest capabilities. This allows you to customise the cluster and not have to reconfigure the client.<pre class="prettyprint">// list of cluster nodes var pool = new SniffingConnectionPool(new [] { new Uri("http://node1:9200"), new Uri("http://node2:9200"), new Uri("http://node3:9200") }); // predicate to select only nodes with ingest capabilities var settings = new ConnectionSettings(pool).NodePredicate(n =&gt; n.IngestEnabled); var indexingClient = new ElasticClient(settings); </pre>
<h2>Ingest pipelines</h2>
Let’s modify our Person type to include some additional information:<pre class="prettyprint">public class Person { public int Id { get; set; } public string FirstName { get; set; } public string LastName { get; set; } public string IpAddress { get; set; } public GeoIp GeoIp { get; set; } } public class GeoIp { public string CityName { get; set; } public string ContinentName { get; set; } public string CountryIsoCode { get; set; } public GeoLocation Location { get; set; } public string RegionName { get; set; } } </pre>
We can create an ingestion pipeline that manipulates the incoming values before they are indexed. Let’s assume that our application always expects surnames to be capitalised, and for initials to be indexed into their own field. We also have an IP address that we'd like to convert into human-readable location.
We could achieve this requirement by creating a custom mapping and creating an ingest pipeline. The new Person type can then be used without making any further changes.
First, we’ll create the index and custom mapping:<pre class="prettyprint">client.CreateIndex("people", c =&gt; c .Mappings(ms =&gt; ms .Map&lt;Person&gt;(p =&gt; p //automatically create the mapping from the type .AutoMap() //override any inferred mappings from AutoMap() .Properties(props =&gt; props // create an additional field to store the initials .Keyword(t =&gt; t.Name("initials")) //map field as IP Address type .Ip(t =&gt; t.Name(dv =&gt; dv.IpAddress)) // map GeoIp as object .Object&lt;GeoIp&gt;(t =&gt; t.Name(dv =&gt; dv.GeoIp)) ) ) ) ); </pre>
Next we’ll create an ingest pipeline, taking advantage of the bundled ingest-geoip plugin, now bundled in version 6.7.<pre class="prettyprint">client.PutPipeline("person-pipeline", p =&gt; p .Processors(ps =&gt; ps //uppercase the lastname .Uppercase&lt;Person&gt;(s =&gt; s .Field(t =&gt; t.LastName) ) // use a painless script to populate the new field .Script(s =&gt; s .Lang("painless") .Source("ctx.initials = ctx.firstName.substring(0,1) + ctx.lastName.substring(0,1)") ) // use ingest-geoip plugin to enrich the GeoIp object from the supplied IP Address .GeoIp&lt;Person&gt;(s =&gt; s .Field(i =&gt; i.IpAddress) .TargetField(i =&gt; i.GeoIp) ) ) ); </pre>
Now let’s index a Person instance using this new index and ingest pipeline.<pre class="prettyprint">var person = new Person { Id = 1, FirstName = "Martijn", LastName = "Laarman", IpAddress = "139.130.4.5" }; // index the document using the created pipeline var indexResponse = client.Index(person, p =&gt; p .Index("people") .Pipeline("person-pipeline") ); </pre>
Searching now shows the indexed document with the enriched values.<pre class="prettyprint">{ "took": 5, "timed_out": false, "_shards": { "total": 1, "successful": 1, "skipped": 0, "failed": 0 }, "hits": { "total": 1, "max_score": 1, "hits": [ { "_index": "people", "_type": "person", "_id": "1", "_score": 1, "_source": { "firstName": "Martijn", "lastName": "LAARMAN", "initials": "ML", "geoIp": { "continent_name": "Oceania", "region_iso_code": "AU-NSW", "city_name": "Sydney", "country_iso_code": "AU", "region_name": "New South Wales", "location": { "lon": 151.2167, "lat": -33.7333 } }, "ipAddress": "139.130.4.5", "id": 1 } } ] } } </pre>
When a pipeline is specified, there will be the added overhead of document enrichment when indexing; in the example given above, the execution of the uppercasing and the Painless script.
For large bulk requests, it could be prudent to increase the default indexing timeout to avoid exceptions.<pre class="prettyprint">client.Bulk(b =&gt; b .Index("people") .Pipeline("person-pipeline") //increases the timeout on Elasticsearch server-side .Timeout("5m") .IndexMany&lt;Person&gt;(people) .RequestConfiguration(rc =&gt; rc // increases the HTTP request timeout on the client, before aborting request .RequestTimeout(TimeSpan.FromMinutes(5)) ) ); </pre>
<h2>In summary</h2>
In this blog post we have covered the simple case of indexing a single document, through to bulk indexing multiple documents with ingest pipelines.
Give it a try in your own cluster, or spin up a 14-day free trial of the <a href="/elasticsearch/service/" target="_self">Elasticsearch Service</a> on <a href="https://www.elastic.co/cloud/" target="_self">Elastic Cloud</a>. And if you run into any problems or have any questions, reach out on the <a href="https://discuss.elastic.co" target="_self">Discuss</a> forums.
For the full documentation of indexing using the NEST Elasticsearch .NET client <a href="https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/index.html" target="_self">please refer to our docs.</a>

Indexing documents with the NEST Elasticsearch .NET client

task-management-with-nest-client

task_management_api_thumb.jpg

Task Management with the NEST Elasticsearch .NET Client

<html><head></head><body>Today we are pleased to announce the General Availability (GA) release of our .NET clients for Elasticsearch 6.x.
&quot;But wait, wasn&apos;t Elasticsearch 6.1 just released?&quot;
Yes, this is true; whilst the .NET clients have had several pre-releases compatible with Elasticsearch 6.0 and 6.1, we intentionally pushed back on our GA release until we were overwhelmingly happy it was ready. We hope that the reasoning for this delay will become clear as you read through this post.
<h2 id="6-0-compatibility">6.0 Compatibility</h2>This client has full feature parity with Elasticsearch 6.0 and has been tested against all currently released Elasticsearch 6.x versions. You will be able use this release with versions 6.1 and the forthcoming 6.2 release without issue.
For some of the newer features in Elasticsearch 6.1, such as the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-composite-aggregation.html">composite aggregation</a> and <a href="https://www.elastic.co/blog/elasticsearch-machine-learning-on-demand-forecasting">machine learning forecasting</a>, we&apos;ll push out a newer release of 6.x in the future, to incorporate these.
<h2 id="ch-ch-changes">Ch... Ch... Changes</h2>The 6.0 GA release comprises well over 650 commits and, in the process, has shrank our Github issues down from four pages to just half of one. We&apos;ll spare you from all the gory details and effort that went into this, and call out only the most impactful changes that you&apos;re likely to come across.
As with any major client release, <a href="https://www.elastic.co/guide/en/elasticsearch/client/net-api/6.x/_breaking_changes.html" target="_blank">we publish an exhaustive list of all the binary breaking changes</a>. Please don&apos;t let this long list intimidate you! Most of these you are unlikely to encounter, and we provide them all simply for completeness. So, onto the big changes!
<h2 id="serialization-changes">Serialization changes</h2>There are two big changes related to how we now perform serialization within the NEST client.
<h3 id="user-defined-types">User defined types</h3>We now have an explicit path for user defined types in the client serialization pipeline.
In NEST 5.x, if you were to use a custom serializer with the client by implementing <code>IElasticsearchSerializer</code> and passing it to the client configuration, the custom serializer would take charge of serializing the entire request and response, including the underlying client types within NEST used to model requests and responses. Now in NEST 6.x, a custom serializer is used only in places where a user defined type is expected, for example,
<ul>
	<li>The <code>_source</code> and <code>fields</code> on <code>hits</code> in the response</li>
	<li>The term query <code>value</code></li>
	<li>A document sent as part of the Multi Term Vector API</li>
	<li>A document returned by the Get API</li>
	<li>A document sent through the Bulk API</li>
	<li>An upsert or document part of the Upsert API</li>
</ul>This change allows you to use your own serialization for your own user defined types without fear of affecting the rest of the client behaviour.
<h3 id="newtonsoft-json">Newtonsoft.Json</h3>NEST 6.x no longer depends on the popular <a href="https://www.nuget.org/packages/newtonsoft.json/" target="_blank">Newtonsoft.Json a.k.a. Json.NET library</a> directly as a nuget package reference. We&apos;ve taken the leap and internalized Newtonsoft.Json within the NEST assembly, and shaded the dependency by re-namespacing it to Nest.Json. This required some IL and assembly rewriting using <a href="https://github.com/gluck/il-repack" target="_blank">ILRepack</a> and <a href="http://www.mono-project.com/docs/tools+libraries/libraries/Mono.Cecil/" target="_blank">Mono.Cecil</a> to get it to work for both .NET Framework and .NET Core target platforms.
What this means in practice as a consumer, is that your application is now free to use whatever version of Newtonsoft.Json that you desire without conflicting with the version that NEST uses. What this means for us as client authors, is that we are now free to work on improving the client serialization pipeline over the course of NEST 6.x without having to wait until NEST 7.x. There are many exciting developments happening in the .NET ecosystem around performance with the likes of <a href="https://msdn.microsoft.com/en-us/magazine/mt814808.aspx" target="_blank"><code>Span&lt;T&gt;</code></a>, <a href="http://adamsitnik.com/Array-Pool/" target="_blank"><code>ArrayPool&lt;T&gt;</code></a> and active discussion around a more <a href="https://github.com/dotnet/coreclr/issues/7083" target="_blank">compact UTF-8 string representation</a>, and we would like to be able to explore these unhindered.
Note we have been indexing our client benchmarking CI test runs into an Elasticsearch cluster running in <a href="https://www.elastic.co/cloud" data-vivaldi-spatnav-clickable="1">Elastic Cloud</a> for a while, and will be looking to expose these results publicly in the future. We&apos;ll also look at creating an Elasticsearch <a href="https://github.com/dotnet/BenchmarkDotNet" target="_blank">BenchmarkDotNet</a> reporter, to allow anyone running benchmarking tests with BenchmarkDotNet to easily index results into Elasticsearch. For the time being however, our implementation is wired together in our <a href="https://github.com/elastic/elasticsearch-net/blob/master/build/scripts/Benchmarking.fsx" data-vivaldi-spatnav-clickable="1">FAKE build</a>.
<h2 id="caution-breaking-changes-ahead">Caution! Breaking changes ahead</h2><h3 id="single-types">Single types</h3>The <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html" data-vivaldi-spatnav-clickable="1">staged removal of types</a> means that in Elasticsearch 6.0 onwards, multiple types in a single index are no longer allowed. This will impact your NEST code in several ways.
As an example, if you are using NEST 5.x to index <code>Company</code> and <code>Employee</code> types in the following way:
<pre class="prettyprint">var settings = new ConnectionSettings()
 .DefaultIndex(&quot;my-index&quot;);
var client = new ElasticClient(settings);

client.IndexDocument(new Company { Id = 1 })
client.IndexDocument(new Employee { Id = 1 })
</pre>The documents would end up as different types within the same index in versions of Elasticsearch prior to 6.0
<pre class="prettyprint">/my-index/company/1
/my-index/employee/1
</pre>This will no longer work in Elasticsearch 6.0, and both <code>Company</code> and <code>Employee</code> types will now need to live in their own index. When upgrading to NEST 6.x, you can use the <a href="https://www.elastic.co/guide/en/elasticsearch/client/net-api/6.x/configuration-options.html#_options_on_connectionsettings" target="_blank"><code>DefaultMappingFor&lt;T&gt;()</code></a> method for each type so that documents are indexed into their own index. Rewriting the above example to take this approach would look as follows
<pre class="prettyprint">var settings = new ConnectionSettings()
 .DefaultIndex(&quot;my-index&quot;)
 .DefaultMappingFor&lt;Company&gt;(m =&gt; m.IndexName(&quot;company-index&quot;))
 .DefaultMappingFor&lt;Employee&gt;(m =&gt; m.IndexName(&quot;employee-index&quot;));
var client = new ElasticClient(settings);

client.IndexDocument(new Company { Id = 1 })
client.IndexDocument(new Employee { Id = 1 })
</pre>and results in documents ending up in their own index:
<pre class="prettyprint">/company-index/company/1
/employee-index/employee/1
</pre>If you really need multiple .NET types in a single index, you need to ensure that both .NET types are indexed using the same type name, which can be done using <code>DefaultTypeName()</code>
<pre class="prettyprint">var settings = new ConnectionSettings()
 .DefaultIndex(&quot;my-index&quot;)
 .DefaultTypeName(&quot;doc&quot;);
var client = new ElasticClient(settings); 

client.IndexDocument(new Company { Id = 1 });
client.IndexDocument(new Employee { Id = 1 });
</pre>However, in doing this, you will now face another problem! Both documents will end up being indexed with the same id
<pre class="prettyprint">/my-index/doc/1
/my-index/doc/1
</pre>with the last document indexed overwriting the previous. The solution here is to change your .NET classes to give the documents a unique id when invoking <code>client.IndexDocument&lt;T&gt;()</code>.
Another problem you&apos;ll encounter if you decide to index multiple .NET types into a single index is when you need to search. When searching in NEST 5.x using
<pre class="prettyprint">client.Search&lt;Company&gt;();
</pre>a search request would be made against the <code>company</code> type within the index
<pre class="prettyprint">/my-index/company/_search
</pre>and result in a collection of <code>Company</code> types being returned from the deserialized _source documents. Now that Elasticsearch 6.0 allows only a single type name within an index and in future, no type name at all, the previous search request will now translate to the following in NEST 6.x
<pre class="prettyprint">/my-index/doc/_search
</pre>which will now be searching across both <code>Company</code> and <code>Employee</code> types within the one index, since they are both indexed using the <code>doc</code> type name. If you are looking to constrain the search to only <code>Company</code> types, you can use a discriminator field within the document source, and apply a query on this field to the search criteria:
<pre class="prettyprint">client.Search&lt;Company&gt;(s =&gt; s
 .Query(q =&gt; +q
 .Term(&quot;my_type_field&quot;, &quot;company&quot;)
 )
);
</pre><h3 id="covariant-searches">Covariant search results</h3>NEST 5.x used to support returning covariant search results by inspecting the <code>_type</code> metadata for each hit, mapping each _type back to a .NET type automagically. For example
<pre class="prettyprint">client.Search&lt;IInterface&gt;(s =&gt; s.Types(typeof(A), typeof(B), typeof(C)));
</pre>would perform a search on <code>a</code>, <code>b</code> and <code>c</code> types within an index. NEST would then use the <code>_type</code> field in the response to deserialize each <code>_source</code> document into the the correct .NET type, and not the common interface <code>IInterface</code>. Since you can no longer have multiple types within the one index, this feature no longer makes sense in NEST 6.x.
It is still possible to achieve covariant search results going forward, but not without telling NEST 6.x how to do so. There is now a new nuget package, <a href="https://www.nuget.org/packages/NEST.JsonNetSerializer/" target="_blank"><code>Nest.JsonNetSerializer</code></a>, with a dependency on Newtonsoft.Json, that you can reference to configure Json.NET to use <a href="https://www.newtonsoft.com/json/help/html/SerializeTypeNameHandling.htm" target="_blank"><code>TypeNameHandling.All</code></a>, such that <code>_source</code> documents are deserialized into the .NET types you expect. You can of course also implement your own <code>IElasticsearchSerializer</code> implementation to use a discriminator field with the <code>_source</code> to control deserialization to your own user defined types instead.
<h3 id="parent-child-joins">Parent/Child joins</h3>If you rely on the parent and child mapping in your usage of Elasticsearch, you still need to, conceptually at least, index multiple types into a single Elasticsearch index. Elasticsearch 6.0 introduced the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.0/parent-join.html" target="_blank"><code>join</code> data type</a> as a mechanism to support parent/child relationships.
The NEST documentation has a <a href="https://www.elastic.co/guide/en/elasticsearch/client/net-api/6.x/parent-child-relationships.html" data-vivaldi-spatnav-clickable="1">full example</a> of how to use the <code>join</code> data type, illustrating the removal of the <code>_parent</code> mapping and request parameter. Be sure to have a thorough read of the documentation if you&apos;re using this feature.
<h3 id="mapping-attributes">Mapping attributes</h3>Since the Newtonsoft.Json package dependency has now been removed, NEST provides additional attributes you can use to decorate your user defined types to indicate how properties should be serialized when sent to Elasticsearch. This allows you to include and rename properties without having to reference <code>Nest.JsonNetSerializer</code> and provide your own serializer instance.
For example:
<pre class="prettyprint">[ElasticsearchType(Name = &quot;employee&quot;)]
public class Employee
{
 [Text(Name = &quot;first_name&quot;)]
 public string FirstName { get; set; }
 [Text(Name = &quot;last_name&quot;)]
 public string LastName { get; set; }
 [Ignore] // Property will not be included in serialization
 public bool IsSmith =&gt; !string.IsNullOrEmpty(LastName) &amp;&amp; LastName.Equals(&quot;smith&quot;, StringComparison.OrdinalIgnoreCase);
 [Number(DocValues = false, IgnoreMalformed = true, Coerce = true)]
 public int Salary { get; set; }
 [Date(Format = &quot;MMddyyyy&quot;)]
 public DateTime Birthday { get; set; }
 [Boolean(NullValue = false, Store = true)]
 public bool IsManager { get; set; }
 [Nested]
 [PropertyName(&quot;empl&quot;)] // serialize this property to a field named &quot;empl&quot; in JSON
 public List&lt;Employee&gt; Employees { get; set; }
}
</pre><h2 id="low-level-client">Low-Level Client</h2><h3 id="code-generation-fixes">Code generation improvements</h3>In generating parts of the low-level client, <a href="https://www.nuget.org/packages/Elasticsearch.Net/" target="_blank">Elasticsearch.Net</a>, we take the <a href="https://github.com/elastic/elasticsearch/tree/master/rest-api-spec" data-vivaldi-spatnav-clickable="1">Elasticsearch REST API specifications</a> and use it to generate C# source code. This code generation implementation has been mostly untouched since the days of NEST 1.x. With this release however, we&apos;ve taken the opportunity to fix several outstanding issues, one of which was that request parameters were fluent methods, meaning it was possible to write code such as
<pre class="prettyprint">client.Search&lt;dynamic&gt;(r =&gt; r.Parameter())
</pre>This was done in the beginning to facilitate the high-level client but now is no longer needed, thus has been removed. Instead of using fluent methods, one can use
<pre class="prettyprint">client.Search&lt;DynamicResponse&gt;(new OperationRequestParameters { Parameter = .. })
</pre><h3 id="response-types">Response types</h3>In previous versions of Elasticsearch.Net, any operation such as <code>client.Search&lt;T&gt;(..)</code> accepted any of <code>string</code>, <code>byte[]</code>, <code>dynamic</code>, <code>Stream</code>, <code>VoidResponse</code> or an <code>object</code> as the generic type parameter <code>T</code>, to deserialize the response from Elasticsearch into, and the return value of the method call was of type <code>ElasticsearchResponse&lt;T&gt;</code>.
In NEST 6.x, the generic type parameter <code>T</code> is now constrained to be of type <code>IElasticsearchResponse</code> and returns <code>T</code> instead of <code>ElasticsearchResponse&lt;T&gt;</code>. To help aid the transition for the common use cases, Elasticsearch.Net provides <code>BytesResponse</code>, <code>StringResponse</code>, <code>DynamicResponse</code>, <code>VoidResponse</code> and <code>ElasticsearchResponse&lt;T&gt;</code> where <code>T</code> can be any object to deserialize the response into.
As part of this change, support has been removed for returning the response from Elasticsearch as a <code>Stream</code>. This was a potentially dangerous feature because the onus for disposing this stream lay with the consumer, and the result of not disposing it could result in memory leaks and keeping connections open to Elasticsearch. If you do need to access the response stream directly for some reason, this is best handled by creating a custom <code>IElasticsearchSerializer</code> implementation.
<h3 id="exception-handling">Exceptions</h3>In previous versions, the low-level client was in charge of deserializing exceptions from the server, and would short-circuit this deserialization if a bad HTTP status code such as a 400 were returned from Elasticsearch. This approach was fine for Elasticsearch 2.x, but starting with Elasticsearch 5.x, several endpoints could also return additional information for bad HTTP status codes, and the client needed to perform a double-read of the response stream in order to surface this additional information to the consumer. How inefficient!
Starting with NEST 6.x, the response stream is now fed directly into the configured serializer for both good and bad HTTP status codes, for deserialization. No more double reads, yay! If you want to disable this behaviour, you can set <code>DisableDirectStreaming()</code> or <code>DebugMode()</code> on the instance of <code>ConnectionSettings</code> that is passed to instantiate the client. This will result in the request and response bytes becoming available on the response through the <code>response.DebugInformation</code> property.
The built-in <code>BytesResponse</code> and <code>StringResponse</code> types within Elasticsearch.Net both have a <code>TryGetServerError()</code> method to access an instance of <code>ServerError</code> from the exception that occurred on the server.
For the high-level client NEST, nothing changes with usage here; the <code>.ServerError</code> property is still available on all responses, but what this does change is that serializer used by NEST is now in charge of deserializing the server error within the response, and not the low-level client.
<h2 id="summary">Summary</h2>Quite a few changes to digest, but we hope you&apos;ll agree that these continue to make working with the client as smooth as possible. We&apos;d like to give a shout out to the community who contributed to this 6.0 GA release by finding and reporting issues, opening pull requests and generally helping to further the betterment of Elasticsearch with .NET. Your contributions are highly appreciated!
Please feel free to take NEST 6.0 and Elasticsearch.Net 6.0 for a spin, and <a href="https://github.com/elastic/elasticsearch-net/issues" target="_blank">let us know if you come across issues</a> &#x1F60A;
</body></html>

nest-elasticsearch-6-ga.jpg

nest-elasticsearch-6-ga-thumb.jpg

NEST and Elasticsearch.Net 6.0: Now GA!

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Stuart Cam

Надсилаючи цю форму ви погоджуєтеся з <a href="/legal/terms-of-use">Умовами надання послуг Elastic</a>. Ваші особисті дані будуть опрацьовані відповідно з <a href="/legal/privacy-statement">Положеннями про конфіденційність Elastic</a>.

Newsletter GDPR Text - Ukrainian

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Intentionally Blank

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

提交即表示您確認已詳閱並同意我們的<a href="/legal/elastic-cloud-account-terms" target="_blank">服務條款</a>，且同意 Elastic 使用您在上方提供的個人資料<a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">與您聯絡</a>，為您提供相關產品和服務的資訊。如需詳細說明或隨時想停止收到產品和服務資訊，請參閱 <a href="/legal/privacy-statement/" target="_blank">Elastic 的隱私聲明</a>。

Newsletter GDPR Text - TW

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.

GDPR Text + Google Cloud

Inviando la registrazione (1) esprimi il consenso ai <a href="/legal/terms-of-use">termini e condizioni di Elastic</a> e <a href="/legal/privacy-statement">all’informativa sulla privacy</a>, nonché (2) a ricevere saltuarie e-mail.

Newsletter GDPR Text - Italian

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Load more

Overview

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Author

Articles by Stuart Cam

Elastic Common Schema .NET library and integrations released

NEST and Elasticsearch.Net: upgrading your Elasticsearch server and codebase

Building real-time address search with the Australian G-NAF dataset

NEST and Elasticsearch.Net 7.0 now GA

Indexing documents with the NEST Elasticsearch .NET client

Task Management with the NEST Elasticsearch .NET Client

NEST and Elasticsearch.Net 6.0: Now GA!

Follow us

About us

Join us

Partners

Trust & Security

Investor relations

Excellence Awards