Introducing the new “wildcard” field type for searching for patterns within strings. With this new field type, we've wrapped up the best practices for efficiently indexing and searching within logs and security data.

blog-banner-search-bar-teal.png

blog-thumb-search-bar-teal.png

Find strings within strings faster with the new wildcard field

<html><head></head><body><p>In Elasticsearch 6.5, we added a new field type plugin:&#xA0;<a href="/guide/en/elasticsearch/plugins/6.5/mapper-annotated-text.html">mapper annotated text</a>.
</p><h2>The limitations of &#x201C;plain&#x201D; text</h2><p>The <a href="/guide/en/elasticsearch/reference/current/text.html">text</a> field type is familiar to most users of Elasticsearch. It is what we use to index content like the text of this document. Elasticsearch breaks a large free-text string into multiple smaller tokens (each token typically representing a single word). The tokens are then organized in an index so that we can efficiently search for these entities.
</p><p>There are a few problems with search using unstructured text:
</p><ol>
	<li>Many subjects are multiple words, e.g &#x201C;John F. Kennedy&#x201D;, but are indexed as independent tokens <code>john</code> <code>f</code> and <code>kennedy</code>. Searchers looking for John F Kennedy would have to use <a href="/guide/en/elasticsearch/reference/6.5/query-dsl-match-query-phrase.html">phrase queries</a> to ensure the tokens are grouped so you don&#x2019;t just match everyone called John.</li>
	<li>Single words often relate to many subjects e.g &#x201C;JFK&#x201D; in a document could refer to the <a href="https://en.wikipedia.org/wiki/John_F._Kennedy_International_Airport">airport</a>, the <a href="https://en.wikipedia.org/wiki/John_F._Kennedy">president</a> or <a href="https://en.wikipedia.org/wiki/JFK_(film)">a movie</a>. Searchers for the president would potentially mistakenly match the airport.</li>
	<li>A subject may have multiple aliases e.g. JFK or John F. Kennedy. A searcher would have to use OR expressions to match all of the aliases.</li>
</ol><p>These problems can be overcome if we search for &#x201C;things, not strings&#x201D; (aka <a href="https://rd.springer.com/book/10.1007/978-3-319-93935-3#about">Entity Oriented Search</a>). This approach requires that the subjects of interest in text are tagged with unique IDs.
</p><h2>What is &#x201C;annotated&#x201D; text?</h2><p>As internet users, we are already quite familiar with the use of annotations. This document contains several of them in the form of hyperlinks. In each case, the hyperlink was used to enrich selected words used in the plain text with a link to the definitive article on the subject they reference. Annotation values don&#x2019;t have to be URLs, but any form of string that uniquely identifies a referenced subject is typically the goal.
</p><p>Elasticsearch has a new field type specifically for holding annotated text and it is defined in your index mapping using the &#x201C;annotated_text&#x201D; field type:
</p><pre class="prettyprint">&quot;mappings&quot;: {
  &quot;_doc&quot;: {
    &quot;properties&quot;: {
      &quot;my_rich_text_field&quot;: {
        &quot;type&quot;: &quot;annotated_text&quot;,
        &quot;analyzer&quot;: &quot;my_analyzer&quot;
      }
    }
  }
}
</pre><p>Note that, like text fields, you can use an arbitrary choice of analyzer to tokenize the plain-text values as you wish, but any special annotation values are indexed untreated by your choice of analyzer.
</p><p>The content of annotated text fields uses a Markdown-like syntax to add annotation values to selected sections of text e.g.
</p><pre class="prettyprint">&quot;my_rich_text_field&quot; : &quot;Today [elastic](Elastic+Inc.) announced&quot;
</pre><p>In the above example, the potentially ambiguous text &#x201C;elastic&#x201D; has been enriched with an annotation for the full entity name &#x201C;Elastic Inc.&#x201D;. Note that annotation values must be <a href="https://www.w3schools.com/tags/ref_urlencode.asp">URL-encoded</a> so the space character in the name has been replaced with &#x201C;+&#x201D;. This value is decoded before insertion into the index (but not lowercased or subjected to any other form of analysis). The annotation is injected into the token stream at the same token position and character offset as the text to which it is anchored. We can see the results of this using the _analyze API:
</p><pre class="prettyprint">POST my_index/_analyze
{
  &quot;field&quot;: &quot;my_rich_text_field&quot;,
  &quot;text&quot;: &quot;Today [elastic](Elastic+Inc.) announced&quot;
}
</pre><p>The tokens produced are as follows:
</p><pre class="prettyprint">{
  &quot;tokens&quot;: [
    {
      &quot;token&quot;: &quot;today&quot;,
      &quot;start_offset&quot;: 0,
      &quot;end_offset&quot;: 5,
      &quot;type&quot;: &quot;&lt;ALPHANUM&gt;&quot;,
      &quot;position&quot;: 0
    },
    {
      &quot;token&quot;: &quot;Elastic Inc.&quot;,
      &quot;start_offset&quot;: 6,
      &quot;end_offset&quot;: 13,
      &quot;type&quot;: &quot;annotation&quot;,
      &quot;position&quot;: 1
    },
    {
      &quot;token&quot;: &quot;elastic&quot;,
      &quot;start_offset&quot;: 6,
      &quot;end_offset&quot;: 13,
      &quot;type&quot;: &quot;&lt;ALPHANUM&gt;&quot;,
      &quot;position&quot;: 1
    },
    {
      &quot;token&quot;: &quot;announced&quot;,
      &quot;start_offset&quot;: 14,
      &quot;end_offset&quot;: 23,
      &quot;type&quot;: &quot;&lt;ALPHANUM&gt;&quot;,
      &quot;position&quot;: 2
    }
  ]
}
</pre><p>Note that the &#x201C;Elastic Inc.&#x201D; token appears at exactly the same position and offset as the word &#x201C;elastic&#x201D; to which it was anchored. In this respect it behaves like a synonym, but unlike synonyms, we aren&#x2019;t employing a one-size-fits-all rule that says all uses of the word &#x201C;elastic&#x201D; must also be tagged as &#x201C;Elastic Inc.&#x201D;. An annotation is more focused, limiting the scope to this particular use of the word &#x201C;elastic&#x201D; in this particular document and sentence. A document that talks about &#x201C;elastic waistbands&#x201D; would not be similarly tagged.
</p><p>We aren&#x2019;t limited to tagging text with a single annotation token - we can inject multiple tokens at the same position using a &#x201C;&amp;&#x201D; character to separate values (in the same way URL parameters are separated):
</p><pre class="prettyprint">Today [elastic](Elastic+Inc.&amp;Search_Company) announced
</pre><h2>How can I use annotated text?</h2><p>Great question. Now that you have structured information woven into your unstructured text this opens up some interesting possibilities. You&#x2019;re now in the position to search for &#x201C;things not strings&#x201D;. By this we mean that regardless of the word strings used in the original text (and any ambiguities, aliases or typos that may entail) you can now search the text using unambiguous, normalized IDs for your subject of interest. <a href="https://en.wikipedia.org/wiki/Precision_and_recall#Precision">Precision</a> is increased because our search for elastic the company does not match documents talking about elastic waistbands and <a href="https://en.wikipedia.org/wiki/Precision_and_recall#Recall">recall</a> is increased because we also find those documents where the authors insisted on calling the company &#x201C;elasticsearch&#x201D;.
</p><p>Because we store position information of annotation tokens, we can:
</p><ol>
	<li>Highlight the text to show exactly where your subject was mentioned.</li>
	<li>Use <a href="/guide/en/elasticsearch/reference/6.5/span-queries.html">positional queries</a>, e.g. find documents where a &#x201C;Search_Company&#x201D; was mentioned next to the text &#x201C;announced&#x201D; or &#x201C;released&#x201D;.</li>
</ol><p>Let&#x2019;s look at examples of these in action:
</p><h3>Demo application: Highlighting things not strings</h3><p><br><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt857d1f3563b761a9/5c5837c7bfa899f60ba15d5b/highlighting-things.png" data-sys-asset-uid="blt857d1f3563b761a9" width="600">
</p><p>In this example application we are looking at Wikipedia content. The user is shown a bar chart from a structured keyword field &#x201C;person&#x201D; which holds just the IDs of articles about people. The user has clicked on the &#x201C;John F. Kennedy&#x201D; term to preview examples of where this value is used in the text of articles. Note that the highlighted text snippet is the string &#x201C;JFK&#x201D; but because it was annotated with the id &#x201C;John F. Kennedy&#x201D; it matches the search and is highlighted in yellow. This is a good example of what it means to &#x201C;match the thing not the string&#x201D;.
</p><p>Let&#x2019;s break down how this example application works under the covers. The relevant section of the mapping for this index looks like this:
</p><pre class="prettyprint">{
  &quot;wikipediaannotated&quot;: {
    &quot;mappings&quot;: {
      &quot;_doc&quot;: {
        &quot;properties&quot;: {
          &quot;person&quot;: {
            &quot;type&quot;: &quot;keyword&quot;,
          },
          &quot;text&quot;: {
            &quot;type&quot;: &quot;annotated_text&quot;
          }
        }
      }
    }
  }
}
</pre><p>The relevant parts of the example document look like this:
</p><pre class="prettyprint">{
    &quot;person&quot;: [&quot;Cornell Capa&quot;, &quot;John F. Kennedy&quot;],
    &quot;text&quot;: &quot;...including [JFK](John%20F.%20Kennedy) for ...&quot;    
}
</pre><p>Note that the client application has ensured the annotation tokens used in the &#x201C;text&#x201D; field are URL-encoded and included in the structured field &#x201C;person&#x201D;.
</p><p>When the user clicks on the bar chart the application runs the following query:
</p><pre class="prettyprint">{
    &quot;query&quot;: {
        &quot;term&quot;: {&quot;person&quot;: &quot;John F. Kennedy&quot;}
    },
    &quot;highlight&quot;: {
        &quot;fragment_size&quot;: 200,
        &quot;require_field_match&quot;: false,
        &quot;type&quot;: &quot;annotated&quot;,
        &quot;fields&quot;: { &quot;text&quot;: {}    }
    }
}
</pre><p>Note that:
</p><ol>
	<li>The highlighter used must be of type <a href="/guide/en/elasticsearch/plugins/6.5/mapper-annotated-text-highlighter.html">annotated</a>.</li>
	<li>The <a href="/guide/en/elasticsearch/reference/current/search-request-highlighting.html#highlighting-settings">require_field_match</a> parameter must be set to false so that our search term strictly targeting the <code>person</code> field can be relaxed to highlight tokens found in the <code>text</code> field.</li>
</ol><p>The highlighter response looks like this:
</p><pre class="prettyprint">&quot;highlight&quot;: {
        &quot;text&quot;: [&quot;.. including [JFK](_hit_term=John+F.+Kennedy&amp;John+F.+Kennedy) for...&quot;]
}
</pre><p>Note that the result includes a combination of the original annotations and search matches. The annotated highlighter doesn&#x2019;t use &lt;em&gt;...&lt;/em&gt; &#xA0;HTML style markup used by other highlighters but reuses the [..](..) Markdown syntax to highlight search hits.
</p><p>The example application chooses to display annotations with blue text and any search matches (which are prefixed with a &#x201C;_hit_term=&#x201D; annotation) are coloured with a yellow background.
</p><h3>Demo application: Finding things near strings</h3><p>The previous demo example illustrates finding examples of structured entities in the text and showing some of the textual context around where they were mentioned. This context may pique our interest, so we can take examples of the free-text words (&#x201C;photographs of&#x201D;) and look for any annotation values that tag politicians near these words. The resulting Wikipedia matches might look like this:
</p><p><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt89278ed8feedc0f7/5c5837cd65157fdb0b6c6531/things-near-strings.png" data-sys-asset-uid="blt89278ed8feedc0f7" alt="">
</p><p>There&#x2019;s two parts to making this work: &#xA0;
</p><ol>
	<li>Fixing the index to tag mentions of politicians</li>
	<li>Writing the query to find mentions of photographs of politicians</li>
</ol><h4>Indexing multi-value annotations</h4><p>Annotations don&#x2019;t have to be restricted to injecting one value. The annotations in our example text include two pieces of information:
</p><ol>
	<li>The specific wikipedia ID of an entity (eg &#x201C;Abraham Lincoln&#x201D;)</li>
	<li>The wikipedia&#x2019;s entity&#x2019;s more general entity type (&#x201C;OfficeHolder&#x201D;) originally obtained from the <a href="https://wiki.dbpedia.org/services-resources/ontology">DBPedia ontology</a></li>
</ol><p>In my dataset, all elected politicians are tagged with the &#x201C;OfficeHolder&#x201D; type from DBPedia so this can be used in queries or highlighting.
</p><p>Here&#x2019;s how an entity ID and its DBPedia type are encoded in this index:
</p><pre class="prettyprint">{
    &quot;text&quot;: &quot;..after [JFK](John%20F.%20Kennedy&amp;OfficeHolder)...&quot;    
}
</pre><p>Note the use of the &#x201C;&amp;&#x201D; character to inject both the &#x201C;John F. Kennedy&#x201D; and &#x201C;OfficeHolder&#x201D; annotations.
</p><h4>Running the proximity query</h4><p>You may be familiar with writing &#x201C;phrase&#x201D; queries using the Lucene <code>query_string</code> syntax. This is a simple way to ensure matches are on words near each other by enclosing the words in quotes e.g
</p><pre class="prettyprint">&quot;John Smith&quot;
</pre><p>Unfortunately we can&#x2019;t use this query syntax to run our example query because:
</p><ol>
	<li>It would try and lower-case the annotation value &#x201C;OfficeHolder&#x201D; and </li>
	<li>It won&#x2019;t let us search for photograph OR photographs</li>
</ol><p>We&#x2019;ll need to use the more sophisticated &#x201C;span&#x201D; query syntax which looks like this:
</p><pre class="prettyprint">{
  &quot;span_near&quot;: {
    &quot;slop&quot;: 1,
    &quot;in_order&quot;: true,
    &quot;clauses&quot;: [
      {&quot;span_term&quot;: {&quot;text&quot;: {&quot;value&quot;: &quot;photographs&quot;}}},
      {&quot;span_term&quot;: {&quot;text&quot;: {&quot;value&quot;: &quot;of&quot;}},
      {&quot;span_term&quot;: {&quot;text&quot;: {&quot;value&quot;: &quot;OfficeHolder&quot;}}}
    ]
  }
}
</pre><h2>Great, where can I get hold of annotated data?</h2><p>Broadly speaking there&#x2019;s two sources of this type of data:
</p><ol>
	<li>Human annotated content e.g. Wikipedia articles</li>
	<li>Machine-annotated content e.g outputs of NER (Named Entity Recognition) software</li>
</ol><h3>Working with Wikipedia </h3><p>The examples used in this blog were based on the human-authored Wikipedia content. The original content is XML format and using a <a href="https://mvnrepository.com/artifact/info.bliki.wiki/bliki-core/3.1.0">Wikimedia parser</a> it is possible to extract the hyperlink URLs and use them as annotation values around the text they cover. The DBPedia ontology can be used to add further annotations, e.g that the Wikipedia article with the ID &#x201C;John F. Kennedy&#x201D; is of the types &#x201C;OfficeHolder&#x201D; and &#x201C;Person&#x201D;. Each document that includes an annotation link to JFK can therefore have these links enriched with this extra type information. Once you have unique IDs like this it is possible to pull in all sorts of linked information.
</p><h3>Build an editor to assist annotation</h3><p>Organisations like the <a href="https://www.icij.org/">ICIJ</a> (the journalists behind Panama Papers) spend long hours manually resolving entities e.g. people or company names found in leaked documents. It&#x2019;s not hard to imagine building a GUI that supports the following tasks:
</p><ol>
	<li>Use the mouse to select what looks like a person name mentioned in plain text (OpenNLP tools could assist by suggesting name candidates)</li>
	<li>Run a free-text search for this string e.g. &#x201C;John Smith&#x201D; on your resolved entities store (possibly using a &#x201C;more like this&#x201D; style query on some context around the name)</li>
	<li>Review the highlighted entity matches and select the most likely entity id (e.g. entity #5323464) or create a new one if this looks like a new person name.</li>
	<li>Tag the plain text with an annotation of the chosen entity ID and update the document.</li>
</ol><h3>Working with Named Entity Recognition tools</h3><p>Commercial NER tools such as <a href="https://www.basistech.com/text-analytics/rosette/entity-extractor/">Rosette</a> or open source tools like <a href="https://opennlp.apache.org/">OpenNLP</a> can identify named entities like people, organisations and places in plain text. With some client-side coding these APIs can be used to create the specific annotated text format used in Elasticsearch.
</p><p>It&#x2019;s important to note that some entity extraction tools only spot &#x201C;entity references&#x201D; &#x2014; they can guess that a section of text is referring to a person&#x2019;s name e.g. &#x201C;John Smith&#x201D; but they don&#x2019;t resolve the entity reference to determine exactly which of many potential real-world John Smiths this might be. <a href="https://en.wikipedia.org/wiki/Entity_linking">Entity linking</a> is the attempt to provide a unique ID for an entity reference, overcoming:
</p><ol>
	<li>ambiguity (same name, different people), or </li>
	<li>aliases (same person, different names).</li>
</ol><p>When adding annotations to text it&#x2019;s important that your annotation values do not clash with regular strings that might exist in the text or your searches will have some irrelevant matches.
</p><p>If your entities are people names then the fact names contain whitespace to separate first and last names and use uppercase for the first characters means they are unlikely to clash with any text tokens produced by the standard Analyzers. However, if your annotation is a single word lowercased e.g. the company &#x201C;elastic&#x201D; then there is a potential for mismatch.
</p><p>Prefixing the annotation value with a type string and a character that is stripped by the text analysis pipeline is one way to ensure there is no overlap e.g prefer &#x201C;company: elastic&#x201D; rather than just &#x201C;elastic&#x201D;.
</p><h2>Conclusion</h2><p>We&#x2019;re pretty excited to see how this feature will be used, so go try it out. Let us know what cool new user interfaces you build or annotated datasets that work well. We&#x2019;re keen to get some feedback on the <a href="https://discuss.elastic.co/c/elasticsearch">discuss forums</a>.
</p></body></html>

annotate_banner.jpg

annotate_thumb.jpg

Search for Things (not Strings) with the Annotated Text Plugin

<p>This is a challenging time for media organizations that distribute content they don't author themselves. Increasingly, sites that serve social media, music, or video are paying closer attention to the messages they're delivering.
</p>
<p>While free speech is a widely cherished principle, in practice, most commercial organizations exercise some form of content review outlined in their own terms of service. These restrictions on content aren't motivated by an organisation's desire to act as some form of moral arbiter - often the pressures are external.
</p>
<p><img src="https://api.contentstack.io/v2/uploads/59f74a0724ec47e4039a7e74/download?uid=blt9d22674ac093d6e4" data-sys-asset-uid="blt9d22674ac093d6e4" alt="1.png">
</p>
<p>While search engines have historically helped surface content that's desirable, they can equally be applied to the challenge of identifying content that's undesirable, as we'll see in this post.
</p>
<p>Volume of content and the complexity of making judgement calls, however, can make this task more challenging. (Not to mention the issue of balancing false-positives and false-negatives.) There are a couple of approaches (which are not mutually exclusive) to do so:
</p>
<ul>
	<li><strong>Proactive:</strong> identify unacceptable content before site users complain</li>
	<li><strong>Reactive</strong>: waiting for users to report terms-of-service violations (but can we trust their judgements? More on this later ...)</li>
</ul>
<p>We'll explore both scenarios using real data and the Elastic Stack.
</p>
<h2>Proactively identifying content</h2>
<p>The good news is you don't necessarily need to adopt complex analysis of your text, audio, or video content to identify candidates for review. We can use the same "people who liked X also like Y…" techniques employed by recommendation algorithms. The difference is we are focusing on users with undesirable tastes rather than desirable ones.
</p>
<p>Let's start by looking at some real data (that we've anonymised for this post) in the form of user profiles from a music streaming service where each user profile contains their list of favourite artists. If we start with a single artist name (we'll refer to them here as fictitious "Band X") that's known to be associated with hate speech we can query the user profiles to see which other artists are favoured by these Band X fans. We can walk these connections simply by hitting the "+" button in X-Pack's Graph UI.
</p>
<p><img src="https://api.contentstack.io/v2/uploads/59f74a0e372f6dca08e21f47/download?uid=blt05a851492787b722" data-sys-asset-uid="blt05a851492787b722" alt="2.png">
</p>
<h4>First, a word on meaningful relationships</h4>
<p>Before we look at any connections, it's important to appreciate that the Graph API that underpins the UI uses some special relevance-ranking logic from our search engine heritage that is not found in typical graph databases. Let's turn this special feature off in the settings to see what problems these significance algorithms help avoid:
</p>
<p><img src="https://api.contentstack.io/v2/uploads/59f74a1493cf229e08b5fc6a/download?uid=blt570e57436f965f5d" data-sys-asset-uid="blt570e57436f965f5d" alt="3.png">
</p>
<p>With this feature turned off our top suggestions for the interests of Band X fans look like this:
</p>
<p><img src="https://api.contentstack.io/v2/uploads/59f74a18d6ee2098087c191b/download?uid=bltbf102bcbafd65f95" data-sys-asset-uid="bltbf102bcbafd65f95" alt="4.png">
</p>
<p>Our resulting graph shows a link between Band X and The Ramones, a popular group not known to be associated with hate speech. When we click on the link between Band X and the "ramones" suggestion, we can drill down into the stats of how many users like Band X and The Ramones using a Venn diagram. There are 97 Band X fans who like The Ramones, and while that may be a large number it is not significant — The Ramones are generally popular (just like The Beatles or <a href="http://thenoisychannel.com/2009/02/24/how-recommendation-engines-quash-diversity">Coldplay</a>) and a huge majority of their fans have no interest in hate speech.
</p>
<p>The Ramones are not relevant to this content exploration -- they are off-topic -- and should not appear in the top recommendations.
</p>
<p>It should be obvious from this example that popularity is not the same thing as significance.
</p>
<p>Let's turn the default significance setting back on:
</p>
<p><img src="https://api.contentstack.io/v2/uploads/59f74a1c994166bd03e775c4/download?uid=blt6950f96ef39fd41d" data-sys-asset-uid="blt6950f96ef39fd41d" alt="5.png">
</p>
<p>Now when we walk the top connections let's see what significant suggestions we find:
</p>
<p><img src="https://api.contentstack.io/v2/uploads/59f74a20aaf733de03ea682f/download?uid=bltfb18fa703240fc5b" data-sys-asset-uid="bltfb18fa703240fc5b" alt="6.png">
</p>
<p>The Ramones are no longer present and instead we find "Band Q". Only 41 of the Band X fans like Band Q — but there are only 53 Band Q fans in total. That's a huge overlap and suggests that the two bands are strongly associated even though they are less popular than The Ramones.
</p>
<p>"Band Q" is an anonymised name but you only need to see the real band names from this dataset to know that the significance algorithms are staying on topic while following the connections.
</p>
<p>This is the first hop in what could be a multi-step expansion. We started with a single known-bad, not knowing too much about hate content, and discovered some very strong leads. When we hit the "+" button in the UI again we are now asking a much better question: find people who like bands X OR Q OR Y OR Z and widen the net to identify more sources of hate-related content.
</p>
<p>Repeat as desired.
</p>
<p>This same iterative operation is possible using custom client code, multiple queries, and use of the significant_terms aggregation. As demonstrated in these examples, this is very simple using the Graph UI in X-Pack.
</p>
<h4>Dealing with outputs</h4>
<p>Using these techniques on real data, it is easy to uncover large amounts of questionable content that will need careful human review before blacklisting. That said, simple blacklisting is unlikely to be a scalable business solution due to sheer content volume, the difficulty of reviewing each case fairly, and dealing with appeals. The highest-scoring outputs of the algorithms can be manually reviewed and then blacklisted, but this may just be the tip of an iceberg.
</p>
<p>A useful coping mechanism might be to adopt a "greylist" policy for the large numbers of lower-scoring matches that staff don't have time to review. These greylist items could only be discoverable by those users searching specifically for that content but are not promoted or recommended in any "people also like…" type user recommendations, nor are they featured alongside advertisers' messages who may not value the brand association and <a href="http://uk.businessinsider.com/why-advertisers-are-pulling-spend-from-youtube-2017-3">pull their funding</a>. &nbsp;
</p>
<h2>Reactive content removal</h2>
<p>Media organisations can also choose to rely on their site users to report undesirable content using "report this" type buttons. Sadly, like the content on a site, takedown requests may not be as innocent as the site owner would wish. Groups of like-thinking users or bots can coordinate their requests to try to remove content which they find objectionable, but is not actually in violation of a site's terms of service.
</p>
<p>Site owners therefore need to review takedown requests and identify coordinated attempts to censor content. This activity is similar to analyzing review fraud in a marketplace where "shill" bots may be employed to artificially boost the reputation of a seller with positive feedback. However, in this use case, bad actors are trying to flood the review subject with negative feedback. The means of detecting artificial feedback is the same, though: coordinating actors typically share something in common that normal independent actors don't. This might be:
</p>
<ul>
	<li>A common IP address or user agent</li>
	<li>A common "hit list" of items being flagged</li>
	<li>A common phrase used in feedback</li>
	<li>The same time-of-day when logging requests</li>
	<li>The same site join-date.</li>
</ul>
<p>One-off sharing coincidences between independent actors are to be expected but high numbers of these coincidences between user accounts would indicate collusion.
</p>
<p>We can use the X-Pack Graph API to identify the coordinating actors and generate alerts which can be reviewed using the Graph UI. We can index takedown requests using a combination of these terms:
</p>
<ul>
	<li>UserId</li>
	<li>ContentId</li>
	<li>Context fields</li>
	<ul>
		<li>
		Review date + hour of day
		</li>
		<li>Ip address</li>
		<li>Town</li>
		<li>MD5 of description text</li>
		<li>Date user joined</li>
	</ul>
</ul>
<p>Using the Graph API and selections of the above terms as vertices in the graph, it is possible to walk the connections for each ContentId to pull together a summary of the users objecting to that content.
</p>
<p>Below is an example of review collusion:
</p>
<p><img src="https://api.contentstack.io/v2/uploads/59f74a2693cf229e08b5fc70/download?uid=blta971f546f528e4e8" data-sys-asset-uid="blta971f546f528e4e8" alt="7.png">
</p>
<p>The red vertices are reviewers of a service and the blue vertices are a token that describes some aspect of their reviews (in this case I chose date+hour of review but I could also use IP address, MD5s of review text or join date). The lines show which reviewers are associated with which review aspects. The cluster on the right shows a healthy, normal reviewer — his blue vertices are not shared by many other reviewers. The cluster on the left however shows an extraordinarily high number of coincidences — many reviewers just happening to synchronise their reviewing behaviours on repeated occasions over many months. This is likely a single actor using multiple fake accounts to game the system, and these accounts are candidates for being blocked. Equally, a network showing a healthy gathering of many mature reviewers with no evidence of collusion could score the subject content as a high-confidence takedown judgement and be prioritised for human review or automatically blacklisted given a suitable score threshold.
</p>
<p>A detailed example of setting up a review monitoring platform is here:
</p>
<div style="height: 478px; margin-bottom: 20px;">
	<iframe width="100%" height="478" src="//www.youtube.com/embed/liMhiiyQ9co?rel=0" class="video-iframe" frameborder="0" allowfullscreen="">
	</iframe>
</div>
<h3>Summary</h3>
<p>Companies can likely expect growing pressure to enforce their terms of service, and audience sizes and advertising revenues will depend on their ability to do so.
</p>
<p>We need tools that help staff quickly separate signal from a sea of noise to successfully face the hard challenge of reviewing and managing content. In these examples, we've seen reactive and proactive solutions to dealing with undesirable content using features from X-Pack.
</p>
<p>Try it for yourself today and get to grips with your own content: <a href="/start">https://www.elastic.co/start</a>
</p>

blog-no-hate-graph-fb.jpg

blog-no-hate-graph-thumb.jpg

Using the Elastic Stack and Graph to Tackle Toxic Content

<html><head></head><body><p>The new Elastic Graph capabilities allow you to analyse connections in data. Whether it is chasing down the tangled web of offshore financial arrangements in the Panama Papers or a high-level overview of click behaviour on a busy ecommerce website, Graph technology helps bring these relationships into focus.
</p>
<p>The Graph capability is bundled as part of the commercial X-Pack plugins for the Elastic Stack and includes a Kibana app and a new Elasticsearch API. In this first Graph blog we&apos;ll take a brief look at what the combination of the Kibana app and the API can offer.<br>
</p>
<h2>Forensic analysis: Panama Papers</h2>
<p><a href="https://panamapapers.icij.org/" target="_blank">The release of financial and legal records</a> from the offshore law firm Mossack Fonseca is one of the most explosive news stories of 2016. The records reveal that many politicians, members of royalty, the rich and their families are exploiting networks of shell companies established in secretive offshore tax regimes. Journalists and financial institutions are now intently focused on this data, but unravelling the connections can be both difficult and time consuming. The Kibana Graph app makes this process simple for anyone:
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4d6a58208ba076e2a623/download?uid=blt0f07da9e0ac0dc6b?uid=blt0f07da9e0ac0dc6b" data-sys-asset-uid="blt28617152f722a049" alt="panama-papers-graph-dashboard-relationships.jpg">
</p>
<p>Above we see the companies and individuals connected to Vladimir Putin&apos;s close friend, <a href="http://www.theguardian.com/news/2016/apr/03/panama-papers-money-hidden-offshore" target="_blank">Sergei Roldugin</a>. &#xA0;This picture was built up from a few simple steps:
</p>
<h2>Selecting the datasource</h2>
<p>Initially we select &quot;panama&quot; from our list of indices and then select one or more fields whose values we want to show in the diagram. Each field can be given an icon and colour for the &quot;vertices&quot; that will appear in the diagram.
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4d6ad8edd48f76938367/download?uid=blta22818e6b496f4d1?uid=blta22818e6b496f4d1" data-sys-asset-uid="blt9b34c8d10c408585" alt="panama-papers-selecting-datasource.jpg">
</p>
<h2>Running a search</h2>
<p>Now we can run a regular free-text search to match documents containing the name of Putin&apos;s friend, &quot;Roldugin&quot;.
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4d6a12798d8e381ffacd/download?uid=bltf24921afe4b5ad83?uid=bltf24921afe4b5ad83" data-sys-asset-uid="blte9f79b695ee90410" alt="panama-papers-running-a-search.jpg">
</p>
<p>The terms found in the matching documents are shown as a network &#x2014;&#xA0;each line representing one or more documents that connect a pair of terms.&#xA0;
</p>
<p>The journalists at the ICIJ who are curating the data have tried to give each real-world entity (person/company/address) a unique ID that is attached to every document that references them.&#xA0;
</p>
<p>Unfortunately people names and addresses can be awkward to match &#x2014;&#xA0;the journalists correctly identified three documents that are connected to Person entity 12180773 but we can see that there are two other people with similar names, but they&#xA0;have been assigned different identity codes. Equally there are two addresses that look similar but have been assigned different identity codes. In future blog posts, we will&#xA0;talk about using the Graph API for automated entity resolution.&#xA0;For now let&apos;s fix this manually with the grouping tool.
</p>
<h2>Grouping vertices</h2>
<p>Using the advanced mode tools we can select, then click the group button to merge vertices. This gives us a cleaner picture.
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4d6a0342dfd7382657e9/download?uid=blt2e6f3a71812dab8f?uid=blt2e6f3a71812dab8f" data-sys-asset-uid="bltcbbd37d09d5def64" alt="panama-papers-grouping-vertices.jpg">
</p>
<p>If we wanted, we could further group already grouped items e.g.&#xA0;merging people with multiple identities into single vertices and then merging those into company vertices.
</p>
<h2>&quot;Spidering out&quot;</h2>
<p>Now what if we wanted to see what else was connected with these entities? We can continue to explore the connections in the data using the &quot;+&quot; button on the toolbar to pull in other related entities.
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4d6a43e9adc538716001/download?uid=blta23d518f9fd9a049?uid=blta23d518f9fd9a049" data-sys-asset-uid="bltfda1e6934dbaeeff" alt="panama-papers-spidering-out.jpg">
</p>
<p>We can expand out the picture further by pressing &quot;+&quot; repeatedly and use selections to focus on expanding only certain areas of the graph. The undo and redo buttons are important parts of backing out of any uninteresting results. Additionally, delete and blacklist buttons allow control over which vertices are currently visible or can return. Snippets of example documents behind selected vertices can also be shown.
</p>
<p>If you&apos;d like to explore the Panama Papers data yourself, grab a copy of <a href="https://www.elastic.co/downloads/elasticsearch" target="_blank">Elasticsearch</a>, <a href="https://www.elastic.co/downloads/kibana" target="_blank">Kibana</a>, and the <a href="https://www.elastic.co/downloads/graph" target="_blank">Graph plugin</a> then follow the index setup at <a href="http://bit.ly/espanama" target="_blank">bit.ly/espanama</a>
</p>
<h2>Wisdom of crowds</h2>
<p>The Panama Papers are an example of a detailed &quot;forensic&quot; type investigation where each single document may represent a highly important connection.
</p>
<p>However, where the Elastic Graph technology can really shine is in its ability to summarise mass user behaviour such as the data found in click logs. Common phrases used to describe this form of analysis are mining &quot;collective intelligence&quot; or &quot;wisdom of crowds.&quot;&#xA0;In these scenarios, each document by itself is not particularly interesting but the emergent patterns from many user behaviours are &#x2014;&#xA0;they can be used to drive recommendations e.g. &quot;people who bought product X also tend to buy product Y.&quot;&#xA0;In these scenarios we need to avoid the one-off documents that make spurious connections and equally avoid overly-obvious associations like people who bought product X also bought milk (most people buy milk). With this in mind the default settings that control graph exploration are tuned for identifying only the most significant associations.
</p>
<p>Let&apos;s look at a recommendation use case using the <a href="http://mtg.upf.edu/node/1671" target="_blank">LastFM dataset</a>.&#xA0;If we build a user-centric index, we have a single document per listener which contains an array of the music artists they like. Let&apos;s query this index for the people who like &quot;Chopin&quot; to see what else they like.<br>
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4d6a58208ba076e2a62c/download?uid=blt3edb78a25587143f?uid=blt3edb78a25587143f" data-sys-asset-uid="bltea221de1470c02c8" alt="lastFM-wisdom-of-crowds.jpg">
</p>
<p>The classical artists returned are obviously strongly related - clicking on a line between two vertices shows us just how many listeners share these interests. Nearly half of all Mendelssohn listeners also like Chopin.
</p>
<p>The Graph API has identified <em>only the significant associations</em>. This is an important distinction from other graph exploration technologies.
</p>
<h2>Popular != Significant</h2>
<p>Let&apos;s see what happens if we open the settings tab and deliberately turn off the feature that looks for only significant connections.
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4d6a58208ba076e2a634/download?uid=bltba33a0df4f361b86?uid=bltba33a0df4f361b86" data-sys-asset-uid="blt39297222a56835bd" alt="graph-popular-equals-significant.jpg">
</p>
<p>With the &quot;significant links&quot; checkbox unchecked let&apos;s re-run the query for Chopin listeners - the results are very different.
</p>
<p><img src="https://api.contentstack.io/v2/assets/575e4d6a9e7a83165490e58e/download?uid=blt2c3bba7188423781?uid=blt2c3bba7188423781" data-sys-asset-uid="blt25af101d10c4ef5f" alt="lastFM-wisdom-of-crowds-new.jpg">
</p>
<p>Notice that the (globally) popular artists such as Radiohead and Coldplay have now crept into the results. Among the 5,721 Chopin fans, 1,843 of them like the Beatles. That&apos;s certainly a popular choice but what we call &quot;commonly common&quot; &#x2014;&#xA0;like people who buy milk. When the switch for &quot;significant links&quot; is turned on we tune out noise and focus on signal &#x2014;&#xA0;what we call the <a href="http://www.infoq.com/presentations/elasticsearch-revealing-uncommonly-common" target="_blank">&quot;uncommonly common&quot;</a>.  For those from an information theory background, the TF-IDF algorithm that has powered search engines for years is based on these same principles.
</p>
<p>By reusing these relevance ranking techniques we can stay &quot;on topic&quot; when exploring connections in data. This is an important distinction from graph databases which have no concept of relevance ranking and are typically forced to employ a strategy of just deleting popular items (see the problem of &quot;supernodes&quot;).
</p>
<p><em>Note: When performing detailed forensic work such as exploring the Panama Papers, it can help to keep &quot;significant links&quot; turned on to try avoid the super-popular companies but the &quot;certainty&quot; setting should be lowered from its default value of three to one. For wisdom-of-crowds type scenarios we want at least three documents to assert a link before we trust it whereas in forensics every document is potentially interesting.</em>
</p>
<h2>Summary</h2>
<p>Hopefully this blog has provided a quick taste of the two main usage modes for Graph:
</p>
<ol>
	<li>Forensics:&#xA0;every document is potentially interesting. Search is &quot;zoomed in&quot; on individual records and actors and no stone goes unturned.&#xA0;</li>
	<li>Wisdom of crowds:&#xA0;zoomed-out &quot;big picture&quot; overviews of mass behaviour. There are too many noisy connections to be drawn so the focus is on summarising only the most significant connections in the data.</li>
</ol>
<p>In future blog posts we&apos;ll look in more depth at specific use cases and get to grips with the Elasticsearch API used behind the scenes by the Kibana app.
</p>
<p>Stay tuned Graph fans...
</p></body></html>

graph-full-bleed.jpg

panama-papers-spidering-out_small.jpg

Using Elastic Graph + Kibana to Analyze Panama Papers

Significant Terms Aggregation

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

speaker-mark-harwood.jpg

Developer

Mark Harwood

<p>Надсилаючи цю форму ви погоджуєтеся з <a href="/legal/terms-of-use">Умовами надання послуг Elastic</a>. Ваші особисті дані будуть опрацьовані відповідно з <a href="/legal/privacy-statement">Положеннями про конфіденційність Elastic</a>.</p>

Newsletter GDPR Text - Ukrainian

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Communication Preferences - ONLY for /communication-preferences

<p>By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.</p>

Cloud Trial GDPR Text

Intentionally Blank

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Serverless GDPR Text

<p>提交即表示您確認已詳閱並同意我們的<a href="/legal/elastic-cloud-account-terms" target="_blank">服務條款</a>，且同意 Elastic 使用您在上方提供的個人資料<a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">與您聯絡</a>，為您提供相關產品和服務的資訊。如需詳細說明或隨時想停止收到產品和服務資訊，請參閱 <a href="/legal/privacy-statement/" target="_blank">Elastic 的隱私聲明</a>。</p>

Newsletter GDPR Text - TW

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p><p>By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.</p>

GDPR Text + Google Cloud

<p>Inviando la registrazione (1) esprimi il consenso ai <a href="/legal/terms-of-use">termini e condizioni di Elastic</a> e <a href="/legal/privacy-statement">all’informativa sulla privacy</a>, nonché (2) a ricevere saltuarie e-mail.</p>

Newsletter GDPR Text - Italian

<p>By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.</p>

Newsletter GDPR Text

Load more

Overview

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Author

Articles by Mark Harwood

Developer, Elastic

Find strings within strings faster with the new wildcard field

Search for Things (not Strings) with the Annotated Text Plugin

Using the Elastic Stack and Graph to Tackle Toxic Content

Using Elastic Graph + Kibana to Analyze Panama Papers

Significant Terms Aggregation

Follow us

About us

Join us

Partners

Trust & Security

Investor relations

Excellence Awards