<h3>Introduction</h3>The discovery module in Elasticsearch is responsible for finding nodes that belong to a cluster and electing a master node. The Google Compute Engine (GCE) discovery plugin uses GCE APIs to perform automatic unicast discovery. In this post, we'll work through setting up a three node Elasticsearch cluster in Google Cloud using the GCE discovery plugin, and create a custom image that can be shared among different projects.<h4>Before we start</h4>To follow along with this blog, there are two prerequisites:<ul><li><a href="https://cloud.google.com/resource-manager/docs/creating-managing-projects">Google Cloud Platform project ID</a></li><li><a href="https://cloud.google.com/sdk/downloads">Install Google Cloud SDK</a></li></ul>After setting up the project ID and installing the SDK, we have a few more administrative items to do. First, set the default project:<pre class="prettyprint">gcloud config set project sherry-test-gce </pre>Then login to Google Cloud:<pre class="prettyprint">gcloud auth login </pre>You can also set the default region and zone at this time to omit the --zone and --region flags while using the gcloud tool. This is optional. Here's how:<pre class="prettyprint">gcloud config set compute/region us-west1-b </pre>To simplify our tasks ahead, we will add the default project and region and omit them from rest of the commands. Also, all commands here that start with gcloud are executed from the local machine and not on the GCE instances.<h3>A GCE instance with Elasticsearch and GCE discovery plugin</h3><h4>Setup Firewall Rules</h4>We will begin with creating a firewall rule so that the nodes in our Elasticsearch cluster can communicate with each other. Elasticsearch uses the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-transport.html#_tcp_transport">TCP transport module</a> for internal communications; the default port range for the transport module is between 9300 - 9400. In addition, we will open up port 9200 to expose Elasticsearch APIs over HTTP.<pre class="prettyprint">gcloud compute firewall-rules create elasticsearch --direction=INGRESS \ --priority=1000 --network=default --action=ALLOW \ --rules=tcp:9200,tcp:9300-9400 --source-ranges=0.0.0.0/0 </pre><h4>Create an instance using gcloud SDK</h4>Now we are ready to set up a GCE instance:<pre class="prettyprint">gcloud compute instances create sherry-instance-gce-discovery \ --machine-type=n1-standard-2 --subnet=default \ --min-cpu-platform=Automatic \ --tags=elasticsearch,http-server,https-server \ --image=ubuntu-1604-xenial-v20180306 \ --image-project=ubuntu-os-cloud \ --boot-disk-size=10GB --boot-disk-type=pd-standard \ --boot-disk-device-name=sherry-gce-discovery --scopes=compute-ro </pre>The tags argument includes the Elasticsearch firewall rule along with http and https, which are optional. Once this process is complete, you should see the name of the instance, the zone it is created in, its internal and external IP, and the status of the instance.A few things to note: <ol><li>The instance we created is called sherry-instance-gce-discovery</li><li>The instance is running ubuntu-16.04.</li><li>The instance is using --scopes=compute-ro. This enables the GCE discovery module to call required APIs.</li></ol>Now, that we've created the instance, we need to connect to it:<pre class="prettyprint">gcloud compute ssh sherry-instance-gce-discovery </pre>If this is the first time you are using gcloud ssh to access a GCE instance, it will create a ssh key for you.Besides using the <a href="https://cloud.google.com/pubsub/docs/quickstart-cli">gcloud command line tool</a>, you can connect to the instance using the <a href="https://console.cloud.google.com/">Google Cloud Web console</a> that is directly accessible from the browser. Another option is to use the external IP address of the instance:<pre class="prettyprint">ssh -i ~/.ssh/google_compute_engine 12.34.56.789 </pre><h4>Install Java</h4>We will need to install JDK 8 on our new instance:<pre class="prettyprint">sudo apt-get update sudo add-apt-repository ppa:webupd8team/java sudo apt-get update sudo apt-get install oracle-java8-installer # Setting Java 8 as the default (optional) sudo apt-get install oracle-java8-set-default # Verify Java version java -version </pre><h4>Install Elasticsearch and GCE discovery plugin</h4>Installing Elasticsearch is the next task. Please see <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.2/deb.html">install Elasticsearch with Debian Package</a> and <a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.2/rpm.html">install Elasticsearch with RPM</a> for installation details. In our case, we will choose an Ubuntu based image:<pre class="prettyprint">wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.3.deb sudo dpkg -i elasticsearch-6.2.3.deb </pre>Once Elasticsearch is ready, <a href="https://www.elastic.co/guide/en/elasticsearch/plugins/current/discovery-gce.html#discovery-gce">install the GCE discovery plugin</a>. By default, the elasticsearch-plugin command can be found at /usr/share/elasticsearch/bin for both deb and rpm distributions.<pre class="prettyprint">sudo /usr/share/elasticsearch/bin/elasticsearch-plugin install discovery-gce </pre>Now, add the following configurations to the /etc/elasticsearch/elasticsearch.yml file:<pre class="prettyprint"># By default, the cluster name is elasticsearch cluster.name: lannister # Use ${HOSTNAME} to set the node name dynamically node.name: ${HOSTNAME} # Using GCE internal hostname to bind the network services network.host: _gce:hostname_ # Set the minimum master eligible nodes to avoid the split brain issue discovery.zen.minimum_master_nodes: 2 # Specific GCE discovery plugin settings cloud.gce.project_id: sherry-test-gce cloud.gce.zone: us-west1-b discovery.zen.hosts_provider: gce </pre>All of the GCE discovery plugin settings here are required. Let's take a quick look at them:cloud.gce.project_id - The Google project ID that we are usingcloud.gce.zone - The GCE zone where the Elasticsearch nodes will livediscovery.zen.hosts_provider - We are using the GCE discovery mechanismIn the near future, we are planning to make the GCE discovery setup even easier by <a href="https://github.com/elastic/elasticsearch/issues/13618">auto-discovering some of these configurations</a>.<h4>Start Elasticsearch</h4>We are ready to start Elasticsearch and verify everything is working.<pre class="prettyprint">sudo /etc/init.d/elasticsearch start </pre>Verify the instance is working as expected by tailing the Elasticsearch cluster log to ensure Elasticsearch has started:<pre class="prettyprint">tail /var/log/elasticsearch/lannister.log </pre>Then use curl to connect to the node and verify the service is running:<pre class="prettyprint">curl ${HOSTNAME}:9200 </pre>Note, the above curl command can only run within the Google Cloud Platform. By default, nothing is opened to internet. Log off from the GCE instance as our work here is done.<h4>Create a GCE custom image</h4>We will use the instance that we built to make a custom image. We can share the image with other projects to create additional GCE instances with Elasticsearch and GCE discovery plugin installed. Although you can use the <a href="https://cloud.google.com/sdk/gcloud/reference/compute/images/export">gcloud compute images export command</a> to create a custom image from a virtual disk, we choose to use the manual method so we can clean up the files on the disk before packaging it.Using our local environment, create a storage bucket to hold the disk image from our instance:<pre class="prettyprint">gsutil mb -p sherry-test-gce -c regional -l us-west1 gs://sherry-test-bucket/ </pre>We will stop the instance sherry-instance-gce-discovery as well. This is an optional step but ensures the integrity of the disk content in the snapshot.<pre class="prettyprint">gcloud compute instances stop sherry-instance-gce-discovery </pre>Take a snapshot of the instance/disk:<pre class="prettyprint">gcloud compute disks snapshot sherry-instance-gce-discovery \ --snapshot-names=sherry-gce-discovery-snapshot </pre>Use the snapshot to build a new GCE disk that we will use to create an image later.<pre class="prettyprint">gcloud compute disks create sherry-gce-discovery-disk \ --source-snapshot sherry-gce-discovery-snapshot </pre>Make a temporary disk that is at least 50% larger than the original image disk size. We will store the raw image of the disk from the instance sherry-instance-gce-discovery and the compressed version that we will be creating on the temporary disk. Since our original disk is 10 GB, our temporary one will be 15 GB.<pre class="prettyprint">gcloud compute disks create sherry-gce-discovery-temp-disk --size 15GB </pre>We will create a temporary GCE instance with the storage-rw scope and attach both the disk from the snapshot we made earlier and the temporary disk.<pre class="prettyprint">gcloud compute instances create sherry-temp-instance --scopes storage-rw \ --disk name=sherry-gce-discovery-disk,device-name=sherry-gce-discovery-disk \ --disk name=sherry-gce-discovery-temp-disk,device-name=sherry-gce-discovery-temp-disk </pre>Connect to the new instance:<pre class="prettyprint">gcloud compute ssh sherry-temp-instance </pre>Once on the instance, list the disks and disk partitions available:<pre class="prettyprint">ls /dev/disk/by-id/ </pre>Now, format and mount the temporary disk: <pre class="prettyprint">sudo mkdir /mnt/tmp sudo mkfs.ext4 -F /dev/disk/by-id/google-sherry-gce-discovery-temp-disk sudo mount -o discard,defaults /dev/disk/by-id/google-sherry-gce-discovery-temp-disk /mnt/tmp </pre>Mount the image disk so we can modify and removed files:<pre class="prettyprint">sudo mkdir /mnt/image-disk sudo mount /dev/disk/by-id/google-sherry-gce-discovery-disk-part1 /mnt/image-disk </pre>Delete the Elasticsearch data directory. Be default, it is located at /mnt/image-disk/var/lib/elasticsearch/nodes. Each new instance will create its own Elasticsearch data directory when the Elasticsearch service starts the first time.Remove SSH key. The authorized_keys file is located in the /mnt/image-disk/home/[USER]/.ssh directory. This step is not required but highly recommended.Unmount the image disk:<pre class="prettyprint">sudo umount /mnt/image-disk/ </pre>Create a disk.raw from the image disk on the temporary disk. Please note, it must be called disk.raw.<pre class="prettyprint">sudo dd if=/dev/disk/by-id/google-sherry-gce-discovery-disk of=/mnt/tmp/disk.raw bs=4096 </pre> Create a tar file of the disk.raw file.<pre class="prettyprint">cd /mnt/tmp sudo tar czvf sherry-gce-discovery-disk.tar.gz disk.raw </pre>Copy the tar file to the bucket we created earlier from the GCE instance directly:<pre class="prettyprint">gsutil cp /mnt/tmp/sherry-gce-discovery-disk.tar.gz gs://sherry-test-bucket </pre>Log off the temporary instance after the file transfer is done.Finally, make a custom image that we can use to create new GCE instances:<pre class="prettyprint">gcloud compute images create sherry-gce-discovery-image \ --source-uri=https://storage.googleapis.com/sherry-test-bucket/sherry-gce-discovery-disk.tar.gz </pre><h4>Create instances based on our custom image</h4>We are ready to create three new instances from the images we just built.<pre class="prettyprint">gcloud compute instances create sherry-instance-1 \ --machine-type=n1-standard-2 --subnet=default \ --min-cpu-platform=Automatic --tags=elasticsearch,http-server,https-server \ --image=sherry-gce-discovery-image --image-project=sherry-test-gce \ --boot-disk-size=20GB --boot-disk-type=pd-standard \ --boot-disk-device-name=sherry-instance-1 --scopes=compute-ro gcloud compute instances create sherry-instance-2 \ --machine-type=n1-standard-2 --subnet=default \ --min-cpu-platform=Automatic --tags=elasticsearch,http-server,https-server \ --image=sherry-gce-discovery-image --image-project=sherry-test-gce \ --boot-disk-size=20GB --boot-disk-type=pd-standard \ --boot-disk-device-name=sherry-instance-2 --scopes=compute-ro gcloud compute instances create sherry-instance-3 \ --machine-type=n1-standard-2 --subnet=default \ --min-cpu-platform=Automatic --tags=elasticsearch,http-server,https-server \ --image=sherry-gce-discovery-image --image-project=sherry-test-gce \ --boot-disk-size=20GB --boot-disk-type=pd-standard \ --boot-disk-device-name=sherry-instance-3 --scopes=compute-ro </pre>Once completed, logon to each instance and start Elasticsearch. Please see <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html">Elasticsearch installation instructions</a> to configure Elasticsearch to start automatically. From the instance sherry-instance-1, we can run curl sherry-instance-1:9200/_cat/nodes. We should see all three nodes in our cluster listed in the output.<pre class="prettyprint">10.138.0.8 14 29 6 0.10 0.11 0.07 mdi * sherry-instance-1 10.138.0.9 17 29 5 0.14 0.17 0.08 mdi - sherry-instance-2 10.138.0.10 13 29 5 0.13 0.22 0.10 mdi - sherry-instance-3 </pre><h4>Conclusion</h4>In a future blog, we will install and configure X-Pack to ensure our cluster is secure and production ready and the Google Cloud Storage repository plugin to provide the snapshot and restore capability. If you prefer the convenience of a managed cluster, <a href="http://cloud.elastic.co/">Elastic Cloud</a>, Elastic's official hosted solution is available on GCP. It comes with all of the X-Pack features and automatic snapshots.

blueprint-small.jpg

blueprint-circuit-board_thumb.jpg

Getting Started with the GCE Discovery Plugin on Google Cloud

if-it-fits-i-sits-scaling-the-elastic-stack-to-fit-fullbleed-three.jpg

small_medium_large.jpeg

Small, Medium, or Large - Scaling Elasticsearch and Evolving the Elastic Stack to Fit

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

headshot-sherry-ger-300x300.jpg

Principal Solutions Architect

Sherry Ger

Надсилаючи цю форму ви погоджуєтеся з <a href="/legal/terms-of-use">Умовами надання послуг Elastic</a>. Ваші особисті дані будуть опрацьовані відповідно з <a href="/legal/privacy-statement">Положеннями про конфіденційність Elastic</a>.

Newsletter GDPR Text - Ukrainian

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Intentionally Blank

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

提交即表示您確認已詳閱並同意我們的<a href="/legal/elastic-cloud-account-terms" target="_blank">服務條款</a>，且同意 Elastic 使用您在上方提供的個人資料<a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">與您聯絡</a>，為您提供相關產品和服務的資訊。如需詳細說明或隨時想停止收到產品和服務資訊，請參閱 <a href="/legal/privacy-statement/" target="_blank">Elastic 的隱私聲明</a>。

Newsletter GDPR Text - TW

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.

GDPR Text + Google Cloud

Inviando la registrazione (1) esprimi il consenso ai <a href="/legal/terms-of-use">termini e condizioni di Elastic</a> e <a href="/legal/privacy-statement">all’informativa sulla privacy</a>, nonché (2) a ricevere saltuarie e-mail.

Newsletter GDPR Text - Italian

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Load more

Overview

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Author

Articles by Sherry Ger

Principal Solutions Architect, Elastic

Getting Started with the GCE Discovery Plugin on Google Cloud

Small, Medium, or Large - Scaling Elasticsearch and Evolving the Elastic Stack to Fit

Follow us

About us

Join us

Partners

Trust & Security

Investor relations

Excellence Awards