Monitor resources on private networks

Stack Serverless

To monitor resources on private networks you can either:

• Allow Elastic’s global managed infrastructure to access your private endpoints.
• Use Elastic Agent to create a Private Location.

Private Locations via Elastic Agent require only outbound connections from your network, while allowing Elastic’s global managed infrastructure to access a private endpoint requires inbound access, thus posing an additional risk that users must assess.

To give Elastic’s global managed infrastructure access to a private endpoint, use IP address filtering, HTTP authentication, or both.

To grant access via IP, use this list of egress IPs. The addresses and locations on this list may change, so automating updates to filtering rules is recommended. IP filtering alone will allow all users of Elastic’s global managed infrastructure access to your endpoints, if this is a concern consider adding additional protection via user/password authentication via a proxy like nginx.

Private Locations allow you to run monitors from your own premises. Before running a monitor on a Private Location, you’ll need to:

• Set up Fleet Server and Elastic Agent.
• Connect Fleet to the Elastic Stack and enroll an Elastic Agent in Fleet.
• Add a Private Location in the Synthetics UI.

Start by setting up Fleet Server and Elastic Agent:

• Set up Fleet Server: If you are using Elastic Cloud, Fleet Server will already be provided and you can skip this step. To learn more, refer to Set up Fleet Server.
• Create an agent policy: For more information on agent policies and creating them, refer to Elastic Agent policy.

After setting up Fleet, you’ll connect Fleet to the Elastic Stack or your Observability Serverless project and enroll an Elastic Agent in Fleet.

Elastic provides Docker images that you can use to run Fleet and an Elastic Agent more easily. For monitors running on Private Locations, you must use the elastic-agent-complete Docker image to create a self-hosted Elastic Agent node. The standard Elastic Cloud or self-hosted Elastic Agent will not work.

To pull the Docker image run:

docker pull docker.elastic.co/elastic-agent/elastic-agent-complete:9.1.4
		

docker pull docker.elastic.co/elastic-agent/elastic-agent-complete:<SPECIFIC.VERSION.NUMBER>
		

Then enroll and run an Elastic Agent. You’ll need an enrollment token and the URL of the Fleet Server. You can use the default enrollment token for your policy or create new policies and enrollment tokens as needed.

For more information on running Elastic Agent with Docker, refer to Run Elastic Agent in a container.

docker run \
  --env FLEET_ENROLL=1 \
  --env FLEET_URL={fleet_server_host_url} \
  --env FLEET_ENROLLMENT_TOKEN={enrollment_token} \
  --cap-add=NET_RAW \
  --cap-add=SETUID \
  --rm docker.elastic.co/elastic-agent/elastic-agent-complete:9.1.4
		

When the Elastic Agent is running you can add a new Private Location in the UI:

1. Find Synthetics in the global search field.
2. Go to Settings.
3. Go to the Private Locations tab.
4. Click Create location.
5. Give your new location a unique Location name.
6. Select the Agent policy you created above.
7. (Optional) In Tags select tags to assign to this location.
8. (Optional) In Spaces specify the spaces where this location will be available.
9. Click Save.

By default Private Locations are configured to allow two simultaneous browser tests, and an unlimited number of lightweight checks. These limits can be set via the environment variables SYNTHETICS_LIMIT_{{TYPE}}, where {{TYPE}} is one of BROWSER, HTTP, TCP, and ICMP for the container running the Elastic Agent docker image.

It is critical to allocate enough memory and CPU capacity to handle configured limits. Resource requirements will vary depending on simultaneous workload and monitor complexity:

For browser monitors: Start by allocating at least 2 GiB of memory and two cores per browser instance to ensure consistent performance and avoid out-of-memory errors. Then adjust as needed. For tcp, http, icmp: Much less memory is needed, start by allocating at least 512MiB of memory and two cores globally. While this will be enough to run a large number of lightweight monitors, it is recommended to track the resource usage and adjust accordingly.

Example: For a private location expected to run 2 concurrent browser monitors and 100 HTTP checks, the recommended allocation is 2 * (2 GiB + 2 vCPU) + (512 MiB + 2 vCPU) => 4,5 GiB + 6 vCPU.

• A single private location will not scale beyond 10,000 monitors. Exceeding this number will result in agent degradation and inconsistent execution, regardless of the resources allocated.
• Complex monitor configuration can disproportionately increase the private location policy size, leading to agent communication errors and degradation even if the limit mentioned above hasn't been reached.

If you're facing one of these scenarios, it is likely that the private location has grown too large and needs to be split into smaller locations, each alloted a portion of the original location monitors.

Now you can add monitors to your Private Location in the Synthetics UI or using the Elastic Synthetics library’s push method.