We wanted to fix that! This blog is the story of publishing a plugin that puts Elastic's capabilities where developers are already working. We already had the elastic/agent-skills repository, and we’ve both blogged about and documented them, but an official Claude plugin makes it that much easier for our customers to use those skills.

Getting here was a journey. Along the way, we learned a lot, and wanted to share our experience. 

It took us three weeks from first commit to live. Claude docs got us 80% of the way, the other 20% was undocumented review criteria and a versioning model that cuts a new release on every commit. There was limited visibility on when, or where, the plugin would appear.

Here’s what we learned:

Agent Skills versus a Claude plugin: Why we needed both

As we worked our way through all the reference materials, we realized that our Agent Skills were only part of what a Claude plugin can include. Although Agent Skills are a generic open standard, adopted by major agent vendors and ecosystems, a Claude plugin can contain much more; however, some of it may not be as generically applicable across platforms. One example from Claude Code Docs:

This contains quite a bit more than skills, and we want room to grow into those.

So we decided that we needed a new repo for our Claude plugin. That way, we’d have a home where all Claude-specific features could live, separate from our generic Agent Skills. Thus was born the elastic/claude-plugin repository.

Submitting a plugin to the Claude directory

Claude has thorough documentation, and I won’t rehash it all here, but for reference, you may want to read:

• Publish to the directory

• Plugin pre-submission checklist

• Build your first plugin

We went through the above and made sure that we addressed everything on the checklist. But when I tried to submit the plugin at https://claude.ai/directory/manage, I got the following message:

I’m an engineer. An individual contributor. I have no business being an “owner” of an account for a software tool used by most Elastic employees. But the IT team that owns it wouldn’t be maintaining and supporting this external-facing plugin. 

This was our first bit of friction, but it also led to our first lesson, namely, if you have a contact at Anthropic, make use of them. Elastic had already established a partnership with Anthropic, so when we hit this issue, we simply messaged our point of contact, who quickly got back to us:

Once we had the right permissions, it was relatively straightforward to fill out the submission form:

Despite the simplicity, this is where we made our first two mistakes.

Choosing your plugin path: Why not the repository root

As the screenshot above shows, this is the folder where Anthropic will look for your .claude-plugin/plugin.json. It’s also the root of what Anthropic’s automated systems will be scanning. 

We started with the default repository root for our plugin path, but this was a mistake, because the files for our own internal development purposes were also at the repository root. They included the CLAUDE.md that our engineers could use to maintain the plugin, along with our continuous integration (CI) pipelines, tests, and package-lock.json.

Scanning those files can lead to unexpected holdups during verification:

We should note that the Anthropic team didn’t let this be a blocker and, after manual review, waived these warnings.

Claude plugin versioning: Why we stopped pointing at main

There’s no single “right” way to version or release software. But at Elastic, we often stick to the following conventions:

• Development is done on feature branches (sometimes in forks).

• Pull requests (PRs) target main.

• Versioning uses SemVer (MAJOR.MINOR.PATCH).

• Releases are cut from release branches (MAJOR.MINOR) and tagged (vMAJOR.MINOR.PATCH).

On my initial read of Claude’s plugin versioning documentation, I knew that we’d need a different system for this repo. We only have one live version at once, rather than maintaining multiple majors and minors. So we planned to just use main with a specific version in the plugin metadata. We’d merge PRs into main, as normal, and when ready to release, we’d bump that version and create a Git tag. Easy.

But that’s not how it works:

That initial approach actually led to manyv0.0.1 “versions” of the plugin, one for each merge commit.

With help from our Anthropic contacts (the astute reader will notice that this is a theme), we got the following clarification:

After realizing this, we changed our plugin to point at a dedicated stable branch. Now we can merge to main as normal but only merge main into stable when we’re ready to release a batch of changes and update our version number. This results in our flow looking like:

What happens after you submit a Claude plugin

You wait.

When we first submitted, we saw only the following:

While we were in the process of writing this blog, Anthropic made some changes to its UI, providing a bit more visibility. Now the submissions UI looks more like this:

Even so, the review of our first plugin submission took about a week, and that may have been faster than normal as a result of our direct line of contact with Anthropic solution architects and account executives (to stress again how valuable that relationship has been). We were given a rough estimate of nine days for average review time.

We couldn’t see it then, but we later learned that part of the holdup was that our submission raised a number of warnings to the Anthropic review team:

Policy warnings from the automated review

In retrospect, the extra visibility is nice; but, at the time, I might have ended up making a bunch of unnecessary changes. The Anthropic team ultimately waived all of these warnings. The takeaway here is to wait for a call to action from the reviewer, rather than jumping straight into trying to resolve every policy warning, even if they sound serious.

Warning

What triggered it

Resolution

Uses a credential from the user’s machine

Example curl commands with credentials templated as environment variables.

Waived after manual review. False positive.

No icon

The icon field was undocumented at submission time, and claude plugin validate --strict passed without it.

Waived. The field is now documented.

Contains a download-and-run command

Skill directives instructing an agent to call an API and interpret the response.

Waived after manual review.

Warning: Uses a credential from the user’s machine

The credential warnings pointed at skills that included example curl commands, where the credentials were templated with environment variable syntax. The Elastic CLI was specially designed to ensure that credentials were never exposed to an agentic caller, so we were initially quite alarmed by this report and relieved to discover that these were false positives.

Warning: No icon

The icon warning surprised us, too. At the time, there was no documentation for an icon field being an option as part of the manifest, and our CI had run claude plugin validate --strict ., so we knew that the manifest was valid without that field.

Today, the Plugin manifest: Fields reference does document icon. We take this as a sign that Anthropic is clearly iterating on the plugin experience. New fields show up, but Anthropic doesn’t retroactively block submissions made without them. 

Warning: Contains a download-and-run command

The download-and-run warning also sounded concerning at first, but on closer inspection, it seems to be taking issue with our skills having directives like the following, where API calls are suggested, as is instruction on interpreting the response:

**Verify Kibana connectivity.** Call `GET kbn:/api/status`. If the call fails, stop and surface the error — do not guess endpoints or credentials. Read `version.number` to confirm the cluster meets the 9.4+ requirement.

Surprise: Your Claude plugin is live

When Anthropic concluded its review, the plugin was immediately visible. This didn’t require an extra confirmation step from us, and there was no staging area. We also received no notification that it was live. This truly surprised us. Our plugin was published late on a Thursday, but I didn’t check on it on Friday. Only on the following Monday did we realize that it was live and visible. In our case, no harm done (the skills had been public for months already), but for anyone who’s not shipping something battle-hardened, be sure to do your testing before submitting. 

Where your Claude plugin actually appears

Publishing doesn’t get you into every Claude directory. There are at least four places where a developer might look for plugins:

Directory

Visible on publish?

How to get in

Claude app’s Setting > Plugins > Discover UI

Yes

Submit your plugin at https://claude.ai/directory/manage.

Claude Marketplace (https://claude.com/plugins, https://claude.com/marketplace)

No

Verification process done separately by Anthropic, after publishing.

anthropics/claude-plugins-community repo

No

Curation process done separately by Anthropic, after publishing.

anthropics/claude-plugins-official repo

No

Partner plugins can be listed through Anthropic contact.

When it’s initially published, your plugin only appears in the Anthropic directory in an unverified state, labeled as “Community.” And this is only if your license or account isn’t configured to exclude unverified plugins. This caused some initial confusion internally (our IT department has Claude Enterprise set up to hide “Community” plugins); we could only see our own plugin when we weren’t logged in with our work accounts.

To get into the other directories and marketplaces, the plugin needs to go through a separate “verification” or “marking” process. Once again, DMs with Anthropic partners proved very helpful:

What’s next for the Elastic Claude plugin

We're still working on getting the plugin verified for the web marketplace and on getting it installed for Elastic's own Claude org so our engineers can dogfood it. The plugin is young, and there are a lot of features we’d like to add. 

But if there's one thing I'd want you to take from this post, it's that the hardest parts of this process weren't technical. They were the gaps between what was expected and what actually happened, and every one of those gaps got closed by a conversation with a real person. The tools are getting smarter, but the communities around them still run on people helping people.

If you're working through publishing your own plugin and get stuck, reach out on our Elastic Stack Community Slack or find me on LinkedIn. And consider sharing what you’ve learned, too. See you in the marketplace.

What is the Kibana chrome?

The chrome is everything that frames an application in Kibana: the global header and the navigation, along with the header of the page that you’re currently on. Every application renders inside it. Because the chrome is shared, its APIs decide how consistent the whole product feels and how hard the next redesign will be.

What inconsistent page headers looked like to users

The clearest way to see the problem is to look at what users saw:

Take something as basic as a primary action, the one button a page most wants you to click, like Create index. Before this work, primary actions lived in three different places, depending on the page:

• In the old app header.

• In the page template header.

• Somewhere in the page body itself.

Common links had the same problem, with share, feedback, and documentation links appearing in separate spots with different styling from one app to the next. Breadcrumbs were misconfigured on many pages. For example, some showed a breadcrumb for each tab within a page, and some weren't clickable. Still others repeated the title of the page you were already on.

Every one of these was the output of a well-intentioned team using a flexible API in the way it’s usually used. Users had to relearn each page's layout, and every attempt to redesign the chrome had to account for hundreds of local variations.

Goals for the page header redesign

The goal of the redesign was simple; that is, pages should express what they need (for example, a title, badge, or primary action), and the shared shell should decide how it looks and where it goes. Common links and primary actions should get one home, everywhere. If it's the primary action, it's always in the same place, styled the same way. And consistency should stop being something that each of dozens of teams has to get right by hand, forever.

Getting there wasn't a restyling exercise; it required changing the contract between applications and the platform.

How typed props replaced EuiPageHeader's open React nodes

Each page header was easy to build in isolation, and the local choices piled up into visible differences across the product: inconsistent spacing, title styling, and action placement from one page to the next.

The fix, everywhere this showed up, was the same mechanism: stricter APIs. Instead of accepting arbitrary React nodes, a control declares what it means through typed props, and the shared shell decides how it looks and where it goes. That one change buys two things at once: consistency, and governance over what the shared surface is allowed to contain.

Take the page header. With EuiPageHeader, the API exposed layout settings and accepted React nodes for nearly every visible area:

Because pageTitle, description, and rightSideItems all accept arbitrary React nodes, every team filled them in differently. One page put a badge next to the title, while another styled its own pill. One team's actions were plain buttons in one order; the next team's were a different mix in another order, some collapsing into a menu and others not. Everyone used the same shell component, yet the headers looked and behaved like they came from different products. The shared component guaranteed the wrapper, not what went inside it.

The AppHeader API closes that off. The same header is expressed as typed props:

Now there’s only one way to express each part, so every page renders it the same. Badges live in their own typed collection, separate from the title. A description carries its text and, optionally, a "Learn more" URL. Actions declare whether they’re primary or secondary, and the component decides how they look and when they collapse. Plus, richer controls fit the same shape. An editable title or a favorite toggle is a structured config rather than a bespoke React tree:

The split is consistent throughout; the application supplies state and behavior (the current name, what happens on rename, how to toggle a favorite) and the header owns presentation. It renders the controls, handles keyboard interaction, shows validation errors, and reflects pending changes. Because those roles are explicit, layout and responsive behavior stay inside the shared component, and applications don't need a coordinated update every time the header changes.

Why a shared UI shell needs a closed set of controls

The global shell had a similar problem that was one step more open-ended. Any plugin could register arbitrary content on the left or right and pick its position with a number, and no conversation with the platform team or designers was required:

That registerRight API is an open door. Any team can add a control, and the header fills up with elements no one designed together. The controls compete for space and carry their own styling, and their order is decided by whoever picked the larger number. The shell only knows that one React node follows another; it can't tell that one opens an AI assistant and another collects feedback, so it can't reason about them or keep them coherent.

The new shell replaces that open canvas with a closed set of named roles under chrome.controls and chrome.help. The old registry hasn't gone anywhere yet, because the classic header still runs alongside the new one while applications migrate, but nothing in the new shell is reachable through it:

Global chrome header versus application page header

Part of what made the old chrome hard to evolve was that "the header" was really several things tangled together. The redesign draws a hard line between two surfaces with different owners:

Chrome (global) header

Application header

Owner

The platform

The page

Scope

What's true everywhere in Kibana

What's true on this page

Contents

Navigation, project or deployment picker, search, help, AI assistant, feedback

Title, badges, description, tabs, page actions

How it's populated

Named slots under chrome.controls and chrome.help

AppHeader typed props, rendered directly or set via chrome.appHeader.set()

Because each surface has one owner and a typed contract between them, the platform can redesign the chrome without auditing hundreds of pages, and an application can evolve its header without colliding with global controls. Setting a header config returns a cleanup callback, so an application tears down its own contribution when it unmounts.

Design decisions that strict APIs force

Stricter APIs forced design decisions that flexible APIs had let everyone defer. Two are worth calling out.

Why we removed breadcrumbs from Kibana's navigation

Removing breadcrumbs from the chrome was one of the more debated calls, and the state of the data made it easier than expected. In practice, breadcrumbs across Kibana were widely misconfigured:

• Some pages generated a breadcrumb for every tab on the page.

• Some breadcrumbs weren't clickable.

• Some repeated the title of the current page.

They added visual weight without reliably adding orientation.

At the same time, the redesigned navigation already conveys hierarchy. The primary and secondary navigation show you where you are and give you a direct way back up. Keeping breadcrumbs would have meant maintaining a third (and frequently wrong) representation of the same information, so we stopped drawing the trail and let the navigation do that job.

The breadcrumb API itself didn't disappear. Applications still register breadcrumbs, and the new shell reads them to derive the back button and to fall back to a page title when an app hasn't supplied one. The data changed jobs rather than going away, which is why migrating an app to the new header rarely starts with ripping breadcrumbs out:

Before:

After:

How many action buttons a page header should show

The other recurring debate was priority. With every page's actions now flowing through one menu structure, which buttons deserve to be visible, and in what order? Priorities shift as products evolve, so this conversation is ongoing.

For launch, we made one deliberate simplification: we locked the number of buttons that can appear in the app menu. A page gets one primary action and a bounded set of secondary actions before the rest collapse into a menu. The constraint keeps any single page from dividing the user's attention across a wall of buttons, and it makes the priority conversation explicit instead of letting it be settled by whoever adds the next button.

We went through several iterations to land here. Initially, we had many visible actions. A page could have up to three buttons, plus a secondary action to the left of the primary one. That took up too much space, so we gradually simplified the layout by removing the secondary action and reducing the number of visible buttons. We also made the overflow menu more structured. Some items, such as feedback and documentation, now have a fixed place in the footer of the overflow menu.

What strict design system governance costs teams

Under the old API, a plugin could add a novel UI by choosing a side, picking an order, and mounting a React tree. Under the stricter API, a new kind of control may need a shared capability before a plugin can add it. That takes more work up front and forces a design decision that teams could previously avoid.

We accepted the cost of stricter APIs because the alternative pushed it into every later redesign. As long as arbitrary React trees mounted at arbitrary points, every change to the shell's layout or accessibility had to account for all of them.

That’s the trade we made, and it pays back on both fronts. Consistency stops being something each team has to get right by hand; there’s one way to express a control, so pages match by default. And the shared surface stays governed. Its set of controls is a deliberate design decision, not whatever accumulated at the edges. Applications describe what their controls mean, and the shell is free to decide how they look, both today and in the next redesign.

Where the redesigned Kibana chrome is available

The redesigned chrome is available now in Elastic Cloud Serverless. Open any project, and you're already using it. For Elastic Cloud Hosted and self-managed users, it ships in 9.6.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.

Start a free Elastic Cloud Serverless trial or log in to Elastic Cloud to follow along, and install the CLI via npm: 

Designing the Elastic CLI for people and AI agents

A useful side effect of building flexible tools for developers is that they’re more useful to AI agents, too. It’s also the tool that our Agent Skills now use to get things done, closing the loop that we opened in March 2026, when we said that a CLI for agent workflows was coming.

The CLI gives every public API across Elasticsearch, Kibana, and Elastic Cloud one shape: the same flags, input and output conventions, authentication method, and failure mode. Consistency is the ergonomic feature; everything else is built on it.

Agents need the same thing, only stricter. An agent won’t know how to craft a valid CLI command or notice if a tool “feels” wrong; it needs output it can parse and input it can validate before sending, along with failures it can branch on. Agents are now a first-class interface to Elastic, alongside people, whether they live on the platform or in your editor and terminal. Agent Skills, and now the CLI, are how we serve the second kind, so those needs are built into the core of the CLI rather than tacked on.

JSON input and output for every command

Agents love structured text, and almost every Elastic API already speaks JSON, so first-class JSON support was a hard requirement. Developers who are quick with a jq query will be equally satisfied.

• JSON output: Any command, like elastic version and elastic es indices delete ..., supports --json, which prints JSON-parseable output to stdout and nothing else. Failed commands print {"error": {"code": "...", "message": "..."}} to stderr.

• JSON input: Every command that takes input accepts JSON on stdin or via --input-file. Every top-level key in that JSON also works as a CLI argument, and inline arguments take precedence, so you can keep a big request body in a file and tweak a value or two per invocation.

• JSON Schema as --help output: Pass --help --json to any command, and it prints a valid JSON Schema for its input, which also feeds nicely into codegen tools. elastic cli-schema prints the whole command tree.

Exit codes that an AI agent can branch on

Agents loop on exit codes as much as they do over stdout. All failure modes are distinguishable from success, even if stdout and stderr are never read.

Safety rails: Keychain storage, allow lists, and validation

No model uses tools perfectly 100% of the time, so an agent-friendly CLI should provide safety rails wherever possible.

• Contexts and secret storage: Connection details live in named contexts in ~/.elasticrc.yml, kubectl-style; switch with --use-context. API commands never take credentials as flags. elastic config context add writes API keys to your OS keychain (macOS, Linux, Windows) and leaves a $(keychain:...) reference in the YAML; $(env:...), $(cmd:...), and $(file:...) work, too. Creating a Serverless project with --save-as writes its credentials straight to the keychain and never prints them, so nothing leaks into logs or LLM transcripts.

• Allowlists/blocklists: A commands.allowed (or commands.blocked) list in the config file, globally or per context, ensures that only the commands an administrator wants are runnable.

• Validation: Every command has a JSON Schema, so inputs are validated before any request is sent. Add --dry-run to any command that takes input, and it validates and exits without sending anything.

• Confirmation: Destructive commands prompt in a terminal. In a noninteractive session, where agents live, they refuse to run without --yes and say so in a structured error.

• Sanitization: Index, field, and pipeline names have length limits and forbidden characters. elastic sanitize index-name '<value>' (and field-name, pipeline-name, …) prints a version stripped of anything invalid.

Keeping API responses inside an agent's context window

Elastic APIs return a lot of data, and an agent’s context window is finite. Three controls help keep unnecessary text out of the context window:

• Field masks: --output-fields takes a comma-separated list, with dot notation for nested fields.

• String templates: For total control, --output-template takes a mustache-style template.

• Command profiles: --command-profile serverless (or default_profile: serverless in your config) hides Elastic Cloud Hosted commands and the Elasticsearch namespaces that don’t exist on Serverless. That means less to scroll past and less for an agent to guess wrong. It’s the profile we recommend for agents.

Control

What it does

Syntax

When to use

Field mask

Returns only the fields you name, using dot notation for nested fields

--output-fields 'name,version.number'

You want valid JSON back, just less of it. This is the default choice for agents parsing structured output.

String template

Renders the response through a mustache-style template

--output-template 'ES version: {{ version.number }}'

You need one value in a specific shape, for a shell variable, a log line, or a prompt.

Command profile

Hides commands and namespaces that don't apply to your deployment

--command-profile serverlessor default_profile: serverless

You want a smaller command surface so an agent has less to scroll past and less to guess wrong. This is recommended for agents.

Helpers for bulk ingest, scroll search, and msearch

Some of Elasticsearch’s most popular APIs have a learning curve, so elastic es helpers wraps them:

• scroll-search: Stream a large result set as NDJSON with paging handled for you.

• bulk-ingest: Ingest from a file, a directory, or stdin (NDJSON, JSON arrays, or CSV) with streaming, batching, concurrency, and retries.

• msearch: Send multiple searches in one request.

• watch: Print new documents from an index to stdout as they’re indexed. This is great for piping into logging tools.

elastic es and elastic kb are aliases for elastic stack elasticsearch and elastic stack kibana. If we don’t ship a command you need, elastic extension create scaffolds one for you.

Searching Elastic docs from the terminal

If you or your agents don’t know which API to use, elastic docs search (plus docs read and docs ask) searches Elastic’s documentation from the terminal, returning Markdown or --json. These are experimental. You’ll see a warning until you pass --accept-experimental, so explore, but don’t script against them yet.

Shell completion for Bash, Zsh, and Fish

Autocomplete hooks are available for Bash, Zsh, and Fish, and they always respect your commands.allowed or commands.blocked policy.

How Elastic Agent Skills use the CLI

Agent Skills teach an AI coding agent how an Elastic expert approaches a job; for example, which cluster health field is the verdict or how to stage a reindex so it doesn’t fall over. They capture process and judgment but not transport. A skill that embeds curl with an auth header has hard-coded a hostname, key, and runtime, and it breaks when any of those change.

So our skills now use a universal format that runs unchanged in any runtime that can execute the elastic CLI, including Claude Code, Codex, Cursor, and GitHub Copilot. The body refers to operations in HTTP shorthand (GET /_cluster/health, POST /_query), and an operations table at the end binds each to a CLI command. That table is the only place transport appears:

HTTP API (shorthand)

elastic CLI command

GET /{index}/_mapping

elastic es indices get-mapping --index '<index>'

POST /_query

elastic es esql query --format tsv --query "<esql>"

POST cloud:/api/v1/serverless/projects/elasticsearch

elastic cloud serverless projects search create --input-file <json> --wait --save-as <ctx>

Every universal skill also inherits a blunt preamble; that is, use the CLI, don’t guess credentials, don’t call the HTTP API directly, and never ask the user to paste an API key into the chat.

The two halves need each other. The skill supplies the expertise that the model doesn’t have, and the CLI supplies a way to act on it that’s validated, credential-safe, and scoped by your allowlist. Tell your agent to spin up a Serverless project and load products.csv into it, and the provisioning skill creates it with --save-as. The ingest skill dry-runs a mapping and loads with elastic es bulk, and the Elasticsearch Query Language (ES|QL) skill writes a query that parses on the first try. Every step returns JSON, exits non-zero on failure, and can only do what your policy allows.

Skills for Elastic Cloud onboarding and provisioning, Elastic Workflows, and Kubernetes investigation are available today. Skills for Elasticsearch query, ingest, reindex, and index design, plus Kibana dashboards and alerting, are close behind.

What’s in the Elastic CLI technical preview, and what’s next

The preview covers all public Elasticsearch Serverless, Kibana Serverless, and Elastic Cloud APIs. Hosted-only 9.x Elasticsearch API coverage is nearly 100%, and hosted-only 9.x Kibana APIs will be added soon.

We’re actively planning more developer experience work, including broader coverage for all supported stack releases, more helpers for common workflows, more skills in the public catalog, and loading the same skills into agents that run on the Elastic platform itself. What shapes that list is hearing how you and your agents use the CLI. Tell us what’s awkward and what’s missing, along with what you’d automate next.

Install the Elastic CLI and Agent Skills

The Elastic CLI is available now on npm (Node.js 22+). Install it and the skills together:

Then, add a context and check it:

Even without a project, you can start a free Serverless trial in about a minute, with no credit card. If you already have a project, log in and create API keys for Elastic Cloud and your Elasticsearch clusters. Before pointing an agent at anything real, start with a trial project, a read-only key, and a scoped commands.allowed list. Be sure to take five minutes to read the security notes in the skills repo.

Replace all those curl commands in your Bash scripts, and add some usage instructions to your AGENTS.md. Then let your agent’s skills work efficiently and accurately with our APIs. Let us know what you think, and don’t hesitate to open an issue if you find a bug or if your use case isn’t well supported. Your feedback directly shapes what we build next.

Elastic CLI and Agent Skills resources

• Elastic CLI on GitHub and CLI documentation

• Elastic Agent Skills on GitHub

• agentskills.io specification

• Elastic Cloud Serverless documentation

• Report a CLI issue · Report a skills issue · Discuss

Why upgrade to Redux Toolkit v2?

RTK v2 was released in late 2023. That's nearly three years of running on a major version behind in one of the most widely used state management libraries in the JavaScript ecosystem. It reflects how hard this upgrade is in a codebase of Kibana's size. A previous attempt tried the big-bang approach and stalled when the real scope became clearer. 

So what does v2 actually bring? It ships alongside Redux core 5.0, React-Redux 9.0, Reselect 5.0, and Redux Thunk 3.0. React-Redux 9.0 requires React 18 and drops the useSyncExternalStore shim that v8 carried for React 16/17. Since Kibana already runs React 18, upgrading sheds legacy compatibility code and keeps Kibana on the actively maintained Redux majors.

RTK v2 also brings genuinely useful new features, including inline selectors in createSlice and opt-in inline async thunks through a customized buildCreateSlice setup, along with a combineSlices API with slice reducer injection for code splitting. That last one is particularly interesting for Kibana's plugin architecture where lazy-loading is the norm.

How Redux is used across the Kibana monorepo

Before diving into the solution, it's worth understanding just how varied Redux usage is across Kibana. A full audit of the codebase (tracked in #239863) revealed several distinct camps:

Pattern

Plugins and packages

What the migration needs

Redux Toolkit v1

Discover, Lens, Synthetics, Security Solution

Full v1 to v2 migration

Plain Redux v4

Canvas, Maps, Index Management, Cross-Cluster Replication

redux-v4 alias only, no RTK migration

Kea

Enterprise Search (150+ files), Content Connectors

react-redux-v7 alias, no RTK migration

redux-saga

Synthetics, Graph, Uptime

Store setup only, saga is version-independent

typescript-fsa

Security Solution data-table package

Out of scope

Types and single imports

Expressions, Monitoring

Alias swap only

Plugins such as Discover, Lens, Synthetics, and Security Solution use RTK v1 APIs,including createSlice, configureStore, createAsyncThunk, and createSelector. These are the ones that actually need the v1 to v2 migration. But even here, complexity varies wildly. Lens uses stand-alone getDefaultMiddleware (removed in v2) and PreloadedState (also removed). Security Solution is the largest consumer at 300+ files, mixing modern RTK with legacy plain Redux patterns.

Canvas, Maps, Index Management, Cross-Cluster Replication, and several others still use plain Redux v4via createStore, combineReducers, applyMiddleware, and connect, which are classic patterns from the pre-RTK era. These don't need RTK migration at all since they're not using it  in the first place, but they do need the redux-v4 alias since the default redux package is now v5.

Enterprise Search and Content Connectors use kea, a Redux abstraction layer with its own logic builders (kea(), useValues, useActions). There are more than 150 files in Enterprise Search alone. RTK migration isn't applicable here, since kea is its own world. But it does depend on react-redux v7 under the hood, which is where the bundler tricks come in.

Synthetics, Graph, and Uptime use redux-saga for side effects. Saga integration is actually independent of the RTK version, but these plugins need their store setup migrated.

The Security Solution data-table package uses typescript-fsa and typescript-fsa-reducers instead of RTK entirely, with its reducer embedded into Security Solution's main store, and isn’t part of the RTK migration at all.

The Expressions plugin only imports shallowEqual from react-redux, and Monitoring only imports types. These just need an alias swap.

Asking every team to migrate simultaneously was a nonstarter. The breaking changes in RTK v2 include stricter type checking and removed APIs, like enableES5() from immer, getDefaultMiddleware and PreloadedState gone entirely, AnyAction replaced by UnknownAction, and behavioral changes in how middleware is configured.

Running Redux Toolkit v1 and v2 side by side

The solution was to flip the typical migration pattern on its head. Instead of keeping the default imports on v1 and introducing v2 under aliases, the default package names (for example, @reduxjs/toolkit, react-redux, and redux) now point to v2. The old versions live under versioned aliases:

• redux-toolkit-v1

• react-redux-v7

• redux-v4

• immer-v9

• reselect-v4

• redux-thunk-v2

This is npm's alias syntax. "react-redux-v7": "npm:react-redux@7.2.8" installs the old version under a different name. Both versions coexist in node_modules without conflicts.

The insight here is that all existing code in this pull request (PR) was moved to v1 aliases. Every import { useSelector } from 'react-redux' became import { useSelector } from 'react-redux-v7'. That's ~1,100 files touched, but the vast majority (~1,000) are mechanical one-liner import swaps. When a team is ready to migrate to v2, they switch back to the default import names. Once all v1 aliases disappear from the codebase, the old packages can be removed entirely.

This avoids the alternative, where v2 imports would end up under nonstandard names permanently, leaving nonstandard imports in the codebase for the long term.

Serving both versions through the bundler

Getting two versions of the same library to coexist at runtime is where things got interesting. Kibana uses kbn-ui-shared-deps-npm to bundle common dependencies as shared webpack externals. This needed to serve both the new v2 packages and the v1 aliases so that both are available at runtime.

Pinning @elastic/charts with yarn resolutions

Then there's @elastic/charts. It depends on RTK v1 internally and can't just be upgraded independently since it's an upstream package. Yarn resolutions pin its nested dependencies to v1 versions:

A NormalModuleReplacementPlugin in the shared deps webpack config detects when an import of immer, @reduxjs/toolkit, redux, react-redux, or reselect originates from within @elastic/charts and redirects resolution to the nested v1 copies. This ensures that @elastic/charts resolves to its compatible v1 dependency set.

Keeping Kea on React Redux v7 with webpack externals

The kea library was another fun case. It declares react-redux as a peer dependency (>= 7), so without special handling its imports resolve to Kibana's default v9 package. The migration keeps Kea consumers on react-redux-v7, so Kea must use that same React context. The fix uses function-based webpack/rspack externals that skip externalizing react-redux when the import comes from node_modules/kea, combined with a NormalModuleReplacementPlugin that rewrites it to react-redux-v7. This ensures that kea uses the v7 React context that matches the <Provider> wrapping its consumers.

Both the webpack (kbn-optimizer) and the rspack (kbn-rspack-optimizer) configs needed these changes, with a shared isKeaReactReduxImport helper extracted to keep the logic consistent.

Using an ESLint rule to prevent cross-version imports

With two versions available, accidental cross-version imports are the biggest risk. A new @kbn/imports/no_redux_toolkit_v2_imports ESLint rule catches any import of the v2 default packages (such as @reduxjs/toolkit, react-redux, or redux, among others) in code that hasn't been migrated yet. It even auto-fixes them to the v1 aliases for file imports and Jest mock paths.

The rule is scoped via an override in .eslintrc.js to the ~36 plugin and package paths currently using v1. When a team migrates, they simply remove their path from the override list. This clean, self-service approach requires no coordination.

Why mixing React Redux v7 and v9 breaks the context

This is worth calling out because it's an easy failure mode to miss during an upgrade. react-redux v9 and v7 create separate React contexts. If a component tree has a v9 <Provider> at the top but a child component calls useSelector from v7 (or vice versa), React-Redux cannot find the matching context. In development, it throws an error explaining that the component must be wrapped in a matching <Provider>; in production, the missing context causes a runtime error when the hook accesses the store.

This means that each plugin needs to be explicitly pinned to one version. Shared packages that use react-redux can only be consumed by code on the same version, since mixing isn't possible. This is a constraint that makes the migration inherently per plugin rather than per file.

Migration batches: What can move independently

The dual-version setup gives every team a clear path forward, and the dependency graph analysis from the tracking issue identified natural migration batches:

• Batch 1: Independent, self-contained stores. Packages like kbn-coloring, transform, timelines, and expandable-flyout have fully internal Redux stores with no types leaking through their public APIs. These can be migrated independently by their owning teams, with minimal risk.

• Batch 2: Coupled packages. Some packages share RTK types across boundaries and must migrate together. The machine learning (ML)/artificial intelligence for IT operations (AIOps) chain is one example: @kbn/ml-response-stream exports a streamSlice (a createSlice return value) that @kbn/aiops-log-rate-analysis embeds directly into its configureStore. Migrating one without the other causes type mismatches between v1 and v2 slice types. Similar coupling exists across the Lens ecosystem. The Lens plugin depends on @kbn/coloring (which has its own RTK store), @kbn/lens-embeddable-utils, and @kbn/lens-common, while itself being consumed by 40+ packages and plugins across chart expressions, visualizations, Maps, Canvas, and observability plugins. Whether Redux types leak through a package's public API determines if it can be migrated independently or needs coordination. kbn-coloring's store is internal to its React components so it's safe to migrate alone, but other coupling points need careful analysis.

• Batch 3+: The big ones. Discover, Security Solution, and Lens each have their own migration timelines. Security Solution's 300+ files and mix of RTK with plain Redux v4 and typescript-fsa make it the largest effort, but the different patterns can be addressed independently. Lens has the trickiest v2 breaking changes around middleware configuration; stand-alone getDefaultMiddleware and PreloadedState are both removed in v2, and it has four custom middleware files with complex typing.

Beyond the batched migrations:

• Deprecated features can stay on v1 aliases. When the feature is removed, the v1 imports disappear through code deletion, without any migration work.

• Plain Redux v4 plugins (Canvas, Maps, and others) are entirely out of scope for RTK migration. They'd benefit from modernization, but that's a separate initiative.

• Kea plugins need react-redux-v7 to react-redux alias updates eventually, but no RTK migration. The longer-term question (whether to keep Kea or migrate to RTK v2) is a separate decision.

• The dual-version approach adds measurable bundle overhead during the transition. It’s a trade-off but is acceptable for the migration period.

Lessons for other large monorepo upgrades

The ESLint rule turned out to be the linchpin. Without automated enforcement, aliased imports would drift back to default names within weeks. With it, the migration state is visible in the paths listed in the override. As of the initial PR, zero files import from @reduxjs/toolkit v2. Every RTK usage goes through the redux-toolkit-v1 alias. That's the starting line.

The preparation work also reached beyond import paths. Jest mocks referencing react-redux needed updating to react-redux-v7, as did Storybook previews, test helpers, and ambient type declarations. Multiple rounds of node scripts/eslint_all_files --no-cache --fix caught the mechanical cases; the remaining cases needed manual fixes.

If you're facing a similar major dependency upgrade in a large monorepo, the pattern of giving the new version the default name and the old version an explicit alias is worth considering. New code naturally uses the current version, while older usage stays visible and trackable until it reaches zero.

What backward compatibility means for the Kibana Dashboards API

During technical preview, the API shape could change between releases.[1] That's no longer the case. General availability (GA) means:

• Complete backward compatibility. New fields and panel types will be added over time, but existing fields and behavior remain unchanged. Any future breaking changes would be very carefully considered and would only be introduced in a new major stack version.

• Production-ready with full support. The API carries Elastic's full support guarantees. You can safely use it in production environments for automated deployments, environment promotion, and programmatic dashboard management.

New Kibana API endpoints for Tags, Markdown, and Links panels

Elastic 9.5 also introduces a new  standalone endpoint for Tags, which let you categorize and filter dashboards. Now you can manage them programmatically through dedicated CRUD endpoints, making it easier to organize dashboards at scale across environments.

New Markdown and Links panel endpoints are available now in Serverless and will land in the next stack release (9.6).

What panel types does the Kibana Dashboards API support?

The Dashboards API supports all by-value panels in 9.5 (those defined directly in a dashboard, as opposed to library panels saved for reuse). Every supported panel type has a typed, validated schema.

Panel type

Status

XY charts

Supported

Metrics

Supported

Pie

Supported

Gauge

Supported

Heatmap

Supported

Data tables

Supported

Treemap

Supported

Discover sessions

Supported

Controls

Supported

Markdown

Supported

Links

Supported

ML panels

Supported

Observability panels

Supported

Maps

Coming soon

Vega

Coming soon

How to manage Kibana dashboards as code

The Dashboards API enables a full dashboards-as-code workflow: Export a dashboard as clean, diffable JSON, commit it to Git as the source of truth, review changes in pull requests, and deploy the same definition across development, staging, and production. Once a dashboard is managed as code, treat Git as the single source of truth: Changes made directly in the UI are overwritten the next time you deploy.

The main challenge when moving a dashboard between spaces, clusters, or stages is that dashboards reference objects like data views and library visualizations by ID. Because these IDs are auto-generated and differ across environments, a dashboard exported from one environment can point at objects that don't exist in another. There are three ways to handle this, listed here from most to least automated:

• Use Terraform. The Elastic Stack Terraform provider tracks each resource and maps IDs per environment automatically, so references stay consistent as you promote a dashboard from development to production.

• Define by-value Elasticsearch Query Language (ES|QL) panels. The most portable way to build a panel is to define its visualization with ES|QL directly in the dashboard. An ES|QL query reads from the indices you name in it, so the panel carries no external references to data views or library objects. The result is a fully self-contained, portable dashboard.

• Assign matching IDs. If you reference saved objects, like data views or library visualizations, create them with a chosen ID using PUT (upsert) rather than POST (which auto-generates an ID). Use human-readable IDs, like logs-prod, so they're easy to reuse and recognize across environments.

For a detailed walkthrough of these portability patterns and the full dashboards-as-code workflow, see the Manage dashboards as code documentation.

Create a Kibana dashboard with the Dashboards API using PUT

Here's a quick example creating a dashboard with a metric panel using PUT instead of POST to assign a custom ID using the dashboard name (service-health-overview). The same logic works for creating standalone visualizations saved in the library.

Kibana Dashboards API roadmap: Maps, Vega, and standalone endpoints

We're actively expanding the API surface. Maps and Vega panel support is next, adding typed schemas for them. We're also building standalone CRUD endpoints for Discover sessions (beyond their existing support as dashboard panels), Vega, Maps, and Annotations, decoupled from the dashboard lifecycle.

For the full schema definitions, visit the Dashboards API documentation. For Terraform users, the Elastic Stack Terraform provider supports the GA Dashboards API.

Note

1. The core endpoints are unchanged from the technical preview. If you built integrations against 9.4, they work in 9.5. The only breaking changes are two minor ones affecting dashboard listing and duration unit formats, documented here.

Dependency management at Elastic

At Elastic, we have to manage hundreds or even thousands of repositories, both private and public. When a critical CVE is discovered, we need immediate answers and actions: Which repositories are vulnerable? How quickly can we patch them? Apart from security, productivity questions also arise: How can we quickly propagate the release of a new package version across all the repositories that depend on it without spending too much time on manual tasks?

The initial trigger for searching ways of doing dependency management was the need to establish a secure foundation with automated updates for reducing CVEs. After carefully considering solutions on dependency management, we first started working on a self-hosted infrastructure. We were using our own Kubernetes cluster to run Mend Renovate Community Self-Hosted. The idea was to be able to provide a dependency management platform that our users could access in a self-service manner.

The initial experiment was successful, so more and more teams started onboarding our platform and using it in their everyday repositories’ lifecycle for updates and CVE patching. This happened so fast that we soon hit the ceiling of our self-hosted installation.

The challenge: How can we scale a dependency management platform in a large organization with a significant number of repositories?

Our dependency management platform was processing one repository at a time and the sequential processing model couldn’t keep up, due to the large number of repositories that we own. We had already identified that the issue resided within the concept that a single instance of our dependency management tool could process our big and ever-growing list of repositories. Repositories waited in a queue, sometimes for many hours. More than 50% of our repositories were not even processed daily. That means that more than 50% of our repositories waited more than 24 hours between scans.

Large repositories created larger bottlenecks, due to their sizable codebases and their multiple open PRs. GitHub webhook events disrupted the sequence. Automerge became unreliable because scan timing was unpredictable. We had made a promise to our users for the frequency of scans, and we couldn’t fulfill it.

The decision to build in-house: Meeting Elastic's unique scale and security needs

While we considered commercial options, including Mend's Renovate Self-Hosted Enterprise Self-Hosted edition, internally at Elastic we had a few key initiatives ramping up.

Our decision to build an in-house platform was driven by the recognition that only a deeply customized solution could meet Elastic's specific, nonnegotiable requirements:

1. Investing in our internal developer platform: At the time, we had already started heavily investing in our internal developer platform. We were discussing and designing ways that each one of our services could fit into that. This meant that we wanted to test-drive our own rules and practices for our dependency management platform. On top of that, new guidelines were coming into play and we wanted to design the platform ahead of events.

2. Native integration and workflow customization: We required straightforward integration with our internal tooling and internal processes. For example, we wanted to centralize configuration as code with our Service Catalog (Backstage). We have specific needs around the usage of Backstage that we wanted to make our platform compatible with. So, although it would be possible to make use of the Renovate Self-Hosted APIs alongside our Backstage automation, this wouldn’t cover completely for our internal processes.

3. Elastic-specific defense-in-depth security: Our stringent security compliance required bespoke security mechanisms tailored to our ecosystem. We were working to harden our usage of “non-human identities.” The way this hardening of access worked meant that the nonstandard means to authenticate to GitHub wouldn’t work with an off-the-shelf tool that didn’t support this internal implementation.Our workflow included implementing a parent-child workflow secret encryption pattern and using transient, single-use GitHub tokens. Building in-house was the only practical way to embed these unique security layers and minimize the attack surface across our complex multicloud environment.

The solution: Workflow orchestration for dependency management

Our solution started from the fact that we wanted to build on the dependency management tool that we already used and not replace it and look for other solutions. It had shown signs of its potential, and its flexibility is important for different needs throughout our organization. We considered different solutions, and what helped us make up our minds was the big and sometimes special needs that we have to cover for. We decided to build a reliable and scalable dependency management platform, where each repository will be processed on its own, removing bottlenecks and setting us up for growth.

We designed the platform abiding to three core principles:

1. Parallel processing

Every repository gets its own dependency management processing environment. No more queues. Our concurrency is only limited by the number of resources we spend. We have also applied smart distributed scheduling to avoid getting rate limited by GitHub.

2. Self-serviceable

We use our Service Catalog (Backstage) to automatically onboard and manage any new repository. We use our own resource definition to give the end user the option to select how often a repository will be processed, how many resources they want to allocate to their schedules, and if they want to turn processing off or back on for any reason. We plan to add more options that way as our users’ needs evolve and they get more fluent with the new installation.

3. Reduced secret scope and namespace isolation

For increased security, we supply our dependency management pods with ephemeral GitHub tokens that are being generated at the start of each workflow. On top of that, we isolate our workloads in specific namespaces so they can be provided only the necessary secrets. We control what secrets can be accessed by each dependency management workflow using Kubernetes RBAC. We also use encryption to propagate the GitHub token from the parent to the child workflows.

We rebuilt our platform using Kubernetes and harnessing the power of Kubernetes, Argo Workflows powers the logic of our processes, and Renovate CLI is set up for scanning and processing one repository at a time.

The beauty: We’re using battle-tested open source projects in an original way, providing new working examples for all of those projects and, at the same time, amplifying development velocity and consolidating CVE reduction for our teams.

Dependency management architecture: Four microservices

The platform comprises four custom-built components:

Workflows Operator (Go/Kubebuilder)

A Kubernetes operator managing workflow lifecycle through three Custom Resource Definitions (CRDs):

• RepoConfig CRD: Single source of truth for repository configuration.

This is how RepoConfig is defined in the operator:

And this is what an instance of RepoConfig would look like:

• Parent CRD: Manages CronWorkflows for scheduled scans.

Inside the reconciliation loop of the parent controller, we make sure that workflow settings are created and kept up to date or even deleted if needed.

First, it gets some globally configured settings for workflows:

It makes sure a mutex configmap is up to date to prevent similar workflows from running together:

Then it creates a Workflow Manager that’s the struct which will create or update the CronWorkflows and the Workflow Templates:

• Child CRD: Manages WorkflowTemplates with per-repository resources.

The child controller has a similar reconciliation duty to the parent, but this time it’s responsible for workflow templates in the child namespace that will be triggered by the parent workflows.

The multi-controller pattern provides clear separation: RepoConfig Controller handles onboarding/offboarding, Parent Controller manages scheduling, and Child Controller handles execution templates.

GitHub Events Gateway (Go)

A secure webhook proxy that receives GitHub webhooks, verifies signatures, filters by organization/repository, and routes to Argo Events. We built 10 distinct sensors responding to dependency dashboard interactions, PR events, and package updates.

This gateway enables integration with GitHub Apps by:

• Verifying incoming GitHub webhook signatures for security.

• Forwarding valid events to the Argo Events EventSource with all relevant headers and authentication.

• We also configure an authSecret on the EventSource and provide this as a Bearer header in forwarded requests.

• Providing logging, metrics, and retry logic.

It performs various validations on each GitHub Event request.

It makes sure some HTTP attributes are present:

While it also validates the signature of each request and its organizsation:.

Finally, it routes to Argo Events based on event type:

On the Argo Events side of things, 10 sensors watch the Argo Events EventBus for new events:.

Then the script applies each sensor’s logic:

Backstage Syncer (Go)

This polls our Service Catalog (Backstage) for Repository Real Resource Entities, transforms them into RepoConfig CRDs, and keeps the platform in sync with configuration changes. Changes apply within three minutes.

Finally, it writes that data into RepoConfig instances.

Workflows base (Mixed: JavaScript, Go, Helm)

The foundation layer contains Helm charts, JavaScript configs, a Go wrapper for Renovate CLI with encryption support, and a custom APK Indexer for Alpine packages.

Self-service configuration

Teams configure their repositories declaratively through Backstage:

Resource groups allocate CPU and memory based on repository size:

• SMALL: 500m CPU, 1Gi memory.

• MEDIUM: 1000m CPU, 2Gi memory.

• LARGE: 2000m CPU, 4Gi memory.

Configuration is version-controlled, auditable, and applies automatically.

The parent-child pattern

The execution model uses a parent-child workflow pattern:

• Parent workflow: Lightweight CronWorkflow running on schedule. Encrypts secrets, determines whether a scan should run, passes configuration to the child.

• Child workflow: Ephemeral pod where Renovate CLI runs. Allocated resources dynamically, decrypts secrets in isolation, terminates after completion.

This separation provides security (secrets encrypted at parent level), resource optimization (parents use minimal resources), and scalability (children run in parallel).

The results

Performance transformation

• Before: One repository at a time, some repositories would not get processed possibly even for a day or more, less than 1,000 scans per day.

• After: 100+ concurrent scans, usually 8,000 scans and up to 10,000 recorded scans per day, limited only by the amount of resources we’re willing to spend and how we handle GitHub rate limits.

Cost efficiency

However weird it may sound, running 8,000 pods a day can get you the same result much cheaper than having one long-running pod trying to achieve the same results.

In the previous setup, we were running a single instance that, on a good day, would perform 500–600 scans. At the same time, due to the fact that different kinds of repositories would be executed on the same pod, we needed to size the pod for the biggest ones. That sizing would be much bigger than our current extra large offering, using 8 CPUs for the pod and 16G of memory.

To meet the current daily output, the single pod would need to run for 12 days. So comparing the cost of that single pod running for 12 days to 8,000 pods of our “MEDIUM” size running each day, our new design is far more efficient for the same output of scans:

Metric

Scenario A (Workflows)

Scenario B (The long-running single pod)

Setup

8,000 pods (1 vCPU / 2GB)

1 pod (8 vCPU / 16 GB)*

Duration

10 minutes each

12 days continuous

Total work time

1,333 compute hours

288 compute hours

Total cost

$65.83

$113.75

However, let’s take into consideration that our default for our workloads is set to “SMALL,” with the great majority running successfully with 0.5 CPU and 1G RAM, and only a few need to change to medium, large. Let’s see what happens if 60% of our workloads are running on “SMALL,” 30% at “MEDIUM,” and 10% at “LARGE,” which is closer to the truth.

Metric

Scenario A (Mixed swarm)

Scenario B (The long runner)

Strategy

8,000 pods (mixed sizes)

1 pod (8 vCPU / 16 GB)*

Duration

10 minutes each

12 days continuous

Total cost

$52.66

$113.75

Savings

$61.09 (54% cheaper)

—

We can see that, for the same output, we’re far more cost-efficient in our current setup.

Enhanced security

• Ephemeral GitHub tokens (minutes of exposure versus days).

• Namespace isolation with Role-Based Access Control (RBAC) boundaries.

• Secret encryption at rest in parent workflows.

• Removed direct vault access.

Predictable performance

With guaranteed scan frequency, we can finally set Service Level Objectives (SLOs). Automerge works reliably. Teams trust the platform to deliver what’s promised.

Key architectural decisions

Here are some of the milestone design decisions that shaped how the platform looks.

• Why parent-child workflows?

We adopted this pattern to enforce a defense-in-depth strategy. By restricting high-value credentials (such as GitHub App secrets) to a dedicated, locked-down namespace, we use RBAC to ensure that ephemeral execution pods cannot arbitrarily access sensitive data. Recent supply chain vulnerabilities (for example, the "Shai Hulud" continuous integration/continuous delivery [CI/CD] attacks) have demonstrated the criticality of isolating runtime environments that execute dynamic scripts from the credential store.

Simultaneously, this decoupling enables granular resource optimization. The "parent" workflows act as lightweight orchestrators with a minimal footprint, while the "child" workflows handle the compute-intensive dependency scanning. This separation simplifies lifecycle management by allowing us to apply distinct reconciliation logic to each layer, granting users control over execution parameters (child) while retaining administrative control over the scheduling and security infrastructure (parent).

• Why self-serviceable?

Eliminating our team as a bottleneck for repository configuration was a critical requirement. Our mission was to architect a scalable, self-service platform capable of supporting diverse use cases. We recognized that acting as gatekeepers for every configuration change was unsustainable, given the sheer volume of repositories. Instead, we adopted a philosophy of enablement: providing the “rails” (infrastructure and guardrails) while empowering users to drive the “trains” (execution and customization). We believe this shift toward team autonomy significantly enhances productivity by allowing users to tailor the system to their specific operational needs.

• Why Kubernetes Operator pattern?

As mentioned above, a foundational design principle was to ensure that the platform was fully self-serviceable. We required an automated mechanism to capture user intent (such as toggling scans, adjusting scheduling frequency, or tuning runtime resource limits) and instantly propagate those changes to the underlying workflows. Anticipating future requirements, the system also needed to be easily extensible.

To achieve this, we developed a custom Dependency Management Kubernetes Operator. By using CRDs as the interface for configuration, we established a Kubernetes-native reconciliation loop. This operator continuously monitors the desired state defined by the user and automatically orchestrates the necessary updates to the workflow infrastructure. This ensures an event-driven, seamless operation, where the platform logic handles all complexity behind the scenes.

• Why design a GitHub Events Gateway?

Adopting an event-driven architecture (EDA) was essential for the platform's responsiveness. While CronWorkflows provided a reliable baseline schedule, we required the agility to handle ad hoc executions, such as users manually triggering scans via the dashboard. To achieve this, we needed a dedicated ingestion gateway to validate payload integrity and route requests intelligently.

We evaluated existing solutions, including the native GitHub EventSource for Argo, but we identified significant risks regarding operational overhead and strict GitHub API quotas (for example, webhook limits per repository). Consequently, we built a custom gateway to decouple our infrastructure from these limitations.

Crucially, this gateway served as a strategic traffic control point during our migration. It acted as a switch, enabling us to perform a gradual, granular rollout (traffic shifting) from the legacy system to the new infrastructure. This ensured that onboarding thousands of repositories was a controlled, risk-free process rather than a “big bang” switchover.

Lessons learned

Some lessons that we learned go hand-in-hand with the Elastic Source Code:

1. Customer First: Platforms are built for users. So it’s important to take users’ needs as priority number one. This shapes the platform into efficiently designed infrastructure and applications that reduce friction with users, simplify the scaling of the platform and ease adoption.

2. Space, Time: Sometimes the path of least resistance leads to shifting sands. We initially tried to optimize the existing sequential processing model, but this failed to resolve our issues; in fact, it only introduced more complexity and loose ends. The bold decision to rearchitect the platform with parallel processing required significant up-front effort. However, it ultimately paved the way for sustainable platform growth and virtually eliminated tedious daily administrative work.

3. IT, Depends: A platform cannot operate in isolation; its success depends on how well it integrates with the broader ecosystem. In our case, integration with Backstage was critical, as it serves as the source of truth for seamless service onboarding. Similarly, connecting to Artifactory allowed us to manage private package updates efficiently, and the list of essential integrations goes on.

4. Progress, SIMPLE Perfection: Throughout the implementation, we constantly pressure-tested our initial assumptions and adapted to new barriers as they emerged. Rather than getting paralyzed by perfectionism, we adopted an iterative approach, tackling challenges one by one and adjusting our migration strategy to meet real-world conditions.

What’s next

The delivery of the platform enables us for more meaningful work that will help us improve the UX and efficiency of our platform. Some examples are:

• Increase and guardrail the adoption of auto-merge

The auto-merge feature significantly accelerates team velocity by eliminating tedious manual tasks. However, we need to make sure that strict guardrails are in place to ensure that this increased speed does not come at the expense of security.

• Improve observability around end-user experience

A critical priority for our roadmap is enhancing observability, not just at the platform level but also specifically from the end-user’s perspective. While capturing infrastructure metrics is straightforward, understanding the actual user experience requires deeper insights. We’re working to define core user-centric key performance indicators (KPIs) so our telemetry can detect friction points and performance issues before they escalate into user complaints.

• Remove barriers for greater adoption

Looking ahead, our priority is to identify and remove any barriers hindering platform adoption. Whether this requires developing new integrations or deploying specific feature sets, we’re committed to data-driven planning. We’ve successfully built a platform designed for scale; our focus now shifts to maximizing its potential.

The bigger picture

The dependency management workflows project demonstrates a broader principle: When you need to scale open source tools beyond their default deployment model, Kubernetes-native patterns provide a path forward.

By embracing:

• CRDs for configuration.

• Operators for lifecycle management.

• Event-driven architecture for responsiveness

• GitOps for deployment.

We built orchestration that scales independently of the number of repositories it manages. The performance of scanning one repository is the same whether we’re managing 100 or 1,000.

When a critical CVE is announced, we now have answers in minutes, not hours. That’s the difference between a bottleneck and a competitive advantage.

Acknowledgments

This platform builds on excellent open source tools:

• Kubebuilder: The open source framework we used to kick-start our Kubernetes Operators that bootstrap and orchestrate our workflows. [1][2]

• Backstage: The open source framework on which we’ve built our Service Catalog and which we use as our source of truth. [1][2]

• Argo Workflows and Argo Events: The open source suite we used to orchestrate complex processes and add dynamic processing based on events. [1][2][3][4]

• Renovate CLI: The open source dependency management tool processing our repositories. [1][2]

* The AWS Fargate pricing model was used as a reference for of the cost of a single pod, although our workloads are not running necessarily on AWS and are running on full- blown Kubernetes clusters.

As context engineering is exploding, ensuring that rapid prototype projects don’t expose business- or mission-critical data to unauthorized recipients is a significant concern. For audiences interested in technology and policy alike, I've championed the concept of a privacy-first GenAI sandbox, which I’ll simply refer to as a sandbox from here on. In this article, the term sandbox refers to a self-service, secure prototyping space (much like a child's sandbox, where the wooden edges prevent sand from escaping), allowing organization members to test their custom context engineering applications safely, without risking exposure of confidential data.

Production-grade GenAI sandboxes = enabling privacy and innovation

GenAI, from text-generating tools, like ChatGPT, Claude, and Gemini, to image creators, such as Google’s Nano Banana, OpenAI’s DALL-E, and Midjourney, has sparked discussions everywhere: in classrooms, at dinner tables, in regulatory circles, in courts, and in boardrooms over the past two years.

I’ve had the privilege of sharing Elastic’s approach to context engineering, and particularly RAG, with customers, including developers and C-suite executives, and with contacts of mine, ranging from friends and family to legislators. Think of context engineering as a librarian that looks up and then serves contextual data to augment text, audio, or image GenAI apps that they don’t have in the data they were trained on for their intended tasks; for example, looking up sports scores and headlines to help a text-generation application answer the question, “What happened in the National Hockey League yesterday?”

Elasticsearch Labs has fabulous primers on context engineering here and RAG here, if you’re unfamiliar with the concept and would like to do more reading.

A privacy first approach ensures that context engineering supplies the GenAI app with protected, selected, or delicate data, fostering responses that are better informed and more pertinent than what might be generated using solely public information. An example of this would be providing a GenAI-powered, interactive text chat experience (chatbot) for university students to obtain financial aid and scholarship information relevant to their personal background, without risking exposure of personally identifiable information (PII), such as their Social Security number or birthdate, to malicious actors extracting information via common vulnerabilities, as per the OWASP Top 10, or the LLM itself.

The core tenets of the logic behind deploying a sandbox are as follows:

1. Users will find a way to incorporate GenAI into their daily workflow, regardless of whether one’s organization provides the tools. Even in organizations where preventing such “shadow IT” is, realistically, impractical or impossible, providing and monitoring access to prevent disclosure of an organization’s sensitive data still remains imperative; a sandbox is just the place to turn such tools loose.

2. Providing a sandbox to deploy applications with Application Performance Monitoring (APM) and information security (InfoSec) best practices embedded allows an organization to derive insights into potential use cases for GenAI while also safeguarding privacy, enabling audit and accountability of GenAI use, and establishing centralized cost management.

3. An organization’s sandbox should allow either self-service or low-touch deployment of peer-reviewed GenAI applications to permit maximum experimentation with minimum friction by those inclined to develop their own applications.

4. If properly implemented and contained within the organization’s controlled perimeter, the sandbox allows leveraging data assets available to the organization without triggering the liabilities that could attach to unauthorized or unintended external sharing or other leakage of protected data such as PII – think California CCPA, or the EU/UK GDPR for instance.

This article will not focus on building a GenAI app; there are numerous excellent examples here on Elasticsearch Labs. Instead, I’ll be focusing on the recipe necessary for deploying a sandbox that provides the security and availability needed to implement principle #3 above.

Foundational ingredients

For a sandbox to be considered production grade, the following foundational ingredients should be considered:

Let's explore why each ingredient plays a crucial role in our sandbox recipe. As we do, please note that brand-name decisions I’ve listed below are based on personal experience and aren’t an endorsement of one technology or another by Elastic. As with any recipe, these then form my preferred ingredients. You can, of course, substitute in each area to make the recipe to your liking:

1. Containerization platform

The first ingredient in our sandbox recipe is the selection of a containerization platform. These platforms, while conceptually similar to the virtual machines that have been a staple of enterprise IT for the past 15+ years, represent a significant evolution in how applications are packaged and deployed. They’re designed for rapid deployment, upgrades without service disruption, and native distribution across both on-premises and cloud computing environments, while also providing increased testability, validation of infrastructure, and auditability. The platform you choose, often managed through infrastructure as code (IaC) to ensure reproducibility and consistency, is the foundation that enables agility and scalability for your GenAI applications.

Key components of a containerization platform

A robust containerization platform is built on several key components:

• Container runtime: The software that executes containers and manages their lifecycle. A popular example is Docker, which provides the tools to build, share, and run container images.

• Image build infrastructure: This is the process and tooling used to create container images from your application's source code. Tools like Dockerfiles provide a clear, repeatable way to define the environment, dependencies, and application code within an image, ensuring consistency across development, testing, and production environments.

• Orchestration engine: For a production-grade environment, you need a system to automate the deployment, scaling, and management of containers. Kubernetes (k8s) is the industry-standard for this, providing powerful features for load balancing, self-healing, and service discovery. More on that below in ingredient #2.

1. 1 Infrastructure as code:

To ensure the reproducibility and maintainability of your sandbox, a containerization platform should be managed using IaC principles. This means that instead of manually configuring your platform, you define your infrastructure (for example, Kubernetes clusters, networking rules, security policies) in code files (for example, using Terraform or Pulumi). This approach provides several benefits:

• Version control: Your infrastructure can be treated like any other code, allowing you to track changes, revert to previous versions, and collaborate with your team using Git.

• Consistency: IaC dramatically reduces manual errors and ensures that your sandbox environment can be recreated identically in any cloud or on-premises location.

• Automation: It enables you to automate the entire setup and teardown process, making it easy to create temporary sandboxes for specific projects or testing.

2. Hosting and orchestration

As we introduced in the "Containerization platform" section, a powerful orchestration engine is needed to manage our containers at scale. For this, k8s is the de facto standard for orchestrating a production-grade sandbox. If you’re unfamiliar, check out the Cloud Native Computing Foundation (CNCF) primer on k8s available here. Whether running in the cloud or on-premises, Kubernetes provides the robust framework needed to deploy, scale, and manage the lifecycle of containerized applications. Major cloud providers, like Google Cloud (Google Kubernetes Engine [GKE]), Amazon Web Services (Elastic Kubernetes Service [EKS]), and Microsoft Azure (Azure Kubernetes Service [AKS]), all offer mature, managed Kubernetes services that handle the underlying complexity, including in particular contractually assured and independently certified compliance with statutory privacy and information security mandates, allowing your teams to focus on building and deploying applications.

For a GenAI sandbox, Kubernetes is particularly valuable because it can efficiently manage and scale GPU resources, which are often necessary for two key components of the GenAI stack: 1) privately hosted LLMs; and 2) the inference processes that power them (discussed in more detail in ingredients #6 and #7). Its ability to automate deployments and manage resources ensures that rapid prototypers can experiment with different models and applications without needing to become infrastructure experts, all within the secure and isolated area, called a namespace in k8s, that you define. This abstraction is key to the sandbox's success, empowering innovation while maintaining centralized control.

3. Code repository / image repository

A centralized code repository is an essential element of a secure and collaborative GenAI sandbox. It provides a single, controlled environment for developers to store, manage, and version their code, preventing the proliferation of sensitive information across disparate, unsecured locations. By establishing a centralized repository, organizations can enforce security policies, monitor for vulnerabilities, and maintain a clear audit trail of all code changes, which is critical for maintaining data privacy and integrity within the sandbox environment.

For instance, a service like GitHub, when integrated with your organization's identity and access management (IAM) and single sign-on (SSO) solutions (see ingredient #4 below), becomes a powerful tool for enforcing the principle of least privilege. This integration ensures that only authenticated and authorized developers can access specific code repositories. You can create teams and apply granular permissions, restricting access to sensitive projects and preventing unauthorized code modifications. This is especially important in a GenAI context where code might contain proprietary algorithms, sensitive data connectors, or even, in some cases, organization or user-level credentials or other confidential information.

Furthermore, modern repository platforms offer automated security scanning features. These tools continuously scan code for known vulnerabilities, insecure coding practices, and exposed secrets. If a developer accidentally commits a password or an API key, the system can automatically flag it and notify the security team. This proactive approach to security is essential for preventing data breaches, enforcing legal requirements and contractual commitments of confidentiality, and ensuring the overall integrity of the GenAI applications being developed to deploy in the sandbox. By mandating that all development occurs in a centralized and secured repository, you create a transparent, auditable, and secure foundation for innovation, allowing your developers the freedom to experiment without compromising organizational security.

4. Identity and access management

IAM is a core component of a secure, privacy-first grounded AI environment. It provides the foundation for ensuring that only authorized individuals and services can access sensitive data and powerful AI models. A robust IAM framework enforces the principle of least privilege, granting the minimum level of access necessary for a user or service to perform its function.

4.1 Single sign-on:

SSO streamlines user access by allowing users to authenticate once and gain access to multiple applications and services without re-entering their credentials. In a sandbox environment, SSO simplifies the user experience for developers, data scientists, and business users who need to interact with various components of the AI ecosystem, such as data repositories, modeling workbenches, and deployment pipelines. By centralizing authentication, SSO also enhances security by reducing the number of passwords that can be compromised and providing a single point for enforcing authentication policies. Importantly, it also lowers the barrier to entry for less-experienced developers to properly protect the data they are using in the sandbox, in turn preventing the inadvertent disclosure of sensitive information to insiders and outsiders alike.

4.2 Role-based access control:

Role-based access control (RBAC) is a method of restricting network access based on the roles of individual users within an organization. In the context of a GenAI sandbox, RBAC is used to define and enforce permissions for different user personas. For example, a data scientist role might have read/write access to specific datasets and the ability to apply machine learning models, while a business analyst role may only have read-only access to the outputs of those models. This ensures a clear separation of duties and prevents unauthorized access to or modification of sensitive data and AI assets.

4.3 Attribute-based access control:

Attribute-based access control (ABAC) provides a more granular and dynamic approach to access control than traditional RBAC. ABAC makes access decisions based on a combination of attributes of the user, the resource being accessed, and the environment. For instance, access to a particularly sensitive dataset could be restricted to users who are on the data scientist team (user attribute), accessing a resource tagged as PII (resource attribute), and are doing so from a corporate network during business hours (environment attributes). This level of granular control is critical in a GenAI sandbox for enforcing complex data governance and privacy requirements. We’ll come back to this later, when discussing the search AI datastore.

4.4 Access auditability:

A robust IAM framework also ensures that the granting, use, review and revocation of all access permissions is granularly logged, discoverable and auditable, so that in case of any suspected or confirmed incident, responders can quickly understand what happened, contain the incident, assess its extent, and comprehensively remedy its consequences. This is not only important for the organization’s own security, but also necessary to comply with any incident reporting and breach notice requirements that could be triggered.

5. Secrets management

Of all the ingredients in our recipe, secrets management is perhaps the most potent, yet most frequently overlooked. Much like a tiny pinch of saffron can dramatically alter a culinary dish, a single mishandled secret can have an outsized and devastating impact on your organization's security and reputation. In our context, a secret is any piece of sensitive information needed for our applications to function: API keys for first- or third-party services, database passwords, trust certificates, or tokens for authenticating to LLMs.

When these secrets are hard-coded into source code or left in plain-text configuration files, they create a massive vulnerability. A leaked API key or an exposed database credential can bypass all other security measures, providing a direct path for attackers to access sensitive data and systems. This is especially critical in a GenAI sandbox, where developers are frequently connecting to various data sources and external model providers. Without a robust secrets management strategy, you’re leaving the keys to your kingdom scattered across your digital landscape, turning your innovative sandbox into a potential source for a major data breach.

To properly secure these secrets, a dedicated secrets management platform is an essential ingredient. These tools provide a centralized, encrypted vault for storing secrets, with robust access control, auditing, and dynamic rotation capabilities. Whether you choose a self-hosted solution, like HashiCorp Vault, or a managed cloud service, such as Google Cloud's Secret Manager, or AWS Key Management Service (KMS), the principle is the same: Programmatically inject secrets into your applications at runtime. This practice ensures that secrets are never exposed in your code, keeping your most valuable credentials secure and your sandbox environment protected.

And this is more than just a best practice: since secret management technology is readily available and widely used, it forms part of the “state-of-the-art” which certain privacy laws and regulators reference as the benchmark against which an organization’s information security posture must be assessed. Failing to protect an organization’s most valuable secrets with the latest and greatest techniques available is not only a missed opportunity, it is also a potential case of regulatory non-compliance, as enforcement agencies and courts of law often recall.

6. Private LLM deployment(s)

Early in the advent of modern GenAI, the primary driver for using managed services, like Azure OpenAI, was the assurance that customer prompts and data would not be used to retrain public models. This was a crucial first step in enterprise adoption. However, as the field has matured, the conversation has shifted. While data privacy remains paramount, the decision to use private LLM instances, whether from major cloud providers or self-hosted, is now equally driven by the need for guaranteed throughput, predictable latency, and fine-grained control over the model's operational environment to support production-grade applications.

This critical ingredient comes in three distinct flavors, each with valid use cases and its own set of trade-offs:

A. Cloud-hosted SaaS

This is the most common and accessible approach. Services like OpenAI Enterprise, Azure OpenAI, Google Cloud's Vertex AI, and AWS Bedrock provide access to powerful, state-of-the-art models through a managed API.

• Pros: This flavor offers the fastest time-to-market. The cloud provider handles all the underlying infrastructure, scaling, and maintenance, allowing teams to focus purely on application development. It provides a simple, pay-as-you-go model and access to a diverse model library of proprietary and open-source options.

• Cons: This approach offers the least control over the underlying infrastructure, which can lead to variability in performance during peak demand. It can also be more expensive at very high volumes, and it creates a dependency on the provider's roadmap and model availability. It also increases the potential vulnerability surface of the application, with data leaving the customer premises: a challenge for highly regulated and/or sovereignty-minded customers.

B. Cloud-hosted GPU + containerized LLMs

This flavor involves running open-source LLMs (like models from Mistral or Meta's Llama series) on your own virtualized GPU infrastructure within a cloud provider. This is typically managed using the containerization and Kubernetes orchestration we've already discussed, often with high-performance inference servers like vLLM.

• Pros: This approach provides a powerful balance of control and flexibility. You gain direct control over resource allocation, model versioning, and the serving configuration, allowing for significant performance tuning. In high-concurrency scenarios, a well-tuned inference server can dramatically increase throughput. For example, benchmarks have shown inference engines like vLLM delivering significantly higher tokens-per-second and lower latency compared to less production-oriented servers under heavy load [Red Hat, 2025].

• Cons: This option carries a higher operational burden. Your team is now responsible for managing the GPU instances, container images, and the inference server configuration. It requires a deeper technical expertise in machine learning operations (MLOps) and infrastructure management to implement and maintain effectively.

C. On-premises GPUs + containerized LLMs

The most controlled, and often most complex, approach involves deploying containerized LLMs on your own dedicated hardware within your own data centers. This setup is functionally similar to the second flavor but removes the reliance on a public cloud provider for the hardware layer.

• Pros: This flavor offers maximum security, control, and data sovereignty. It’s the only option for organizations that require a completely air-gapped environment, where no data leaves the physical premises. For massive, predictable workloads, it can become more cost-effective in the long run by avoiding cloud data egress fees and per-transaction costs.

• Cons: The initial capital expenditure for purchasing and maintaining high-end GPU hardware is substantial. It requires a highly specialized team to manage the physical infrastructure, networking, and the entire software stack. This approach is more difficult to scale, as it requires the physical procurement and installation of new hardware.

7. Search AI data store

If the LLM is the brain of our GenAI application, then the datastore is its heart, pumping relevant, context-rich information to be reasoned upon. For a RAG application to be truly effective, it cannot rely on a simple vector database alone. The grounding data is often complex, containing a mix of unstructured text, structured metadata, and a variety of data types. Therefore, the datastore you select must possess a unique set of characteristics to handle this complexity at scale.

Underpinning this entire process is the creation of vector embeddings, numerical representations of your data relative to the knowledge set of that embedding space. To enable semantic search, your data must first be converted into these numerical representations by an inference model. A flexible datastore should not only store these vectors but also be capable of hosting the inference process itself. Crucially, it should allow you to use your model of choice, whether it's a state-of-the-art multilingual model, a fine-tuned model for a specific domain like finance or law, a compact model built for very high-speed results, or even a model that can process images. By managing inference, the platform ensures that your data is consistently and efficiently vectorized, paving the way for the powerful search capabilities that follow.

First, it must master hybrid search. The best retrieval systems don't force a choice between traditional keyword search, like BM25, which excels at finding specific keywords, and modern vector search, which excels at finding results using semantic meaning (that is, natural language). A truly capable datastore allows you to use both simultaneously in a single query. This ensures you can find documents that match exact product codes or acronyms while also finding documents that are conceptually related, providing the LLM with the most relevant possible context.

Second, it needs a sophisticated method for intelligent result reranking. When you run a hybrid search that combines multiple approaches, you need a way to merge the different result sets into a single, coherent ranking. Techniques like reciprocal rank fusion (RRF) are crucial here, as they intelligently combine the relevance scores from different queries to produce a final list that is more accurate and relevant than any single approach could deliver on its own.

Finally, a search AI-oriented datastore must be a unified engine with security built in. For enterprise RAG, it's not enough to just find similar vectors. You must be able to apply security and access controls to data before the search even happens. The aforementioned RBAC and ABAC capabilities allow prefiltering of content at search time, ensuring that the vector search is only performed on data a user is authorized to see. This mitigates risks of accidental or malicious circumvention of your access controls through the sandbox preserving demonstrable compliance with privacy and confidentiality requirements. This capability, which combines filtering, full-text search, and vector search in a single, scalable platform, is the defining characteristic of a datastore truly ready to power a secure, privacy-first GenAI sandbox.

8. APM and security

The final ingredient in our recipe ensures the health, security, and performance of the entire sandbox: a unified platform for APM and security information and event monitoring (SIEM). A key characteristic of a truly versatile search AI datastore is its ability to power the R in your RAG applications, while also acting as the standards-based repository for all logs, metrics, and traces generated by your infrastructure and applications. By consolidating this operational data into the same powerful datastore, you create a single pane of glass for observability and security.

This approach provides several critical capabilities. At the infrastructure level, you can monitor the performance and resource utilization of both the k8s clusters hosting your sandbox and the underlying GPUs that power your LLMs, allowing you to proactively identify bottlenecks or failures. At the application layer, APM provides detailed traces to diagnose latency issues or errors within your GenAI prototypes. For security, this centralized datastore becomes your SIEM, correlating login events, application logs, and network traffic to detect anomalous behavior or potential threats within the sandbox.

Most importantly, this unified platform allows you to gain deep insights into the usage of the GenAI applications themselves. By ingesting and analyzing the application telemetry, which should include the prompts being submitted by users wherever permissible, potentially with PII redacted, you can identify trends, understand what types of questions are being asked, and discover popular use cases. This provides an invaluable feedback loop for improving your RAG applications and demonstrates the power of using a single, scalable datastore to secure, monitor, and optimize your entire GenAI ecosystem.

Cooking the recipe

With all of the ingredients in place, let’s talk about the steps for assembling them into a production-grade sandbox.

As with any recipe book, let’s start with a photo of the cooked dish. Here’s a view of what a final architecture might look like:

The holistic environment depicted here consists of a Kubernetes cluster to host your sandboxed AI applications (with dev/preprod/prod namespaces for a continuous integration and continuous deployment [CI/CD] pipeline), an IAM infrastructure for authentication, a few GenAI applications, a repository for code and container images, and a wrapper of APM and cyber monitoring around the entire sandbox.

Recipe step 0: Policy baseline

Before you begin mixing any ingredients, every good chef performs their mise en place, that is, setting up their station for success. In our recipe, this means establishing clear policies for how the sandbox will be used. This is the foundational step, where you decide the rules of your kitchen. Will developers be allowed to use internal production data, or production data sanitized with techniques like pseudonymization and differential privacy, or life-like synthetic data, or only public data? Will the sandbox be a completely self-service platform, or a managed service with guardrails? Will application updates require a formal Change Review Board, or is a peer-review process sufficient? These questions are highly specific to each organization’s context and purposes. Answering them up front is critical, as these policy decisions will directly influence how you configure every other ingredient in the recipe.

Recipe step 1: InfoSec baseline

As stated in the “Ingredients” section, IAM is a nonnegotiable part of our recipe. Before letting anyone into the kitchen, you must secure the perimeter and ensure only authorized chefs wearing your approved uniform and compliant protective gear can access the tools and ingredients. This means working directly with your information security organization from day one to build the sandbox on a foundation of strong security principles. Access to your datastore, your code repository, your Kubernetes hosting environment, and the applications themselves must be restricted based on established best practices.

With your organization’s IAM policies enforced in the environment, a practical authentication flow might look like the one depicted in figure 3.

As you can see in the figure, no communication can occur between applications in the Kubernetes production namespace without first passing through an OAuth proxy, such as Vouch. This ensures every user is authenticated against a central provider, like Okta, which enforces policies such as two-factor authentication. In this model, critical user context, such as username and IP address, can be passed along with every request, enabling robust auditing and nonrepudiation at the application layer.

Recipe step 2: Container configuration baseline

Assuming that many of your rapid prototypers are passionate innovators but not necessarily seasoned software engineers or legally trained data compliance experts, it’s critical to provide a baseline configuration to ensure their success and security, without putting them at risk of breaching any rules or policies inadvertently. Think of this step as providing a master recipe card that guarantees consistency. At a minimum, you should provide clear documentation on how to build a container image, deploy it into the Kubernetes cluster, and test that all connectivity is secure.

Even better, you can create a “Clone This Starter App” template in your code repository. This gives developers a preconfigured, security-blessed starting point, complete with Dockerfiles and pipeline scripts, that they can immediately fork to begin tinkering, dramatically lowering the barrier to entry while enforcing best practices from the outset.

Additionally, many real life GenAI use cases will inevitably involve some form of PII processing, or can produce outputs that will materially impact individuals such as your employees, your consumers, or your customers’ staff. In such cases, more and more state, federal and international laws require completing various risk assessments before actual work can begin. These assessments can be cumbersome to conduct and are difficult to scale if they are carried out case by case. The “Clone This Starter App” approach also helps to prevent such compliance mandates from becoming bottlenecks to innovation, since under most legal mandates, the required assessments can be completed once for your template, and they need not be repeated for any clone that doesn’t exceed your initially defined parameters.

Recipe step 3: Deploy user applications

With your policies defined, your security baseline established, and your developer templates in place, it’s finally time to serve the dish. Whether you've chosen a self-service or managed deployment model, you can now confidently invite the rapid prototypers in your organization to start creating in the sandbox.

Because you’ve included APM and security logging (ingredient #8) from the beginning, you have the necessary observability to monitor application performance and user activity. This is where the magic happens: You can now learn from the applications people build, identify powerful new use cases, and gather real-world data to improve the platform, all while safeguarding organizational data. Coincidentally, this approach will also allow you to organically collect the information you might need to put on record, disclose to users, or share with auditors and regulators to demonstrate the transparency, accountability and explainability of your GenAI application, ticking many compliance boxes as you build (and not after the fact) – a textbook best practice of Privacy by Design.

Where do you go from here?

We've now walked through the entire cookbook, from selecting your fresh ingredients to following the recipe step by step. Most of the domains we've discussed (containerization, APM, IAM, and more) are culinary specialties in and of themselves.

Conclusion

This cookbook was designed to provide a clear recipe for building a production-grade GenAI sandbox. By carefully selecting each foundational ingredient, from your containerization platform and Kubernetes orchestration to your search AI datastore and unified APM, you ensure your final dish will be both successful and secure. Following the recipe ensures that this powerful environment is built on a foundation of security and thoughtful policy from day one.

The goal is to empower your rapid prototypers, not restrict them, and to foster a culture of responsible innovation. By providing a secure, observable, and well-equipped kitchen for experimentation, you get ahead of the curve, fostering a culture of responsible innovation. This proactive approach enables you to harness the creativity of your entire organization, transforming brilliant ideas into tangible prototypes while preventing the rise of shadow AI. You've cooked the meal; now you can enjoy the innovation it serves.

If you want to chat about this or anything else related to Elasticsearch, come join us in our Discuss forum.