Dense vectors and kNN search

When working with vector embeddings for semantic search or machine learning applications, the Go client provides support for dense vector indexing and k-nearest neighbors (kNN) search. This page uses the typed API with the esdsl builders. For the full kNN search reference, see kNN search in the Elasticsearch documentation.

import "github.com/elastic/go-elasticsearch/v9/typedapi/esdsl"
		

Before indexing documents with vectors, create an index with the appropriate dense vector mapping. The esdsl mapping builders provide a concise, fluent syntax:

mappings := esdsl.NewTypeMapping().
    AddProperty("docid", esdsl.NewKeywordProperty()).
    AddProperty("title", esdsl.NewTextProperty()).
    AddProperty("emb", esdsl.NewDenseVectorProperty().
        Dims(1536).
        Index(true).
        Similarity(densevectorsimilarity.Cosine))

res, err := es.Indices.
    Create("my-vectors").
    Mappings(mappings).
    Do(context.Background())
		

1. Define the vector field as a DenseVectorProperty.
2. Specify the dimensionality of your vectors (e.g., 1536 for OpenAI embeddings).
3. Enable indexing to support kNN search capabilities.
4. Set the similarity metric: Cosine, DotProduct, or L2Norm.
5. Pass the mapping builder directly; no Request wrapper needed.

The types.DenseVectorF32 type automatically encodes []float32 vectors as base64 strings during JSON serialization, reducing payload size and improving indexing speed:

type Document struct {
    DocID string               `json:"docid"`
    Title string               `json:"title"`
    Emb   types.DenseVectorF32 `json:"emb"`
}

document := Document{
    DocID: "doc1",
    Title: "Example document with vector embedding",
    Emb:   types.DenseVectorF32{0.1, 0.2, 0.3, 0.4, 0.5},
}

res, err := es.Index("my-vectors").
    Document(document).
    Do(context.Background())
		

1. Use types.DenseVectorF32 instead of []float32 for vector fields.
2. Assign float32 slices directly; base64 encoding happens automatically during serialization.

Once your vectors are indexed, you can perform k-nearest neighbors (kNN) search to find similar documents. The esdsl query builders provide a fluent syntax for kNN search:

queryVector := []float32{0.1, 0.2, 0.3, 0.4, 0.5}

res, err := es.Search().
    Index("my-vectors").
    Query(
        esdsl.NewKnnQuery().
            Field("emb").
            QueryVector(queryVector...).
            K(10).
            NumCandidates(100),
    ).
    Do(context.Background())
		

1. Define your query vector (typically from the same embedding model).
2. Use esdsl.NewKnnQuery() to build a kNN query with a fluent chain.
3. Specify which vector field to search.
4. Provide the query vector to find similar vectors.
5. Return the top 10 nearest neighbors.
6. Consider 100 candidates during the search for better accuracy.

Using types.DenseVectorF32 provides significant performance improvements over standard JSON arrays of floats:

• Reduced payload size: base64 encoding is more compact than JSON number arrays
• Faster parsing: Eliminates JSON number parsing overhead
• Improved indexing speed: Performance gains increase with vector dimensionality and can improve indexing speeds by up to 3x

If you already have pre-encoded vector bytes from another system, use types.DenseVectorBytes:

type Document struct {
    Emb types.DenseVectorBytes `json:"emb"`
}

vectorBytes := []byte{...}
document := Document{
    Emb: types.DenseVectorBytes(vectorBytes),
}

res, err := es.Index("my-vectors").
    Document(document).
    Do(context.Background())
		

1. Use types.DenseVectorBytes when you have pre-encoded bytes.
2. Provide the raw byte representation of your vector data.