ELASTICSEARCH-VECTOR

Elasticsearch vector dense_vector HNSW Lucene.

Définition

Elasticsearch vector features : (1) dense_vector field type : up to 4096 dimensions, similarity (cosine, dot_product, l2_norm), index_options (HNSW parameters m, ef_construction). (2) HNSW index : Lucene native implementation Elasticsearch 8.0+, similar OpenSearch Lucene engine. (3) KNN search : knn query type '{knn: {field: 'embedding', query_vector: [...], k: 10, num_candidates: 100}}'. (4) Hybrid search : combine knn + match + range queries, RRF Reciprocal Rank Fusion (8.8+) for combining multiple result sets. (5) Filtering : pre-filtering during HNSW traversal (efficient). (6) Aggregations : compatible with vector search results, integrate analytics. (7) ELSER : Elastic Learned Sparse EncodeR (Elastic-trained sparse encoder model, alternative dense vectors, BM25-style sparse term matching). (8) Quantization : int8 quantization 4x compression 8.12+. Pricing : Elastic Cloud managed service variable per node + storage, vector search additional resources typical. Customers : Microsoft Bing partial, eBay, Wikimedia, ~10000+ Elasticsearch enterprise customers. License complex : SSPL + Elastic License v2 since 2021 (no longer Apache-2.0).

Origine

Elasticsearch dense_vector field added 7.3 2019 (exact KNN only) ; HNSW ANN added 8.0 GA fevrier 2022 ; ELSER added 8.8 2023 ; int8 quantization 8.12 fevrier 2024.

Exemple en contexte

Wikimedia Foundation uses Elasticsearch 8.x with vector search for Wikipedia semantic search : ~50M articles embedded via multilingual Sentence Transformers model (768 dim), HNSW + BM25 hybrid search RRF, 200+ language support, integrate existing Wikipedia Elasticsearch infrastructure.

Termes liés

• OpenSearch k-NN — fork AWS Apache 2.0.