OPENSEARCH-KNN

OpenSearch k-NN AWS plugin vector + full-text.

Définition

OpenSearch k-NN features : (1) Vector field type : knn_vector with dimension parameter, methods (engine, name HNSW/IVF, space_type L2/InnerProduct/CosineSimilarity), index parameters. (2) Engines : (a) Nmslib HNSW (default, mature, good performance), (b) Faiss IVF + HNSW + IVF-PQ (Facebook AI Faiss library integration, GPU-acceleration possible, advanced quantization), (c) Lucene HNSW (added 2.4+, native Apache Lucene HNSW implementation, less mature mais well-integrated OpenSearch index lifecycle). (3) Query : KNN query DSL '{knn: {embedding: {vector: [...], k: 10}}}'. (4) Hybrid search : combine knn + match (BM25) + filter clauses in bool query, score boosting. (5) Pre-filtering : Lucene engine supports pre-filtering during HNSW traversal (efficient avec selective filters), Nmslib post-filtering only. (6) Quantization : Faiss PQ + Scalar Quantization byte-based. Pricing : AWS OpenSearch Service managed Standard tier pricing par instance type + storage GB-month, vector search additional CPU/memory typical workload. Customers : Pinterest Search, Amazon retail, BMW media search.

Origine

Elasticsearch k-NN initial release 2019 ; OpenSearch fork avril 2021 (AWS after Elastic SSPL license change) ; OpenSearch k-NN AWS-maintained Apache-2.0 ; OpenSearch 2.x active 2024.

Exemple en contexte

Pinterest Visual Search uses OpenSearch k-NN Faiss IVF-PQ engine pour 3B+ images embeddings : Pinterest custom Vision Transformer model embedding 128-dim, IVF-PQ 8x compression, query latency ~100ms for 'shop the look' visual search feature.

Termes liés

• Elasticsearch vector — fork upstream.