opensearch-project
/

opensearch-neural-sparse-encoding-doc-v1

@@ -14,9 +14,23 @@ tags:
 # opensearch-neural-sparse-encoding-v1
 This is a learned sparse retrieval model. It encodes the documents to 30522 dimensional **sparse vectors**. For queries, it just use a tokenizer and a weight look-up table to generate sparse vectors. The non-zero dimension index means the corresponding token in the vocabulary, and the weight means the importance of the token. And the similarity score is the inner product of query/document sparse vectors. In the real-world use case, the search performance of opensearch-neural-sparse-encoding-v1 is comparable to BM25.
 OpenSearch neural sparse feature supports learned sparse retrieval with lucene inverted index. Link: https://opensearch.org/docs/latest/query-dsl/specialized/neural-sparse/. The indexing and search can be performed with OpenSearch high-level API.
 ## Usage (HuggingFace)
 This model is supposed to run inside OpenSearch cluster. But you can also use it outside the cluster, with HuggingFace models API.
@@ -116,5 +130,21 @@ for token in sorted(query_token_weight, key=lambda x:query_token_weight[x], reve
 The above code sample shows an example of neural sparse search. Although there is no overlap token in original query and document, but this model performs a good match.
-## Performance
-This model is trained on MS MARCO dataset. The search relevance score of it can be found here (Neural sparse search document-only) https://opensearch.org/blog/improving-document-retrieval-with-sparse-semantic-encoders/.

 # opensearch-neural-sparse-encoding-v1
 This is a learned sparse retrieval model. It encodes the documents to 30522 dimensional **sparse vectors**. For queries, it just use a tokenizer and a weight look-up table to generate sparse vectors. The non-zero dimension index means the corresponding token in the vocabulary, and the weight means the importance of the token. And the similarity score is the inner product of query/document sparse vectors. In the real-world use case, the search performance of opensearch-neural-sparse-encoding-v1 is comparable to BM25.
+This model is trained on MS MARCO dataset.
 OpenSearch neural sparse feature supports learned sparse retrieval with lucene inverted index. Link: https://opensearch.org/docs/latest/query-dsl/specialized/neural-sparse/. The indexing and search can be performed with OpenSearch high-level API.
+## Select the model
+The model should be selected considering search relevance, model inference and retrieval efficiency(FLOPS). We benchmark models' **zero-shot performance** on a subset of BEIR benchmark: TrecCovid,NFCorpus,NQ,HotpotQA,FiQA,ArguAna,Touche,DBPedia,SCIDOCS,FEVER,Climate FEVER,SciFact,Quora.
+Overall, the v2 series of models have better search relevance, efficiency and inference speed than the v1 series. The specific advantages and disadvantages may vary across different datasets.
+| Model | Inference-free for Retrieval | Model Parameters | AVG NDCG@10 | AVG FLOPS |
+|-------|------------------------------|------------------|-------------|-----------|
+| [opensearch-neural-sparse-encoding-v1](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-v1) |  | 133M | 0.524 | 11.4 |
+| [opensearch-neural-sparse-encoding-v2-distill](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-v2-distill) |  | 67M | 0.528 | 8.3 |
+| [opensearch-neural-sparse-encoding-doc-v1](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-doc-v1) | ✔️ | 133M | 0.490 | 2.3 |
+| [opensearch-neural-sparse-encoding-doc-v2-distill](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-doc-v2-distill) | ✔️ | 67M | 0.504 | 1.8 |
 ## Usage (HuggingFace)
 This model is supposed to run inside OpenSearch cluster. But you can also use it outside the cluster, with HuggingFace models API.
 The above code sample shows an example of neural sparse search. Although there is no overlap token in original query and document, but this model performs a good match.
+## Detailed Search Relevance
+| Dataset | [opensearch-neural-sparse-encoding-v1](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-v1) | [opensearch-neural-sparse-encoding-v2-distill](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-v2-distill) | [opensearch-neural-sparse-encoding-doc-v1](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-doc-v1) | [opensearch-neural-sparse-encoding-doc-v2-distill](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-doc-v2-distill) |
+|---------|-------------------------------------------------------------------------|-------------------------------------------------------------------------------------|------------------------------------------------------------------------------|------------------------------------------------------------------------------------------|
+| Trec Covid | 0.771 | 0.775 | 0.707 | 0.690 |
+| NFCorpus | 0.360 | 0.347 | 0.352 | 0.343 |
+| NQ | 0.553 | 0.561 | 0.521 | 0.528 |
+| HotpotQA | 0.697 | 0.685 | 0.677 | 0.675 |
+| FiQA | 0.376 | 0.374 | 0.344 | 0.357 |
+| ArguAna | 0.508 | 0.551 | 0.461 | 0.496 |
+| Touche | 0.278 | 0.278 | 0.294 | 0.287 |
+| DBPedia | 0.447 | 0.435 | 0.412 | 0.418 |
+| SCIDOCS | 0.164 | 0.173 | 0.154 | 0.166 |
+| FEVER | 0.821 | 0.849 | 0.743 | 0.818 |
+| Climate FEVER | 0.263 | 0.249 | 0.202 | 0.224 |
+| SciFact | 0.723 | 0.722 | 0.716 | 0.715 |
+| Quora | 0.856 | 0.863 | 0.788 | 0.841 |
+| **Average** | **0.524** | **0.528** | **0.490** | **0.504** |