Быстрая модель BERT для расчетов эмбеддингов предложений на русском языке. Модель основана на cointegrated/rubert-tiny2 - имеет аналогичные размеры контекста (2048), ембеддинга (312) и быстродействие.
Использование
from sentence_transformers import SentenceTransformer, util
model = SentenceTransformer('sergeyzh/rubert-tiny-turbo')
sentences = ["привет мир", "hello world", "здравствуй вселенная"]
embeddings = model.encode(sentences)
print(util.dot_score(embeddings, embeddings))
Метрики
Оценки модели на бенчмарке encodechka:
model | CPU | GPU | size | Mean S | Mean S+W | dim |
---|---|---|---|---|---|---|
sergeyzh/LaBSE-ru-turbo | 120.40 | 8.05 | 490 | 0.789 | 0.702 | 768 |
BAAI/bge-m3 | 523.40 | 22.50 | 2166 | 0.787 | 0.696 | 1024 |
intfloat/multilingual-e5-large | 506.80 | 30.80 | 2136 | 0.780 | 0.686 | 1024 |
intfloat/multilingual-e5-base | 130.61 | 14.39 | 1061 | 0.761 | 0.669 | 768 |
sergeyzh/rubert-tiny-turbo | 5.51 | 3.25 | 111 | 0.749 | 0.667 | 312 |
intfloat/multilingual-e5-small | 40.86 | 12.09 | 449 | 0.742 | 0.645 | 384 |
cointegrated/rubert-tiny2 | 5.51 | 3.25 | 111 | 0.704 | 0.638 | 312 |
model | STS | PI | NLI | SA | TI | IA | IC | ICX | NE1 | NE2 |
---|---|---|---|---|---|---|---|---|---|---|
sergeyzh/LaBSE-ru-turbo | 0.864 | 0.748 | 0.490 | 0.814 | 0.974 | 0.806 | 0.815 | 0.801 | 0.305 | 0.404 |
BAAI/bge-m3 | 0.864 | 0.749 | 0.510 | 0.819 | 0.973 | 0.792 | 0.809 | 0.783 | 0.240 | 0.422 |
intfloat/multilingual-e5-large | 0.862 | 0.727 | 0.473 | 0.810 | 0.979 | 0.798 | 0.819 | 0.773 | 0.224 | 0.374 |
intfloat/multilingual-e5-base | 0.835 | 0.704 | 0.459 | 0.796 | 0.964 | 0.783 | 0.802 | 0.738 | 0.235 | 0.376 |
sergeyzh/rubert-tiny-turbo | 0.828 | 0.722 | 0.476 | 0.787 | 0.955 | 0.757 | 0.780 | 0.685 | 0.305 | 0.373 |
intfloat/multilingual-e5-small | 0.822 | 0.714 | 0.457 | 0.758 | 0.957 | 0.761 | 0.779 | 0.691 | 0.234 | 0.275 |
cointegrated/rubert-tiny2 | 0.750 | 0.651 | 0.417 | 0.737 | 0.937 | 0.746 | 0.757 | 0.638 | 0.360 | 0.386 |
Оценки модели на бенчмарке ruMTEB:
Model Name | Metric | sbert_large_ mt_nlu_ru | sbert_large_ nlu_ru | rubert-tiny2 | rubert-tiny-turbo | multilingual-e5-small | multilingual-e5-base | multilingual-e5-large |
---|---|---|---|---|---|---|---|---|
CEDRClassification | Accuracy | 0.368 | 0.358 | 0.369 | 0.390 | 0.401 | 0.423 | 0.448 |
GeoreviewClassification | Accuracy | 0.397 | 0.400 | 0.396 | 0.414 | 0.447 | 0.461 | 0.497 |
GeoreviewClusteringP2P | V-measure | 0.584 | 0.590 | 0.442 | 0.597 | 0.586 | 0.545 | 0.605 |
HeadlineClassification | Accuracy | 0.772 | 0.793 | 0.742 | 0.686 | 0.732 | 0.757 | 0.758 |
InappropriatenessClassification | Accuracy | 0.646 | 0.625 | 0.586 | 0.591 | 0.592 | 0.588 | 0.616 |
KinopoiskClassification | Accuracy | 0.503 | 0.495 | 0.491 | 0.505 | 0.500 | 0.509 | 0.566 |
RiaNewsRetrieval | NDCG@10 | 0.214 | 0.111 | 0.140 | 0.513 | 0.700 | 0.702 | 0.807 |
RuBQReranking | MAP@10 | 0.561 | 0.468 | 0.461 | 0.622 | 0.715 | 0.720 | 0.756 |
RuBQRetrieval | NDCG@10 | 0.298 | 0.124 | 0.109 | 0.517 | 0.685 | 0.696 | 0.741 |
RuReviewsClassification | Accuracy | 0.589 | 0.583 | 0.570 | 0.607 | 0.612 | 0.630 | 0.653 |
RuSTSBenchmarkSTS | Pearson correlation | 0.712 | 0.588 | 0.694 | 0.787 | 0.781 | 0.796 | 0.831 |
RuSciBenchGRNTIClassification | Accuracy | 0.542 | 0.539 | 0.456 | 0.529 | 0.550 | 0.563 | 0.582 |
RuSciBenchGRNTIClusteringP2P | V-measure | 0.522 | 0.504 | 0.414 | 0.481 | 0.511 | 0.516 | 0.520 |
RuSciBenchOECDClassification | Accuracy | 0.438 | 0.430 | 0.355 | 0.415 | 0.427 | 0.423 | 0.445 |
RuSciBenchOECDClusteringP2P | V-measure | 0.473 | 0.464 | 0.381 | 0.411 | 0.443 | 0.448 | 0.450 |
SensitiveTopicsClassification | Accuracy | 0.285 | 0.280 | 0.220 | 0.244 | 0.228 | 0.234 | 0.257 |
TERRaClassification | Average Precision | 0.520 | 0.502 | 0.519 | 0.563 | 0.551 | 0.550 | 0.584 |
Model Name | Metric | sbert_large_ mt_nlu_ru | sbert_large_ nlu_ru | rubert-tiny2 | rubert-tiny-turbo | multilingual-e5-small | multilingual-e5-base | multilingual-e5-large |
---|---|---|---|---|---|---|---|---|
Classification | Accuracy | 0.554 | 0.552 | 0.514 | 0.535 | 0.551 | 0.561 | 0.588 |
Clustering | V-measure | 0.526 | 0.519 | 0.412 | 0.496 | 0.513 | 0.503 | 0.525 |
MultiLabelClassification | Accuracy | 0.326 | 0.319 | 0.294 | 0.317 | 0.314 | 0.329 | 0.353 |
PairClassification | Average Precision | 0.520 | 0.502 | 0.519 | 0.563 | 0.551 | 0.550 | 0.584 |
Reranking | MAP@10 | 0.561 | 0.468 | 0.461 | 0.622 | 0.715 | 0.720 | 0.756 |
Retrieval | NDCG@10 | 0.256 | 0.118 | 0.124 | 0.515 | 0.697 | 0.699 | 0.774 |
STS | Pearson correlation | 0.712 | 0.588 | 0.694 | 0.787 | 0.781 | 0.796 | 0.831 |
Average | Average | 0.494 | 0.438 | 0.431 | 0.548 | 0.588 | 0.594 | 0.630 |
- Downloads last month
- 47,444
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.
Model tree for sergeyzh/rubert-tiny-turbo
Base model
cointegrated/rubert-tiny2Datasets used to train sergeyzh/rubert-tiny-turbo
Space using sergeyzh/rubert-tiny-turbo 1
Evaluation results
- main_score on MTEB AILACasedocs (default)test set self-reported7.433
- map_at_1 on MTEB AILACasedocs (default)test set self-reported0.604
- map_at_10 on MTEB AILACasedocs (default)test set self-reported3.899
- map_at_100 on MTEB AILACasedocs (default)test set self-reported7.890
- map_at_1000 on MTEB AILACasedocs (default)test set self-reported8.417
- map_at_20 on MTEB AILACasedocs (default)test set self-reported5.007
- map_at_3 on MTEB AILACasedocs (default)test set self-reported2.688
- map_at_5 on MTEB AILACasedocs (default)test set self-reported3.038
- mrr_at_1 on MTEB AILACasedocs (default)test set self-reported6.000
- mrr_at_10 on MTEB AILACasedocs (default)test set self-reported11.800