eduardofv commited on
Commit
dfd9817
1 Parent(s): baa3979

Corrected README

Browse files
Files changed (1) hide show
  1. README.md +32 -99
README.md CHANGED
@@ -1,126 +1,59 @@
1
  ---
2
- pipeline_tag: sentence-similarity
 
 
3
  tags:
4
- - sentence-transformers
5
- - feature-extraction
6
  - sentence-similarity
7
- - transformers
8
  ---
9
 
10
- # {MODEL_NAME}
11
-
12
- This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search.
13
-
14
- <!--- Describe your model here -->
15
-
16
- ## Usage (Sentence-Transformers)
17
-
18
- Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
19
-
20
- ```
21
- pip install -U sentence-transformers
22
- ```
23
-
24
- Then you can use the model like this:
25
-
26
- ```python
27
- from sentence_transformers import SentenceTransformer
28
- sentences = ["This is an example sentence", "Each sentence is converted"]
29
-
30
- model = SentenceTransformer('{MODEL_NAME}')
31
- embeddings = model.encode(sentences)
32
- print(embeddings)
33
- ```
34
-
35
 
 
36
 
37
- ## Usage (HuggingFace Transformers)
38
- Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
39
 
40
- ```python
41
- from transformers import AutoTokenizer, AutoModel
42
- import torch
43
 
 
44
 
45
- #Mean Pooling - Take attention mask into account for correct averaging
46
- def mean_pooling(model_output, attention_mask):
47
- token_embeddings = model_output[0] #First element of model_output contains all token embeddings
48
- input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
49
- return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
50
 
 
51
 
52
- # Sentences we want sentence embeddings for
53
- sentences = ['This is an example sentence', 'Each sentence is converted']
54
 
55
- # Load model from HuggingFace Hub
56
- tokenizer = AutoTokenizer.from_pretrained('{MODEL_NAME}')
57
- model = AutoModel.from_pretrained('{MODEL_NAME}')
58
 
59
- # Tokenize sentences
60
- encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
61
 
62
- # Compute token embeddings
63
- with torch.no_grad():
64
- model_output = model(**encoded_input)
65
 
66
- # Perform pooling. In this case, max pooling.
67
- sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
68
 
69
- print("Sentence embeddings:")
70
- print(sentence_embeddings)
71
  ```
72
-
73
-
74
-
75
- ## Evaluation Results
76
-
77
- <!--- Describe how your model was evaluated -->
78
-
79
- For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name={MODEL_NAME})
80
-
81
-
82
- ## Training
83
- The model was trained with the parameters:
84
-
85
- **DataLoader**:
86
-
87
- `torch.utils.data.dataloader.DataLoader` of length 360 with parameters:
88
- ```
89
- {'batch_size': 16, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}
90
  ```
91
 
92
- **Loss**:
93
 
94
- `sentence_transformers.losses.CosineSimilarityLoss.CosineSimilarityLoss`
95
-
96
- Parameters of the fit()-Method:
97
  ```
98
- {
99
- "callback": null,
100
- "epochs": 4,
101
- "evaluation_steps": 1000,
102
- "evaluator": "sentence_transformers.evaluation.EmbeddingSimilarityEvaluator.EmbeddingSimilarityEvaluator",
103
- "max_grad_norm": 1,
104
- "optimizer_class": "<class 'transformers.optimization.AdamW'>",
105
- "optimizer_params": {
106
- "lr": 2e-05
107
- },
108
- "scheduler": "WarmupLinear",
109
- "steps_per_epoch": null,
110
- "warmup_steps": 144,
111
- "weight_decay": 0.01
112
- }
113
  ```
114
 
 
115
 
116
- ## Full Model Architecture
117
- ```
118
- SentenceTransformer(
119
- (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: DistilBertModel
120
- (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
121
- )
122
- ```
123
 
124
- ## Citing & Authors
125
 
126
- <!--- Describe where people can find more information -->
 
 
 
 
1
  ---
2
+ language: es
3
+ datasets:
4
+ - stsb_multi_mt
5
  tags:
 
 
6
  - sentence-similarity
7
+ - sentence-transformers
8
  ---
9
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
10
 
11
+ This is a test model that was fine-tuned using the Spanish datasets from [stsb_multi_mt](https://huggingface.co/datasets/stsb_multi_mt) in order to understand and benchmark STS models.
12
 
13
+ ## Model and training data description
 
14
 
15
+ This model was built taking `distiluse-base-multilingual-cased-v1` and training it on a Semantic Textual Similarity task using a modified version of the training script for STS from Sentece Transformers (the modified script is included in the repo). It was trained using the Spanish datasets from [stsb_multi_mt](https://huggingface.co/datasets/stsb_multi_mt) which are the STSBenchmark datasets automatically translated to other languages using deepl.com. Refer to the dataset repository for more details.
 
 
16
 
17
+ ## Intended uses & limitations
18
 
19
+ This model was built just as a proof-of-concept on STS fine-tuning using Spanish data and no specific use other than getting a sense on how this training works.
 
 
 
 
20
 
21
+ ## How to use
22
 
23
+ You may use it as any other STS trained model to extract sentence embeddings. Check Sentence Transformers documentation.
 
24
 
25
+ ## Training procedure
 
 
26
 
27
+ This model was trained using this [Colab Notebook](https://colab.research.google.com/drive/1ZNjDMFdy_lKhnD9BtbqzSbQ4LNz638ZA?usp=sharing)
 
28
 
29
+ ## Evaluation results
 
 
30
 
31
+ Evaluating `distiluse-base-multilingual-cased-v1` on the Spanish test dataset before training results in:
 
32
 
 
 
33
  ```
34
+ 2021-07-06 17:44:46 - EmbeddingSimilarityEvaluator: Evaluating the model on dataset:
35
+ 2021-07-06 17:45:00 - Cosine-Similarity : Pearson: 0.7662 Spearman: 0.7583
36
+ 2021-07-06 17:45:00 - Manhattan-Distance: Pearson: 0.7805 Spearman: 0.7772
37
+ 2021-07-06 17:45:00 - Euclidean-Distance: Pearson: 0.7816 Spearman: 0.7778
38
+ 2021-07-06 17:45:00 - Dot-Product-Similarity: Pearson: 0.6610 Spearman: 0.6536
 
 
 
 
 
 
 
 
 
 
 
 
 
39
  ```
40
 
41
+ While the fine-tuned version with the defaults of the training script and the Spanish training dataset results in:
42
 
 
 
 
43
  ```
44
+ 2021-07-06 17:49:22 - EmbeddingSimilarityEvaluator: Evaluating the model on stsb-multi-mt-test dataset:
45
+ 2021-07-06 17:49:24 - Cosine-Similarity : Pearson: 0.8265 Spearman: 0.8207
46
+ 2021-07-06 17:49:24 - Manhattan-Distance: Pearson: 0.8131 Spearman: 0.8190
47
+ 2021-07-06 17:49:24 - Euclidean-Distance: Pearson: 0.8129 Spearman: 0.8190
48
+ 2021-07-06 17:49:24 - Dot-Product-Similarity: Pearson: 0.7773 Spearman: 0.7692
 
 
 
 
 
 
 
 
 
 
49
  ```
50
 
51
+ In our [STS Evaluation repository](https://github.com/eduardofv/sts_eval) we compare the performance of this model with other models from Sentence Transformers and Tensorflow Hub using the standard STSBenchmark and the 2017 STSBenchmark Task 3 for Spanish.
52
 
 
 
 
 
 
 
 
53
 
54
+ ## Resources
55
 
56
+ - Training dataset [stsb_multi_mt](https://huggingface.co/datasets/stsb_multi_mt)
57
+ - Sentence Transformers [Semantic Textual Similarity](https://www.sbert.net/examples/training/sts/README.html)
58
+ - Check [sts_eval](https://github.com/eduardofv/sts_eval) for a comparison with Tensorflow and Sentence-Transformers models
59
+ - Check the [development environment to run the scripts and evaluation](https://github.com/eduardofv/ai-denv)