Hugging Face's logo

botbot-ai
/
CabraLlama3-8b

Text Generation
Transformers
Safetensors
GGUF
Portuguese
English
llama
text-generation-inference
brazil
brasil
8b
portuguese
conversational
Eval Results
Inference Endpoints
Model card Files Community
2
nicolasdec commited on
Commit
489c6fd
•
1 Parent(s): 2d7411f

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +165 -1
README.md CHANGED
@@ -14,7 +14,153 @@ tags:
14
  - portuguese
15
  base_model: meta-llama/Meta-Llama-3-8B-Instruct
16
  pipeline_tag: text-generation
17
-
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
18
  ---
19
  # Cabra Llama-3 8B
20
  <img src="https://uploads-ssl.webflow.com/65f77c0240ae1c68f8192771/6627fb36d544dd8ea250136a_llama38b.png" width="300" height="300">
@@ -87,3 +233,21 @@ O modelo é destinado, por agora, a fins de pesquisa. As áreas e tarefas de pes
87
  |portuguese_hate_speech_binary|f1_macro|0.5982 |0.0120|0.5954 |0.0120|
88
  | |acc |0.5993 |0.0119|0.5993 |0.0119|
89
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
14
  - portuguese
15
  base_model: meta-llama/Meta-Llama-3-8B-Instruct
16
  pipeline_tag: text-generation
17
+ model-index:
18
+ - name: CabraLlama3-8b
19
+ results:
20
+ - task:
21
+ type: text-generation
22
+ name: Text Generation
23
+ dataset:
24
+ name: ENEM Challenge (No Images)
25
+ type: eduagarcia/enem_challenge
26
+ split: train
27
+ args:
28
+ num_few_shot: 3
29
+ metrics:
30
+ - type: acc
31
+ value: 74.67
32
+ name: accuracy
33
+ source:
34
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=botbot-ai/CabraLlama3-8b
35
+ name: Open Portuguese LLM Leaderboard
36
+ - task:
37
+ type: text-generation
38
+ name: Text Generation
39
+ dataset:
40
+ name: BLUEX (No Images)
41
+ type: eduagarcia-temp/BLUEX_without_images
42
+ split: train
43
+ args:
44
+ num_few_shot: 3
45
+ metrics:
46
+ - type: acc
47
+ value: 56.88
48
+ name: accuracy
49
+ source:
50
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=botbot-ai/CabraLlama3-8b
51
+ name: Open Portuguese LLM Leaderboard
52
+ - task:
53
+ type: text-generation
54
+ name: Text Generation
55
+ dataset:
56
+ name: OAB Exams
57
+ type: eduagarcia/oab_exams
58
+ split: train
59
+ args:
60
+ num_few_shot: 3
61
+ metrics:
62
+ - type: acc
63
+ value: 49.29
64
+ name: accuracy
65
+ source:
66
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=botbot-ai/CabraLlama3-8b
67
+ name: Open Portuguese LLM Leaderboard
68
+ - task:
69
+ type: text-generation
70
+ name: Text Generation
71
+ dataset:
72
+ name: Assin2 RTE
73
+ type: assin2
74
+ split: test
75
+ args:
76
+ num_few_shot: 15
77
+ metrics:
78
+ - type: f1_macro
79
+ value: 90.44
80
+ name: f1-macro
81
+ source:
82
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=botbot-ai/CabraLlama3-8b
83
+ name: Open Portuguese LLM Leaderboard
84
+ - task:
85
+ type: text-generation
86
+ name: Text Generation
87
+ dataset:
88
+ name: Assin2 STS
89
+ type: eduagarcia/portuguese_benchmark
90
+ split: test
91
+ args:
92
+ num_few_shot: 15
93
+ metrics:
94
+ - type: pearson
95
+ value: 69.85
96
+ name: pearson
97
+ source:
98
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=botbot-ai/CabraLlama3-8b
99
+ name: Open Portuguese LLM Leaderboard
100
+ - task:
101
+ type: text-generation
102
+ name: Text Generation
103
+ dataset:
104
+ name: FaQuAD NLI
105
+ type: ruanchaves/faquad-nli
106
+ split: test
107
+ args:
108
+ num_few_shot: 15
109
+ metrics:
110
+ - type: f1_macro
111
+ value: 70.38
112
+ name: f1-macro
113
+ source:
114
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=botbot-ai/CabraLlama3-8b
115
+ name: Open Portuguese LLM Leaderboard
116
+ - task:
117
+ type: text-generation
118
+ name: Text Generation
119
+ dataset:
120
+ name: HateBR Binary
121
+ type: ruanchaves/hatebr
122
+ split: test
123
+ args:
124
+ num_few_shot: 25
125
+ metrics:
126
+ - type: f1_macro
127
+ value: 85.05
128
+ name: f1-macro
129
+ source:
130
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=botbot-ai/CabraLlama3-8b
131
+ name: Open Portuguese LLM Leaderboard
132
+ - task:
133
+ type: text-generation
134
+ name: Text Generation
135
+ dataset:
136
+ name: PT Hate Speech Binary
137
+ type: hate_speech_portuguese
138
+ split: test
139
+ args:
140
+ num_few_shot: 25
141
+ metrics:
142
+ - type: f1_macro
143
+ value: 60.1
144
+ name: f1-macro
145
+ source:
146
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=botbot-ai/CabraLlama3-8b
147
+ name: Open Portuguese LLM Leaderboard
148
+ - task:
149
+ type: text-generation
150
+ name: Text Generation
151
+ dataset:
152
+ name: tweetSentBR
153
+ type: eduagarcia/tweetsentbr_fewshot
154
+ split: test
155
+ args:
156
+ num_few_shot: 25
157
+ metrics:
158
+ - type: f1_macro
159
+ value: 68.08
160
+ name: f1-macro
161
+ source:
162
+ url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=botbot-ai/CabraLlama3-8b
163
+ name: Open Portuguese LLM Leaderboard
164
  ---
165
  # Cabra Llama-3 8B
166
  <img src="https://uploads-ssl.webflow.com/65f77c0240ae1c68f8192771/6627fb36d544dd8ea250136a_llama38b.png" width="300" height="300">
 
233
  |portuguese_hate_speech_binary|f1_macro|0.5982 |0.0120|0.5954 |0.0120|
234
  | |acc |0.5993 |0.0119|0.5993 |0.0119|
235
 
236
+
237
+ # Open Portuguese LLM Leaderboard Evaluation Results
238
+
239
+ Detailed results can be found [here](https://huggingface.co/datasets/eduagarcia-temp/llm_pt_leaderboard_raw_results/tree/main/botbot-ai/CabraLlama3-8b) and on the [🚀 Open Portuguese LLM Leaderboard](https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard)
240
+
241
+ | Metric | Value |
242
+ |--------------------------|---------|
243
+ |Average |**69.42**|
244
+ |ENEM Challenge (No Images)| 74.67|
245
+ |BLUEX (No Images) | 56.88|
246
+ |OAB Exams | 49.29|
247
+ |Assin2 RTE | 90.44|
248
+ |Assin2 STS | 69.85|
249
+ |FaQuAD NLI | 70.38|
250
+ |HateBR Binary | 85.05|
251
+ |PT Hate Speech Binary | 60.10|
252
+ |tweetSentBR | 68.08|
253
+