Update to v0.2 checkpoint

#14
by loubnabnl HF staff - opened
README.md CHANGED
@@ -1,12 +1,21 @@
1
  ---
2
- library_name: transformers
3
  license: apache-2.0
 
 
 
 
 
 
 
 
 
 
 
4
  language:
5
  - en
6
  ---
7
 
8
-
9
- # SmolLM-Instruct
10
 
11
  <center>
12
  <img src="https://huggingface.co/datasets/HuggingFaceTB/images/resolve/main/banner_smol.png" alt="SmolLM" width="1100" height="600">
@@ -15,14 +24,35 @@ language:
15
 
16
  ## Model Summary
17
 
18
- SmolLM is a series of state-of-the-art small language models available in three sizes: 135M, 360M, and 1.7B parameters. These models are built on Cosmo-Corpus, a meticulously curated high-quality training dataset. Cosmo-Corpus includes Cosmopedia v2 (28B tokens of synthetic textbooks and stories generated by Mixtral), Python-Edu (4B tokens of educational Python samples from The Stack), and FineWeb-Edu (220B tokens of deduplicated educational web samples from FineWeb). For further details, we refer to our [blogpost](https://huggingface.co/blog/smollm).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
19
 
20
- To build SmolLM-Instruct, we instruction tuned the models using publicly available permissive instruction datasets. We trained all three models for one epoch on the permissive subset of the WebInstructSub dataset, combined with StarCoder2-Self-OSS-Instruct. Following this, we performed DPO (Direct Preference Optimization) for one epoch: using HelpSteer for the 135M and 1.7B models, and argilla/dpo-mix-7k for the 360M model. We followed the training parameters from the Zephyr-Gemma recipe in the alignment handbook, but adjusted the SFT (Supervised Fine-Tuning) learning rate to 3e-4.
21
- [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
 
 
 
 
22
 
23
- This is the SmolLM-1.7B-Instruct.
 
24
 
25
- ### Generation
 
 
26
  ```bash
27
  pip install transformers
28
  ```
@@ -37,18 +67,35 @@ tokenizer = AutoTokenizer.from_pretrained(checkpoint)
37
  # for multiple GPUs install accelerate and do `model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto")`
38
  model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)
39
 
40
- messages = [{"role": "user", "content": "List the steps to bake a chocolate cake from scratch."}]
41
  input_text=tokenizer.apply_chat_template(messages, tokenize=False)
42
  print(input_text)
43
  inputs = tokenizer.encode(input_text, return_tensors="pt").to(device)
44
- outputs = model.generate(inputs, max_new_tokens=100, temperature=0.6, top_p=0.92, do_sample=True)
45
  print(tokenizer.decode(outputs[0]))
46
  ```
47
 
 
 
 
 
 
 
 
 
 
 
 
 
 
48
 
49
- # Limitations
 
 
 
 
50
 
51
- While SmolLM models have been trained on a diverse dataset including educational content and synthetic texts, they have limitations. The models primarily understand and generate content in English. They can produce text on a variety of topics, but the generated content may not always be factually accurate, logically consistent, or free from biases present in the training data. These models should be used as assistive tools rather than definitive sources of information. Users should always verify important information and critically evaluate any generated content. For a more comprehensive discussion of the models' capabilities and limitations, please refer to our full blog post.
52
 
53
  # Citation
54
  ```bash
 
1
  ---
 
2
  license: apache-2.0
3
+ base_model: HuggingFaceTB/SmolLM-1.7B
4
+ tags:
5
+ - alignment-handbook
6
+ - trl
7
+ - sft
8
+ datasets:
9
+ - Magpie-Align/Magpie-Pro-300K-Filtered
10
+ - bigcode/self-oss-instruct-sc2-exec-filter-50k
11
+ - teknium/OpenHermes-2.5
12
+ - HuggingFaceTB/everyday-conversations-llama3.1-2k
13
+ library_name: transformers
14
  language:
15
  - en
16
  ---
17
 
18
+ # SmolLM-1.7B-Instruct
 
19
 
20
  <center>
21
  <img src="https://huggingface.co/datasets/HuggingFaceTB/images/resolve/main/banner_smol.png" alt="SmolLM" width="1100" height="600">
 
24
 
25
  ## Model Summary
26
 
27
+ SmolLM is a series of small language models available in three sizes: 135M, 360M, and 1.7B parameters.
28
+
29
+ These models are pre-trained on [SmolLM-Corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus), a curated collection of high-quality educational and synthetic data designed for training LLMs. For further details, we refer to our [blogpost](https://huggingface.co/blog/smollm).
30
+
31
+ To build SmolLM-Instruct, we finetuned the base models on publicly available datasets.
32
+
33
+ ## Changelog
34
+
35
+
36
+ |Release|Description|
37
+ |-|-|
38
+ |v0.1| Initial release of SmolLM-Instruct. We finetune on the permissive subset of the [WebInstructSub](https://huggingface.co/datasets/TIGER-Lab/WebInstructSub) dataset, combined with [StarCoder2-Self-OSS-Instruct](https://huggingface.co/datasets/bigcode/self-oss-instruct-sc2-exec-filter-50k). Then, we perform DPO (Direct Preference Optimization) for one epoch on [HelpSteer](https://huggingface.co/datasets/nvidia/HelpSteer) for the 135M and 1.7B models, and [argilla/dpo-mix-7k](https://huggingface.co/datasets/argilla/dpo-mix-7k) for the 360M model.|
39
+ |v0.2| We changed the finetuning mix to datasets more suitable for smol models. We train on a new dataset of 2k simple everyday conversations we generated by llama3.1-70B [everyday-conversations-llama3.1-2k](https://huggingface.co/datasets/HuggingFaceTB/everyday-conversations-llama3.1-2k/), [Magpie-Pro-300K-Filtered](https://huggingface.co/datasets/Magpie-Align/Magpie-Pro-300K-Filtered), [StarCoder2-Self-OSS-Instruct](https://huggingface.co/datasets/bigcode/self-oss-instruct-sc2-exec-filter-50k), and a small subset of [OpenHermes-2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5)|
40
+
41
+ v0.2 models are better at staying on topic and responding appropriately to standard prompts, such as greetings and questions about their role as AI assistants. SmolLM-360M-Instruct (v0.2) has a 63.3% win rate over SmolLM-360M-Instruct (v0.1) on AlpacaEval. You can find the details [here](https://huggingface.co/datasets/HuggingFaceTB/alpaca_eval_details/).
42
 
43
+ You can load v0.1 checkpoint by specifying `revision="v0.1"` in the transformers code:
44
+ ```python
45
+ model = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/SmolLM-1.7B-Instruct", revision="v0.1")
46
+ ```
47
+
48
+ ## Usage
49
 
50
+ ### Local Applications
51
+ ⚡ For local applications, you can find optimized implementations of the model in MLC, GGUF and Transformers.js formats, in addition to fast in-browser demos in this collection: https://huggingface.co/collections/HuggingFaceTB/local-smollms-66c0f3b2a15b4eed7fb198d0
52
 
53
+ We noticed that 4bit quantization degrades the quality of the 135M and 360M, so we use `q016` for MLC and ONNX/Transformers.js checkpoints for the WebGPU demos. We also suggest using temperature 0.2 and top-p 0.9.
54
+
55
+ ### Transformers
56
  ```bash
57
  pip install transformers
58
  ```
 
67
  # for multiple GPUs install accelerate and do `model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto")`
68
  model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)
69
 
70
+ messages = [{"role": "user", "content": "What is the capital of France."}]
71
  input_text=tokenizer.apply_chat_template(messages, tokenize=False)
72
  print(input_text)
73
  inputs = tokenizer.encode(input_text, return_tensors="pt").to(device)
74
+ outputs = model.generate(inputs, max_new_tokens=50, temperature=0.2, top_p=0.9, do_sample=True)
75
  print(tokenizer.decode(outputs[0]))
76
  ```
77
 
78
+ ### Chat in TRL
79
+ You can also use the TRL CLI to chat with the model from the terminal:
80
+ ```bash
81
+ pip install trl
82
+ trl chat --model_name_or_path HuggingFaceTB/SmolLM-1.7B-Instruct --device cpu
83
+ ```
84
+
85
+ ## Limitations
86
+
87
+ Additionally, the generated content may not always be factually accurate, logically consistent, or free from biases present in the training data, we invite users to leverage them as assistive tools rather than definitive sources of information. We find that they can handle general knowledge questions, creative writing and basic Python programming. But they are English only and may have difficulty with arithmetics, editing tasks and complex reasoning. For more details about the models' capabilities, please refer to our [blog post](https://huggingface.co/blog/smollm).
88
+
89
+ ## Training parameters
90
+ We train the models using the [alignment-handbook](https://github.com/huggingface/alignment-handbook) with the datasets mentioned in the changelog, using these parameters v0.2:
91
 
92
+ - 1 epoch
93
+ - lr 1e-3
94
+ - cosine schedule
95
+ - warmup ratio 0.1
96
+ - global batch size 262k tokens
97
 
98
+ You can find the training recipe here: https://github.com/huggingface/alignment-handbook/tree/smollm/recipes/smollm
99
 
100
  # Citation
101
  ```bash
all_results.json DELETED
@@ -1,22 +0,0 @@
1
- {
2
- "epoch": 0.9872958257713249,
3
- "eval_logits/chosen": 0.9550817608833313,
4
- "eval_logits/rejected": 0.8631864190101624,
5
- "eval_logps/chosen": -121.98506164550781,
6
- "eval_logps/rejected": -149.43112182617188,
7
- "eval_loss": 0.6671856641769409,
8
- "eval_rewards/accuracies": 0.550000011920929,
9
- "eval_rewards/chosen": -0.0465608611702919,
10
- "eval_rewards/margins": 0.04670196771621704,
11
- "eval_rewards/rejected": -0.09326283633708954,
12
- "eval_runtime": 4.6203,
13
- "eval_samples": 464,
14
- "eval_samples_per_second": 100.425,
15
- "eval_steps_per_second": 3.247,
16
- "total_flos": 0.0,
17
- "train_loss": 0.6753217893488267,
18
- "train_runtime": 321.7901,
19
- "train_samples": 8806,
20
- "train_samples_per_second": 27.366,
21
- "train_steps_per_second": 0.211
22
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
eval_results.json DELETED
@@ -1,16 +0,0 @@
1
- {
2
- "epoch": 0.9872958257713249,
3
- "eval_logits/chosen": 0.9550817608833313,
4
- "eval_logits/rejected": 0.8631864190101624,
5
- "eval_logps/chosen": -121.98506164550781,
6
- "eval_logps/rejected": -149.43112182617188,
7
- "eval_loss": 0.6671856641769409,
8
- "eval_rewards/accuracies": 0.550000011920929,
9
- "eval_rewards/chosen": -0.0465608611702919,
10
- "eval_rewards/margins": 0.04670196771621704,
11
- "eval_rewards/rejected": -0.09326283633708954,
12
- "eval_runtime": 4.6203,
13
- "eval_samples": 464,
14
- "eval_samples_per_second": 100.425,
15
- "eval_steps_per_second": 3.247
16
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
generation_config.json CHANGED
@@ -3,5 +3,6 @@
3
  "bos_token_id": 1,
4
  "eos_token_id": 2,
5
  "pad_token_id": 2,
 
6
  "transformers_version": "4.42.3"
7
  }
 
3
  "bos_token_id": 1,
4
  "eos_token_id": 2,
5
  "pad_token_id": 2,
6
+ "max_new_tokens": 40,
7
  "transformers_version": "4.42.3"
8
  }
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:9bd664d6e8e96531952daac8360d9260f4e91b287b9ea8ebebf113d25687e55f
3
  size 3422777952
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7108f45b86e4b0c9b0c8f611264b381522e093b366ec0fdf83c0e07149b0c398
3
  size 3422777952
onnx/model.onnx CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:4726fb8174a4b324f7880bf27477195617e68b9e873ca397454246dbff02a203
3
- size 563625
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5e2b596dd6bd22b4337eaf3bf4161313306dfc61d12227dab61a28a1afecd95e
3
+ size 562234
onnx/model.onnx_data CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:63c222e386f91f648ae40f9a99ebc1fe40403a00dc443a9f8b5abf0fc0a9dbc0
3
  size 6845505536
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:43c2ff1f9af866f2fc88d29d9ef6c283c2c3908908bfc45e8b7acb948a4b41bc
3
  size 6845505536
onnx/model_bnb4.onnx CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:2366a4b360655ea43277cd5f2cf8c0883b2b740dbf33b5a933da1bb668c6dd1f
3
- size 1309600577
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a632c41a9cd1bcd7116a75ce4c77c290b1e65532f001107d873c4759462a4d6b
3
+ size 1309599186
onnx/model_fp16.onnx CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:5f16d1593b975e7561250d8a2c502927eb7c989401f553b67a6edcd12757a7e9
3
- size 806094042
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a38d5ebf9f8d3556de0aa6b5c7fde73bbd75b89b603f03eadc5903fb47dac346
3
+ size 578785
onnx/model_fp16.onnx_data CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:b465f6b5cb92208c0be17064b0de448cab9f29387aa3a234cc3a46d5e4ae0463
3
- size 2617245696
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bfb0a58ad2b44ced92f2cb3b5c3c6554b007d210d4fd45b090d0e78156aeb292
3
+ size 3422752768
onnx/model_int8.onnx CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:b5fbaba287e62562085daca45a9119c2ce5a33422bed8ed3288393403cb1eee2
3
- size 1712410874
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c7bcbc362755972f15beb94e1d8a9e4d83e8ccf08e669d40c668061e4d712ac0
3
+ size 1712409483
onnx/model_q4.onnx CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:5bbced44bc9d8d6afa0f7d37e93df11028548129349fa2924a03559366737b0a
3
- size 1410262529
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e80eb151f4c5c50e842bbf338a774436d0f4c3e3a81176d4bfc6760853d8a7a6
3
+ size 1410261138
onnx/model_q4f16.onnx CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:7c389d12651b9f89e7f7efd023eba2a0f9458a819ffebdcafb26c99b0b36052d
3
- size 1108103905
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:54ca7d257e47e478ecb9f1306399aa111305556eabac5485c470294df3273cf9
3
+ size 1108085974
onnx/model_quantized.onnx CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:b5fbaba287e62562085daca45a9119c2ce5a33422bed8ed3288393403cb1eee2
3
- size 1712410874
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c7bcbc362755972f15beb94e1d8a9e4d83e8ccf08e669d40c668061e4d712ac0
3
+ size 1712409483
onnx/model_uint8.onnx CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:d5dbd948d100c1f6893581301e12a7cdf15b327111cb45638f5b043198804788
3
- size 1712410958
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b28a17162551f5165479e10819627426cdb82d3e9dfff339a2d5f651d6a6429c
3
+ size 1712409564
runs/Jul15_23-01-52_ip-26-0-171-168/events.out.tfevents.1721084661.ip-26-0-171-168.1537695.0 DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:f9e82e0a6588823112621478b95e6fc4db5d7bb0750bbc260d45a4f6d5eaa35c
3
- size 10294
 
 
 
 
runs/Jul15_23-01-52_ip-26-0-171-168/events.out.tfevents.1721085183.ip-26-0-171-168.1537695.1 DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:27a6a863f4bf58d6242e050e5e56402f3611cf866ad775259a318e644ad89c3d
3
- size 815
 
 
 
 
runs/Jul15_23-02-15_ip-26-0-175-34/events.out.tfevents.1721084673.ip-26-0-175-34.1294397.0 DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:fee64781b1f43c757e014bacbcdc4b210d3b76ebe5fc1f4a63a28d24172c2419
3
- size 10293
 
 
 
 
runs/Jul15_23-02-15_ip-26-0-175-34/events.out.tfevents.1721085077.ip-26-0-175-34.1294397.1 DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:038d227d30387508a3dd8bb86a277ae6c97ddc940005a81c01cc74689ec17040
3
- size 815
 
 
 
 
train_results.json DELETED
@@ -1,9 +0,0 @@
1
- {
2
- "epoch": 0.9872958257713249,
3
- "total_flos": 0.0,
4
- "train_loss": 0.6753217893488267,
5
- "train_runtime": 321.7901,
6
- "train_samples": 8806,
7
- "train_samples_per_second": 27.366,
8
- "train_steps_per_second": 0.211
9
- }
 
 
 
 
 
 
 
 
 
 
trainer_state.json DELETED
@@ -1,147 +0,0 @@
1
- {
2
- "best_metric": null,
3
- "best_model_checkpoint": null,
4
- "epoch": 0.9872958257713249,
5
- "eval_steps": 100,
6
- "global_step": 68,
7
- "is_hyper_param_search": false,
8
- "is_local_process_zero": true,
9
- "is_world_process_zero": true,
10
- "log_history": [
11
- {
12
- "epoch": 0.014519056261343012,
13
- "grad_norm": 2.5819620488760253,
14
- "learning_rate": 7.142857142857143e-07,
15
- "logits/chosen": 0.902682900428772,
16
- "logits/rejected": 0.8661031723022461,
17
- "logps/chosen": -125.2274169921875,
18
- "logps/rejected": -145.89674377441406,
19
- "loss": 0.6931,
20
- "rewards/accuracies": 0.0,
21
- "rewards/chosen": 0.0,
22
- "rewards/margins": 0.0,
23
- "rewards/rejected": 0.0,
24
- "step": 1
25
- },
26
- {
27
- "epoch": 0.14519056261343014,
28
- "grad_norm": 3.2242486672937045,
29
- "learning_rate": 4.970219740227693e-06,
30
- "logits/chosen": 0.8677890300750732,
31
- "logits/rejected": 0.88941490650177,
32
- "logps/chosen": -120.65478515625,
33
- "logps/rejected": -149.0773468017578,
34
- "loss": 0.6925,
35
- "rewards/accuracies": 0.4097222089767456,
36
- "rewards/chosen": 0.0005513947107829154,
37
- "rewards/margins": -0.0005504629807546735,
38
- "rewards/rejected": 0.001101857633329928,
39
- "step": 10
40
- },
41
- {
42
- "epoch": 0.29038112522686027,
43
- "grad_norm": 4.0763010346326904,
44
- "learning_rate": 4.460299516441777e-06,
45
- "logits/chosen": 1.0478929281234741,
46
- "logits/rejected": 1.0530675649642944,
47
- "logps/chosen": -115.20359802246094,
48
- "logps/rejected": -142.4711456298828,
49
- "loss": 0.6887,
50
- "rewards/accuracies": 0.543749988079071,
51
- "rewards/chosen": -0.007147049065679312,
52
- "rewards/margins": 0.007046631071716547,
53
- "rewards/rejected": -0.014193681068718433,
54
- "step": 20
55
- },
56
- {
57
- "epoch": 0.4355716878402904,
58
- "grad_norm": 2.868995154924774,
59
- "learning_rate": 3.441819734087963e-06,
60
- "logits/chosen": 1.0493364334106445,
61
- "logits/rejected": 1.0137263536453247,
62
- "logps/chosen": -115.47831726074219,
63
- "logps/rejected": -145.4583282470703,
64
- "loss": 0.681,
65
- "rewards/accuracies": 0.53125,
66
- "rewards/chosen": -0.013122054748237133,
67
- "rewards/margins": 0.018696457147598267,
68
- "rewards/rejected": -0.031818509101867676,
69
- "step": 30
70
- },
71
- {
72
- "epoch": 0.5807622504537205,
73
- "grad_norm": 2.4635583297317165,
74
- "learning_rate": 2.1790041121336223e-06,
75
- "logits/chosen": 1.2130175828933716,
76
- "logits/rejected": 1.1768478155136108,
77
- "logps/chosen": -154.19398498535156,
78
- "logps/rejected": -181.88272094726562,
79
- "loss": 0.6762,
80
- "rewards/accuracies": 0.581250011920929,
81
- "rewards/chosen": -0.04483573138713837,
82
- "rewards/margins": 0.041850604116916656,
83
- "rewards/rejected": -0.08668633550405502,
84
- "step": 40
85
- },
86
- {
87
- "epoch": 0.7259528130671506,
88
- "grad_norm": 3.188683944287027,
89
- "learning_rate": 9.994642986290797e-07,
90
- "logits/chosen": 1.3518016338348389,
91
- "logits/rejected": 1.2714142799377441,
92
- "logps/chosen": -131.69985961914062,
93
- "logps/rejected": -161.8300018310547,
94
- "loss": 0.6622,
95
- "rewards/accuracies": 0.6499999761581421,
96
- "rewards/chosen": -0.011605720035731792,
97
- "rewards/margins": 0.06830872595310211,
98
- "rewards/rejected": -0.07991443574428558,
99
- "step": 50
100
- },
101
- {
102
- "epoch": 0.8711433756805808,
103
- "grad_norm": 3.3445679325076476,
104
- "learning_rate": 2.092077387824884e-07,
105
- "logits/chosen": 1.367594838142395,
106
- "logits/rejected": 1.2992292642593384,
107
- "logps/chosen": -121.10174560546875,
108
- "logps/rejected": -139.6844940185547,
109
- "loss": 0.6612,
110
- "rewards/accuracies": 0.612500011920929,
111
- "rewards/chosen": -0.017885591834783554,
112
- "rewards/margins": 0.07065734267234802,
113
- "rewards/rejected": -0.08854293823242188,
114
- "step": 60
115
- },
116
- {
117
- "epoch": 0.9872958257713249,
118
- "step": 68,
119
- "total_flos": 0.0,
120
- "train_loss": 0.6753217893488267,
121
- "train_runtime": 321.7901,
122
- "train_samples_per_second": 27.366,
123
- "train_steps_per_second": 0.211
124
- }
125
- ],
126
- "logging_steps": 10,
127
- "max_steps": 68,
128
- "num_input_tokens_seen": 0,
129
- "num_train_epochs": 1,
130
- "save_steps": 500,
131
- "stateful_callbacks": {
132
- "TrainerControl": {
133
- "args": {
134
- "should_epoch_stop": false,
135
- "should_evaluate": false,
136
- "should_log": false,
137
- "should_save": false,
138
- "should_training_stop": false
139
- },
140
- "attributes": {}
141
- }
142
- },
143
- "total_flos": 0.0,
144
- "train_batch_size": 2,
145
- "trial_name": null,
146
- "trial_params": null
147
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
training_args.bin DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:c619b9db98ea959618a4839c7d36e8399c8ce4dac6b0fc49c96d5e1d3b9e1fc4
3
- size 6520