add files

README.md

@@ -1,3 +1,127 @@
 ---
+tags:
+- generated_from_trainer
+model-index:
+- name: starchat-beta
+  results: []
 license: bigcode-openrail-m
 ---
+
+
+<img src="https://huggingface.co/HuggingFaceH4/starchat-beta/resolve/main/model_logo.png" alt="StarChat Beta Logo" width="800" style="margin-left:'auto' margin-right:'auto' display:'block'"/>
+
+# Model Card for StarChat Beta
+
+StarChat is a series of language models that are trained to act as helpful coding assistants. StarChat Beta is the second model in the series, and is a fine-tuned version of [StarCoderPlus](https://huggingface.co/bigcode/starcoderplus) that was trained on an ["uncensored"](https://erichartford.com/uncensored-models) variant of the [`openassistant-guanaco` dataset](https://huggingface.co/datasets/timdettmers/openassistant-guanaco). We found that removing the in-built alignment of the OpenAssistant dataset boosted performance on the [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) and made the model more helpful at coding tasks. However, this means that model is likely to generate problematic text when prompted to do so and should only be used for educational and research purposes.
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+- **Model type:** A 16B parameter GPT-like model fine-tuned on an ["uncensored"](https://erichartford.com/uncensored-models) variant of the [`openassistant-guanaco` dataset](https://huggingface.co/datasets/timdettmers/openassistant-guanaco).
+- **Language(s) (NLP):** Primarily English and 80+ programming languages.
+- **License:** BigCode Open RAIL-M v1
+- **Finetuned from model:** [bigcode/starcoderplus](https://huggingface.co/bigcode/starcoderplus)
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** https://github.com/bigcode-project/starcoder
+- **Demo:** https://huggingface.co/spaces/HuggingFaceH4/starchat-playground
+
+
+## Intended uses & limitations
+
+The model was fine-tuned on a variant of the [`OpenAssistant/oasst1`](https://huggingface.co/datasets/OpenAssistant/oasst1) dataset, which contains a diverse range of dialogues in over 35 languages. As a result, the model can be used for chat and you can check out our [demo](https://huggingface.co/spaces/HuggingFaceH4/starchat-playground) to test its coding capabilities. 
+
+Here's how you can run the model using the `pipeline()` function from 🤗 Transformers:
+
+```python
+import torch
+from transformers import pipeline
+
+pipe = pipeline("text-generation", model="HuggingFaceH4/starchat-beta", torch_dtype=torch.bfloat16, device_map="auto")
+
+prompt_template = "<|system|>\n<|end|>\n<|user|>\n{query}<|end|>\n<|assistant|>"
+prompt = prompt_template.format(query="How do I sort a list in Python?")
+# We use a special <|end|> token with ID 49155 to denote ends of a turn
+outputs = pipe(prompt, max_new_tokens=256, do_sample=True, temperature=0.2, top_k=50, top_p=0.95, eos_token_id=49155)
+# You can sort a list in Python by using the sort() method. Here's an example:\n\n```\nnumbers = [3, 1, 4, 1, 5, 9, 2, 6, 5, 3, 5]\nnumbers.sort()\nprint(numbers)\n```\n\nThis will sort the list in place and print the sorted list.
+```
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+StarChat Alpha has not been aligned to human preferences with techniques like RLHF or deployed with in-the-loop filtering of responses like ChatGPT, so the model can produce problematic outputs (especially when prompted to do so). 
+Models trained primarily on code data will also have a more skewed demographic bias commensurate with the demographics of the GitHub community, for more on this see the [StarCoder dataset](https://huggingface.co/datasets/bigcode/starcoderdata) which is derived from The Stack.
+
+
+Since the base model was pretrained on a large corpus of code, it may produce code snippets that are syntactically valid but semantically incorrect. 
+For example, it may produce code that does not compile or that produces incorrect results.  
+It may also produce code that is vulnerable to security exploits.  
+We have observed the model also has a tendency to produce false URLs which should be carefully inspected before clicking.
+
+StarChat Alpha was fine-tuned from the base model [StarCoder Base](https://huggingface.co/bigcode/starcoderbase), please refer to its model card's [Limitations Section](https://huggingface.co/bigcode/starcoderbase#limitations) for relevant information. 
+In particular, the model was evaluated on some categories of gender biases, propensity for toxicity, and risk of suggesting code completions with known security flaws; these evaluations are reported in its [technical report](https://drive.google.com/file/d/1cN-b9GnWtHzQRoE7M7gAEyivY0kl4BYs/view).
+
+## Training and evaluation data
+
+StarChat Beta is trained on an ["uncensored"](https://erichartford.com/uncensored-models) variant of the [`openassistant-guanaco` dataset](https://huggingface.co/datasets/timdettmers/openassistant-guanaco). We applied the same [recipe](https://huggingface.co/datasets/ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered/blob/main/wizardlm_clean.py) used to filter the ShareGPT datasets behind the [WizardLM](https://huggingface.co/datasets/ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered).
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 2e-05
+- train_batch_size: 4
+- eval_batch_size: 4
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 8
+- gradient_accumulation_steps: 8
+- total_train_batch_size: 256
+- total_eval_batch_size: 32
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_ratio: 0.03
+- num_epochs: 6
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss |
+|:-------------:|:-----:|:----:|:---------------:|
+| 1.5321        | 0.98  | 15   | 1.2856          |
+| 1.2071        | 1.97  | 30   | 1.2620          |
+| 1.0162        | 2.95  | 45   | 1.2853          |
+| 0.8484        | 4.0   | 61   | 1.3274          |
+| 0.6981        | 4.98  | 76   | 1.3994          |
+| 0.5668        | 5.9   | 90   | 1.4720          |
+
+
+### Framework versions
+
+- Transformers 4.28.1
+- Pytorch 2.0.1+cu118
+- Datasets 2.12.0
+- Tokenizers 0.13.3
+
+## Citation
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+```
+@article{Tunstall2023starchat-alpha,
+  author = {Tunstall, Lewis and Lambert, Nathan and Rajani, Nazneen and Beeching, Edward and Le Scao, Teven and von Werra, Leandro and Han, Sheon and Schmid, Philipp and Rush, Alexander},
+  title = {Creating a Coding Assistant with StarCoder},
+  journal = {Hugging Face Blog},
+  year = {2023},
+  note = {https://huggingface.co/blog/starchat},
+}
+```

added_tokens.json

@@ -0,0 +1,6 @@
+{
+  "<|assistant|>": 49154,
+  "<|end|>": 49155,
+  "<|system|>": 49152,
+  "<|user|>": 49153
+}

all_results.json

@@ -0,0 +1,14 @@
+{
+    "epoch": 5.9,
+    "eval_loss": 1.4719988107681274,
+    "eval_runtime": 5.2039,
+    "eval_samples": 202,
+    "eval_samples_per_second": 38.817,
+    "eval_steps_per_second": 1.345,
+    "perplexity": 4.35793713301621,
+    "train_loss": 0.9728257921006944,
+    "train_runtime": 2307.7787,
+    "train_samples": 3888,
+    "train_samples_per_second": 10.108,
+    "train_steps_per_second": 0.039
+}

config.json

@@ -0,0 +1,39 @@
+{
+  "_name_or_path": "data/starcoderplus-ift-v4.1",
+  "activation_function": "gelu",
+  "architectures": [
+    "GPTBigCodeForCausalLM"
+  ],
+  "attention_softmax_in_fp32": true,
+  "attn_pdrop": 0.1,
+  "bos_token_id": 0,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 0,
+  "inference_runner": 0,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "max_batch_size": null,
+  "max_sequence_length": null,
+  "model_type": "gpt_bigcode",
+  "multi_query": true,
+  "n_embd": 6144,
+  "n_head": 48,
+  "n_inner": 24576,
+  "n_layer": 40,
+  "n_positions": 8192,
+  "pad_key_length": true,
+  "pre_allocate_kv_cache": false,
+  "resid_pdrop": 0.1,
+  "scale_attention_softmax_in_fp32": true,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.28.1",
+  "use_cache": true,
+  "validate_runner_input": true,
+  "vocab_size": 49156
+}

dialogue_template.json

@@ -0,0 +1,12 @@
+{
+  "system": "",
+  "messages": null,
+  "system_token": "<|system|>",
+  "system_format": "standard",
+  "user_token": "<|user|>",
+  "assistant_token": "<|assistant|>",
+  "end_token": "<|end|>",
+  "mid_str": "\n",
+  "end_str": "\n",
+  "extra_end_text": ""
+}

eval_results.json

@@ -0,0 +1,9 @@
+{
+    "epoch": 5.9,
+    "eval_loss": 1.4719988107681274,
+    "eval_runtime": 5.2039,
+    "eval_samples": 202,
+    "eval_samples_per_second": 38.817,
+    "eval_steps_per_second": 1.345,
+    "perplexity": 4.35793713301621
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "eos_token_id": 0,
+  "transformers_version": "4.28.1"
+}

model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e4566397ac87311d6b0b103f11688e579720f618cd4d9903482a384ed8d6ff9
+size 15577696155

model.safetensors.index.json

@@ -0,0 +1,491 @@
+{
+  "metadata": {
+    "total_size": 31034961920
+  },
+  "weight_map": {
+    "transformer.h.0.attn.c_attn.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.0.attn.c_attn.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.0.attn.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.0.attn.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.0.ln_1.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.0.ln_1.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.0.ln_2.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.0.ln_2.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.0.mlp.c_fc.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.0.mlp.c_fc.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.0.mlp.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.0.mlp.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.1.attn.c_attn.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.1.attn.c_attn.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.1.attn.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.1.attn.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.1.ln_1.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.1.ln_1.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.1.ln_2.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.1.ln_2.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.1.mlp.c_fc.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.1.mlp.c_fc.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.1.mlp.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.1.mlp.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.10.attn.c_attn.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.10.attn.c_attn.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.10.attn.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.10.attn.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.10.ln_1.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.10.ln_1.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.10.ln_2.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.10.ln_2.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.10.mlp.c_fc.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.10.mlp.c_fc.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.10.mlp.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.10.mlp.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.11.attn.c_attn.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.11.attn.c_attn.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.11.attn.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.11.attn.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.11.ln_1.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.11.ln_1.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.11.ln_2.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.11.ln_2.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.11.mlp.c_fc.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.11.mlp.c_fc.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.11.mlp.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.11.mlp.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.12.attn.c_attn.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.12.attn.c_attn.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.12.attn.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.12.attn.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.12.ln_1.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.12.ln_1.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.12.ln_2.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.12.ln_2.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.12.mlp.c_fc.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.12.mlp.c_fc.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.12.mlp.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.12.mlp.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.13.attn.c_attn.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.13.attn.c_attn.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.13.attn.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.13.attn.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.13.ln_1.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.13.ln_1.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.13.ln_2.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.13.ln_2.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.13.mlp.c_fc.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.13.mlp.c_fc.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.13.mlp.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.13.mlp.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.14.attn.c_attn.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.14.attn.c_attn.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.14.attn.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.14.attn.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.14.ln_1.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.14.ln_1.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.14.ln_2.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.14.ln_2.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.14.mlp.c_fc.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.14.mlp.c_fc.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.14.mlp.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.14.mlp.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.15.attn.c_attn.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.15.attn.c_attn.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.15.attn.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.15.attn.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.15.ln_1.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.15.ln_1.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.15.ln_2.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.15.ln_2.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.15.mlp.c_fc.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.15.mlp.c_fc.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.15.mlp.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.15.mlp.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.16.attn.c_attn.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.16.attn.c_attn.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.16.attn.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.16.attn.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.16.ln_1.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.16.ln_1.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.16.ln_2.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.16.ln_2.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.16.mlp.c_fc.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.16.mlp.c_fc.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.16.mlp.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.16.mlp.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.17.attn.c_attn.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.17.attn.c_attn.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.17.attn.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.17.attn.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.17.ln_1.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.17.ln_1.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.17.ln_2.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.17.ln_2.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.17.mlp.c_fc.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.17.mlp.c_fc.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.17.mlp.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.17.mlp.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.18.attn.c_attn.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.18.attn.c_attn.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.18.attn.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.18.attn.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.18.ln_1.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.18.ln_1.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.18.ln_2.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.18.ln_2.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.18.mlp.c_fc.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.18.mlp.c_fc.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.18.mlp.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.18.mlp.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.19.attn.c_attn.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.19.attn.c_attn.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.19.attn.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.19.attn.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.19.ln_1.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.19.ln_1.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.19.ln_2.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.19.ln_2.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.19.mlp.c_fc.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.19.mlp.c_fc.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.19.mlp.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.19.mlp.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.2.attn.c_attn.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.2.attn.c_attn.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.2.attn.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.2.attn.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.2.ln_1.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.2.ln_1.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.2.ln_2.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.2.ln_2.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.2.mlp.c_fc.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.2.mlp.c_fc.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.2.mlp.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.2.mlp.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.20.attn.c_attn.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.20.attn.c_attn.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.20.attn.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.20.attn.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.20.ln_1.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.20.ln_1.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.20.ln_2.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.20.ln_2.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.20.mlp.c_fc.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.20.mlp.c_fc.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.20.mlp.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.20.mlp.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.21.attn.c_attn.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.21.attn.c_attn.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.21.attn.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.21.attn.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.21.ln_1.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.21.ln_1.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.21.ln_2.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.21.ln_2.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.21.mlp.c_fc.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.21.mlp.c_fc.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.21.mlp.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.21.mlp.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.22.attn.c_attn.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.22.attn.c_attn.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.22.attn.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.22.attn.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.22.ln_1.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.22.ln_1.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.22.ln_2.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.22.ln_2.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.22.mlp.c_fc.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.22.mlp.c_fc.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.22.mlp.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.22.mlp.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.23.attn.c_attn.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.23.attn.c_attn.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.23.attn.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.23.attn.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.23.ln_1.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.23.ln_1.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.23.ln_2.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.23.ln_2.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.23.mlp.c_fc.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.23.mlp.c_fc.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.23.mlp.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.23.mlp.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.24.attn.c_attn.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.24.attn.c_attn.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.24.attn.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.24.attn.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.24.ln_1.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.24.ln_1.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.24.ln_2.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.24.ln_2.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.24.mlp.c_fc.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.24.mlp.c_fc.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.24.mlp.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.24.mlp.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.25.attn.c_attn.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.25.attn.c_attn.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.25.attn.c_proj.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.25.attn.c_proj.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.25.ln_1.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.25.ln_1.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.25.ln_2.bias": "model-00002-of-00004.safetensors",
+    "transformer.h.25.ln_2.weight": "model-00002-of-00004.safetensors",
+    "transformer.h.25.mlp.c_fc.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.25.mlp.c_fc.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.25.mlp.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.25.mlp.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.26.attn.c_attn.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.26.attn.c_attn.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.26.attn.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.26.attn.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.26.ln_1.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.26.ln_1.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.26.ln_2.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.26.ln_2.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.26.mlp.c_fc.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.26.mlp.c_fc.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.26.mlp.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.26.mlp.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.27.attn.c_attn.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.27.attn.c_attn.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.27.attn.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.27.attn.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.27.ln_1.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.27.ln_1.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.27.ln_2.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.27.ln_2.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.27.mlp.c_fc.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.27.mlp.c_fc.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.27.mlp.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.27.mlp.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.28.attn.c_attn.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.28.attn.c_attn.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.28.attn.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.28.attn.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.28.ln_1.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.28.ln_1.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.28.ln_2.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.28.ln_2.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.28.mlp.c_fc.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.28.mlp.c_fc.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.28.mlp.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.28.mlp.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.29.attn.c_attn.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.29.attn.c_attn.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.29.attn.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.29.attn.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.29.ln_1.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.29.ln_1.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.29.ln_2.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.29.ln_2.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.29.mlp.c_fc.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.29.mlp.c_fc.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.29.mlp.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.29.mlp.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.3.attn.c_attn.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.3.attn.c_attn.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.3.attn.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.3.attn.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.3.ln_1.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.3.ln_1.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.3.ln_2.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.3.ln_2.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.3.mlp.c_fc.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.3.mlp.c_fc.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.3.mlp.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.3.mlp.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.30.attn.c_attn.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.30.attn.c_attn.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.30.attn.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.30.attn.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.30.ln_1.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.30.ln_1.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.30.ln_2.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.30.ln_2.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.30.mlp.c_fc.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.30.mlp.c_fc.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.30.mlp.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.30.mlp.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.31.attn.c_attn.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.31.attn.c_attn.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.31.attn.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.31.attn.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.31.ln_1.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.31.ln_1.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.31.ln_2.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.31.ln_2.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.31.mlp.c_fc.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.31.mlp.c_fc.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.31.mlp.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.31.mlp.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.32.attn.c_attn.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.32.attn.c_attn.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.32.attn.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.32.attn.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.32.ln_1.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.32.ln_1.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.32.ln_2.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.32.ln_2.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.32.mlp.c_fc.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.32.mlp.c_fc.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.32.mlp.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.32.mlp.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.33.attn.c_attn.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.33.attn.c_attn.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.33.attn.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.33.attn.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.33.ln_1.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.33.ln_1.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.33.ln_2.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.33.ln_2.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.33.mlp.c_fc.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.33.mlp.c_fc.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.33.mlp.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.33.mlp.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.34.attn.c_attn.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.34.attn.c_attn.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.34.attn.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.34.attn.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.34.ln_1.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.34.ln_1.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.34.ln_2.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.34.ln_2.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.34.mlp.c_fc.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.34.mlp.c_fc.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.34.mlp.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.34.mlp.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.35.attn.c_attn.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.35.attn.c_attn.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.35.attn.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.35.attn.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.35.ln_1.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.35.ln_1.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.35.ln_2.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.35.ln_2.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.35.mlp.c_fc.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.35.mlp.c_fc.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.35.mlp.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.35.mlp.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.36.attn.c_attn.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.36.attn.c_attn.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.36.attn.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.36.attn.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.36.ln_1.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.36.ln_1.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.36.ln_2.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.36.ln_2.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.36.mlp.c_fc.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.36.mlp.c_fc.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.36.mlp.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.36.mlp.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.37.attn.c_attn.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.37.attn.c_attn.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.37.attn.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.37.attn.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.37.ln_1.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.37.ln_1.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.37.ln_2.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.37.ln_2.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.37.mlp.c_fc.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.37.mlp.c_fc.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.37.mlp.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.37.mlp.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.38.attn.c_attn.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.38.attn.c_attn.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.38.attn.c_proj.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.38.attn.c_proj.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.38.ln_1.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.38.ln_1.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.38.ln_2.bias": "model-00003-of-00004.safetensors",
+    "transformer.h.38.ln_2.weight": "model-00003-of-00004.safetensors",
+    "transformer.h.38.mlp.c_fc.bias": "model-00004-of-00004.safetensors",
+    "transformer.h.38.mlp.c_fc.weight": "model-00004-of-00004.safetensors",
+    "transformer.h.38.mlp.c_proj.bias": "model-00004-of-00004.safetensors",
+    "transformer.h.38.mlp.c_proj.weight": "model-00004-of-00004.safetensors",
+    "transformer.h.39.attn.c_attn.bias": "model-00004-of-00004.safetensors",
+    "transformer.h.39.attn.c_attn.weight": "model-00004-of-00004.safetensors",
+    "transformer.h.39.attn.c_proj.bias": "model-00004-of-00004.safetensors",
+    "transformer.h.39.attn.c_proj.weight": "model-00004-of-00004.safetensors",
+    "transformer.h.39.ln_1.bias": "model-00004-of-00004.safetensors",
+    "transformer.h.39.ln_1.weight": "model-00004-of-00004.safetensors",
+    "transformer.h.39.ln_2.bias": "model-00004-of-00004.safetensors",
+    "transformer.h.39.ln_2.weight": "model-00004-of-00004.safetensors",
+    "transformer.h.39.mlp.c_fc.bias": "model-00004-of-00004.safetensors",
+    "transformer.h.39.mlp.c_fc.weight": "model-00004-of-00004.safetensors",
+    "transformer.h.39.mlp.c_proj.bias": "model-00004-of-00004.safetensors",
+    "transformer.h.39.mlp.c_proj.weight": "model-00004-of-00004.safetensors",
+    "transformer.h.4.attn.c_attn.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.4.attn.c_attn.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.4.attn.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.4.attn.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.4.ln_1.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.4.ln_1.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.4.ln_2.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.4.ln_2.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.4.mlp.c_fc.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.4.mlp.c_fc.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.4.mlp.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.4.mlp.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.5.attn.c_attn.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.5.attn.c_attn.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.5.attn.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.5.attn.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.5.ln_1.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.5.ln_1.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.5.ln_2.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.5.ln_2.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.5.mlp.c_fc.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.5.mlp.c_fc.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.5.mlp.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.5.mlp.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.6.attn.c_attn.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.6.attn.c_attn.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.6.attn.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.6.attn.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.6.ln_1.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.6.ln_1.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.6.ln_2.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.6.ln_2.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.6.mlp.c_fc.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.6.mlp.c_fc.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.6.mlp.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.6.mlp.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.7.attn.c_attn.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.7.attn.c_attn.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.7.attn.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.7.attn.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.7.ln_1.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.7.ln_1.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.7.ln_2.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.7.ln_2.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.7.mlp.c_fc.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.7.mlp.c_fc.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.7.mlp.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.7.mlp.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.8.attn.c_attn.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.8.attn.c_attn.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.8.attn.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.8.attn.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.8.ln_1.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.8.ln_1.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.8.ln_2.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.8.ln_2.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.8.mlp.c_fc.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.8.mlp.c_fc.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.8.mlp.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.8.mlp.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.9.attn.c_attn.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.9.attn.c_attn.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.9.attn.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.9.attn.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.9.ln_1.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.9.ln_1.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.9.ln_2.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.9.ln_2.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.9.mlp.c_fc.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.9.mlp.c_fc.weight": "model-00001-of-00004.safetensors",
+    "transformer.h.9.mlp.c_proj.bias": "model-00001-of-00004.safetensors",
+    "transformer.h.9.mlp.c_proj.weight": "model-00001-of-00004.safetensors",
+    "transformer.ln_f.bias": "model-00004-of-00004.safetensors",
+    "transformer.ln_f.weight": "model-00004-of-00004.safetensors",
+    "transformer.wpe.weight": "model-00001-of-00004.safetensors",
+    "transformer.wte.weight": "model-00001-of-00004.safetensors"
+  }
+}

pytorch_model.bin.index.json

@@ -0,0 +1,492 @@
+{
+  "metadata": {
+    "total_size": 31638990848
+  },
+  "weight_map": {
+    "lm_head.weight": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.0.attn.c_attn.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.0.attn.c_attn.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.0.attn.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.0.attn.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.0.ln_1.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.0.ln_1.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.0.ln_2.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.0.ln_2.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.0.mlp.c_fc.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.0.mlp.c_fc.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.0.mlp.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.0.mlp.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.1.attn.c_attn.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.1.attn.c_attn.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.1.attn.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.1.attn.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.1.ln_1.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.1.ln_1.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.1.ln_2.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.1.ln_2.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.1.mlp.c_fc.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.1.mlp.c_fc.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.1.mlp.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.1.mlp.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.10.attn.c_attn.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.10.attn.c_attn.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.10.attn.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.10.attn.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.10.ln_1.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.10.ln_1.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.10.ln_2.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.10.ln_2.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.10.mlp.c_fc.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.10.mlp.c_fc.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.10.mlp.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.10.mlp.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.11.attn.c_attn.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.11.attn.c_attn.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.11.attn.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.11.attn.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.11.ln_1.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.11.ln_1.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.11.ln_2.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.11.ln_2.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.11.mlp.c_fc.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.11.mlp.c_fc.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.11.mlp.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.11.mlp.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.12.attn.c_attn.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.12.attn.c_attn.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.12.attn.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.12.attn.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.12.ln_1.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.12.ln_1.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.12.ln_2.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.12.ln_2.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.12.mlp.c_fc.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.12.mlp.c_fc.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.12.mlp.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.12.mlp.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.13.attn.c_attn.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.13.attn.c_attn.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.13.attn.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.13.attn.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.13.ln_1.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.13.ln_1.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.13.ln_2.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.13.ln_2.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.13.mlp.c_fc.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.13.mlp.c_fc.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.13.mlp.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.13.mlp.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.14.attn.c_attn.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.14.attn.c_attn.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.14.attn.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.14.attn.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.14.ln_1.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.14.ln_1.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.14.ln_2.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.14.ln_2.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.14.mlp.c_fc.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.14.mlp.c_fc.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.14.mlp.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.14.mlp.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.15.attn.c_attn.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.15.attn.c_attn.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.15.attn.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.15.attn.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.15.ln_1.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.15.ln_1.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.15.ln_2.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.15.ln_2.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.15.mlp.c_fc.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.15.mlp.c_fc.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.15.mlp.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.15.mlp.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.16.attn.c_attn.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.16.attn.c_attn.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.16.attn.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.16.attn.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.16.ln_1.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.16.ln_1.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.16.ln_2.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.16.ln_2.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.16.mlp.c_fc.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.16.mlp.c_fc.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.16.mlp.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.16.mlp.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.17.attn.c_attn.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.17.attn.c_attn.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.17.attn.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.17.attn.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.17.ln_1.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.17.ln_1.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.17.ln_2.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.17.ln_2.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.17.mlp.c_fc.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.17.mlp.c_fc.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.17.mlp.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.17.mlp.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.18.attn.c_attn.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.18.attn.c_attn.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.18.attn.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.18.attn.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.18.ln_1.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.18.ln_1.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.18.ln_2.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.18.ln_2.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.18.mlp.c_fc.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.18.mlp.c_fc.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.18.mlp.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.18.mlp.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.19.attn.c_attn.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.19.attn.c_attn.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.19.attn.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.19.attn.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.19.ln_1.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.19.ln_1.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.19.ln_2.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.19.ln_2.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.19.mlp.c_fc.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.19.mlp.c_fc.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.19.mlp.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.19.mlp.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.2.attn.c_attn.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.2.attn.c_attn.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.2.attn.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.2.attn.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.2.ln_1.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.2.ln_1.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.2.ln_2.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.2.ln_2.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.2.mlp.c_fc.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.2.mlp.c_fc.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.2.mlp.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.2.mlp.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.20.attn.c_attn.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.20.attn.c_attn.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.20.attn.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.20.attn.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.20.ln_1.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.20.ln_1.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.20.ln_2.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.20.ln_2.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.20.mlp.c_fc.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.20.mlp.c_fc.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.20.mlp.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.20.mlp.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.21.attn.c_attn.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.21.attn.c_attn.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.21.attn.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.21.attn.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.21.ln_1.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.21.ln_1.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.21.ln_2.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.21.ln_2.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.21.mlp.c_fc.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.21.mlp.c_fc.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.21.mlp.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.21.mlp.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.22.attn.c_attn.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.22.attn.c_attn.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.22.attn.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.22.attn.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.22.ln_1.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.22.ln_1.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.22.ln_2.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.22.ln_2.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.22.mlp.c_fc.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.22.mlp.c_fc.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.22.mlp.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.22.mlp.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.23.attn.c_attn.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.23.attn.c_attn.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.23.attn.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.23.attn.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.23.ln_1.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.23.ln_1.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.23.ln_2.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.23.ln_2.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.23.mlp.c_fc.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.23.mlp.c_fc.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.23.mlp.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.23.mlp.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.24.attn.c_attn.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.24.attn.c_attn.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.24.attn.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.24.attn.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.24.ln_1.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.24.ln_1.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.24.ln_2.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.24.ln_2.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.24.mlp.c_fc.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.24.mlp.c_fc.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.24.mlp.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.24.mlp.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.25.attn.c_attn.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.25.attn.c_attn.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.25.attn.c_proj.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.25.attn.c_proj.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.25.ln_1.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.25.ln_1.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.25.ln_2.bias": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.25.ln_2.weight": "pytorch_model-00002-of-00004.bin",
+    "transformer.h.25.mlp.c_fc.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.25.mlp.c_fc.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.25.mlp.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.25.mlp.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.26.attn.c_attn.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.26.attn.c_attn.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.26.attn.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.26.attn.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.26.ln_1.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.26.ln_1.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.26.ln_2.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.26.ln_2.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.26.mlp.c_fc.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.26.mlp.c_fc.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.26.mlp.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.26.mlp.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.27.attn.c_attn.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.27.attn.c_attn.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.27.attn.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.27.attn.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.27.ln_1.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.27.ln_1.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.27.ln_2.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.27.ln_2.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.27.mlp.c_fc.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.27.mlp.c_fc.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.27.mlp.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.27.mlp.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.28.attn.c_attn.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.28.attn.c_attn.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.28.attn.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.28.attn.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.28.ln_1.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.28.ln_1.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.28.ln_2.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.28.ln_2.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.28.mlp.c_fc.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.28.mlp.c_fc.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.28.mlp.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.28.mlp.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.29.attn.c_attn.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.29.attn.c_attn.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.29.attn.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.29.attn.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.29.ln_1.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.29.ln_1.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.29.ln_2.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.29.ln_2.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.29.mlp.c_fc.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.29.mlp.c_fc.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.29.mlp.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.29.mlp.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.3.attn.c_attn.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.3.attn.c_attn.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.3.attn.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.3.attn.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.3.ln_1.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.3.ln_1.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.3.ln_2.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.3.ln_2.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.3.mlp.c_fc.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.3.mlp.c_fc.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.3.mlp.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.3.mlp.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.30.attn.c_attn.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.30.attn.c_attn.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.30.attn.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.30.attn.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.30.ln_1.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.30.ln_1.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.30.ln_2.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.30.ln_2.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.30.mlp.c_fc.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.30.mlp.c_fc.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.30.mlp.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.30.mlp.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.31.attn.c_attn.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.31.attn.c_attn.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.31.attn.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.31.attn.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.31.ln_1.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.31.ln_1.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.31.ln_2.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.31.ln_2.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.31.mlp.c_fc.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.31.mlp.c_fc.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.31.mlp.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.31.mlp.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.32.attn.c_attn.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.32.attn.c_attn.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.32.attn.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.32.attn.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.32.ln_1.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.32.ln_1.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.32.ln_2.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.32.ln_2.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.32.mlp.c_fc.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.32.mlp.c_fc.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.32.mlp.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.32.mlp.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.33.attn.c_attn.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.33.attn.c_attn.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.33.attn.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.33.attn.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.33.ln_1.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.33.ln_1.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.33.ln_2.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.33.ln_2.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.33.mlp.c_fc.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.33.mlp.c_fc.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.33.mlp.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.33.mlp.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.34.attn.c_attn.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.34.attn.c_attn.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.34.attn.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.34.attn.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.34.ln_1.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.34.ln_1.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.34.ln_2.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.34.ln_2.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.34.mlp.c_fc.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.34.mlp.c_fc.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.34.mlp.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.34.mlp.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.35.attn.c_attn.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.35.attn.c_attn.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.35.attn.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.35.attn.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.35.ln_1.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.35.ln_1.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.35.ln_2.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.35.ln_2.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.35.mlp.c_fc.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.35.mlp.c_fc.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.35.mlp.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.35.mlp.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.36.attn.c_attn.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.36.attn.c_attn.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.36.attn.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.36.attn.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.36.ln_1.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.36.ln_1.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.36.ln_2.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.36.ln_2.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.36.mlp.c_fc.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.36.mlp.c_fc.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.36.mlp.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.36.mlp.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.37.attn.c_attn.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.37.attn.c_attn.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.37.attn.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.37.attn.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.37.ln_1.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.37.ln_1.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.37.ln_2.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.37.ln_2.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.37.mlp.c_fc.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.37.mlp.c_fc.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.37.mlp.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.37.mlp.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.38.attn.c_attn.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.38.attn.c_attn.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.38.attn.c_proj.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.38.attn.c_proj.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.38.ln_1.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.38.ln_1.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.38.ln_2.bias": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.38.ln_2.weight": "pytorch_model-00003-of-00004.bin",
+    "transformer.h.38.mlp.c_fc.bias": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.38.mlp.c_fc.weight": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.38.mlp.c_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.38.mlp.c_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.39.attn.c_attn.bias": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.39.attn.c_attn.weight": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.39.attn.c_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.39.attn.c_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.39.ln_1.bias": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.39.ln_1.weight": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.39.ln_2.bias": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.39.ln_2.weight": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.39.mlp.c_fc.bias": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.39.mlp.c_fc.weight": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.39.mlp.c_proj.bias": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.39.mlp.c_proj.weight": "pytorch_model-00004-of-00004.bin",
+    "transformer.h.4.attn.c_attn.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.4.attn.c_attn.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.4.attn.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.4.attn.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.4.ln_1.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.4.ln_1.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.4.ln_2.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.4.ln_2.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.4.mlp.c_fc.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.4.mlp.c_fc.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.4.mlp.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.4.mlp.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.5.attn.c_attn.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.5.attn.c_attn.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.5.attn.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.5.attn.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.5.ln_1.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.5.ln_1.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.5.ln_2.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.5.ln_2.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.5.mlp.c_fc.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.5.mlp.c_fc.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.5.mlp.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.5.mlp.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.6.attn.c_attn.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.6.attn.c_attn.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.6.attn.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.6.attn.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.6.ln_1.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.6.ln_1.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.6.ln_2.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.6.ln_2.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.6.mlp.c_fc.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.6.mlp.c_fc.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.6.mlp.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.6.mlp.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.7.attn.c_attn.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.7.attn.c_attn.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.7.attn.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.7.attn.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.7.ln_1.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.7.ln_1.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.7.ln_2.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.7.ln_2.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.7.mlp.c_fc.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.7.mlp.c_fc.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.7.mlp.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.7.mlp.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.8.attn.c_attn.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.8.attn.c_attn.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.8.attn.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.8.attn.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.8.ln_1.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.8.ln_1.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.8.ln_2.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.8.ln_2.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.8.mlp.c_fc.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.8.mlp.c_fc.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.8.mlp.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.8.mlp.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.9.attn.c_attn.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.9.attn.c_attn.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.9.attn.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.9.attn.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.9.ln_1.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.9.ln_1.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.9.ln_2.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.9.ln_2.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.9.mlp.c_fc.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.9.mlp.c_fc.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.9.mlp.c_proj.bias": "pytorch_model-00001-of-00004.bin",
+    "transformer.h.9.mlp.c_proj.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.ln_f.bias": "pytorch_model-00004-of-00004.bin",
+    "transformer.ln_f.weight": "pytorch_model-00004-of-00004.bin",
+    "transformer.wpe.weight": "pytorch_model-00001-of-00004.bin",
+    "transformer.wte.weight": "pytorch_model-00001-of-00004.bin"
+  }
+}

special_tokens_map.json

@@ -0,0 +1,11 @@
+{
+  "additional_special_tokens": [
+    "<|system|>",
+    "<|user|>",
+    "<|assistant|>",
+    "<|end|>"
+  ],
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

tokenizer_config.json

@@ -0,0 +1,31 @@
+{
+  "add_prefix_space": false,
+  "additional_special_tokens": [
+    "<|endoftext|>",
+    "<fim_prefix>",
+    "<fim_middle>",
+    "<fim_suffix>",
+    "<fim_pad>",
+    "<filename>",
+    "<gh_stars>",
+    "<issue_start>",
+    "<issue_comment>",
+    "<issue_closed>",
+    "<jupyter_start>",
+    "<jupyter_text>",
+    "<jupyter_code>",
+    "<jupyter_output>",
+    "<empty_output>",
+    "<commit_before>",
+    "<commit_msg>",
+    "<commit_after>",
+    "<reponame>"
+  ],
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "model_max_length": 1000000000000000019884624838656,
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>",
+  "vocab_size": 49152
+}

train_results.json

@@ -0,0 +1,8 @@
+{
+    "epoch": 5.9,
+    "train_loss": 0.9728257921006944,
+    "train_runtime": 2307.7787,
+    "train_samples": 3888,
+    "train_samples_per_second": 10.108,
+    "train_steps_per_second": 0.039
+}

trainer_state.json

@@ -0,0 +1,145 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 5.901639344262295,
+  "global_step": 90,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.07,
+      "learning_rate": 0.0,
+      "loss": 1.8309,
+      "step": 1
+    },
+    {
+      "epoch": 0.52,
+      "learning_rate": 2e-05,
+      "loss": 1.5321,
+      "step": 8
+    },
+    {
+      "epoch": 0.98,
+      "eval_loss": 1.2855817079544067,
+      "eval_runtime": 6.6367,
+      "eval_samples_per_second": 30.437,
+      "eval_steps_per_second": 1.055,
+      "step": 15
+    },
+    {
+      "epoch": 1.05,
+      "learning_rate": 2e-05,
+      "loss": 1.35,
+      "step": 16
+    },
+    {
+      "epoch": 1.57,
+      "learning_rate": 2e-05,
+      "loss": 1.2071,
+      "step": 24
+    },
+    {
+      "epoch": 1.97,
+      "eval_loss": 1.2619894742965698,
+      "eval_runtime": 5.1554,
+      "eval_samples_per_second": 39.183,
+      "eval_steps_per_second": 1.358,
+      "step": 30
+    },
+    {
+      "epoch": 2.1,
+      "learning_rate": 2e-05,
+      "loss": 1.1502,
+      "step": 32
+    },
+    {
+      "epoch": 2.62,
+      "learning_rate": 2e-05,
+      "loss": 1.0162,
+      "step": 40
+    },
+    {
+      "epoch": 2.95,
+      "eval_loss": 1.285272240638733,
+      "eval_runtime": 5.1992,
+      "eval_samples_per_second": 38.852,
+      "eval_steps_per_second": 1.346,
+      "step": 45
+    },
+    {
+      "epoch": 3.15,
+      "learning_rate": 2e-05,
+      "loss": 0.9511,
+      "step": 48
+    },
+    {
+      "epoch": 3.67,
+      "learning_rate": 2e-05,
+      "loss": 0.8484,
+      "step": 56
+    },
+    {
+      "epoch": 4.0,
+      "eval_loss": 1.3274288177490234,
+      "eval_runtime": 5.1899,
+      "eval_samples_per_second": 38.922,
+      "eval_steps_per_second": 1.349,
+      "step": 61
+    },
+    {
+      "epoch": 4.2,
+      "learning_rate": 2e-05,
+      "loss": 0.7971,
+      "step": 64
+    },
+    {
+      "epoch": 4.72,
+      "learning_rate": 2e-05,
+      "loss": 0.6981,
+      "step": 72
+    },
+    {
+      "epoch": 4.98,
+      "eval_loss": 1.3993656635284424,
+      "eval_runtime": 5.213,
+      "eval_samples_per_second": 38.749,
+      "eval_steps_per_second": 1.343,
+      "step": 76
+    },
+    {
+      "epoch": 5.25,
+      "learning_rate": 2e-05,
+      "loss": 0.6462,
+      "step": 80
+    },
+    {
+      "epoch": 5.77,
+      "learning_rate": 2e-05,
+      "loss": 0.5668,
+      "step": 88
+    },
+    {
+      "epoch": 5.9,
+      "eval_loss": 1.4719988107681274,
+      "eval_runtime": 5.1996,
+      "eval_samples_per_second": 38.849,
+      "eval_steps_per_second": 1.346,
+      "step": 90
+    },
+    {
+      "epoch": 5.9,
+      "step": 90,
+      "total_flos": 383994839433216.0,
+      "train_loss": 0.9728257921006944,
+      "train_runtime": 2307.7787,
+      "train_samples_per_second": 10.108,
+      "train_steps_per_second": 0.039
+    }
+  ],
+  "max_steps": 90,
+  "num_train_epochs": 6,
+  "total_flos": 383994839433216.0,
+  "trial_name": null,
+  "trial_params": null
+}