Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,27 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
base_model: alpindale/WizardLM-2-8x22B
|
| 4 |
+
---
|
| 5 |
+
|
| 6 |
+
# SorcererLM-8x22b-bf16
|
| 7 |
+
|
| 8 |
+
Oh boy, here we go. Low-rank LoRA on top of [WizardLM-2-8x22B](https://huggingface.co/alpindale/WizardLM-2-8x22B) trained on 2 epochs of the (cleaned & deduped) c2-logs.
|
| 9 |
+
|
| 10 |
+
## Why LoRA?
|
| 11 |
+
|
| 12 |
+
Fully intentional. `WizardLM-2-8x22B` is smart by itself but its used vocabulary leaves much to be desired when it comes to RP. By training a low-rank LoRA on top of it to teach it some of Claude's writing style, we remedy that.
|
| 13 |
+
|
| 14 |
+
## Prompting
|
| 15 |
+
|
| 16 |
+
- Use the templates in [Quant-Cartel/Recommended-Settings](https://huggingface.co/Quant-Cartel/Recommended-Settings) under the `SorcererLM`-folder.
|
| 17 |
+
- Or Vicuna 1.1 and a sane context template. It's sensitive to samplers, I'd recommend Temperature 1, MinP 0.05 and a dash of DRY but YMMV.
|
| 18 |
+
|
| 19 |
+
## Acknowledgments
|
| 20 |
+
|
| 21 |
+
- My [Cartel](https://huggingface.co/Quant-Cartel) bros, [Envoid](https://huggingface.co/Envoid) and [I^2](https://huggingface.co/InferenceIllusionist) for being amazing
|
| 22 |
+
- My wallet for making sure I could do this
|
| 23 |
+
|
| 24 |
+
## Training
|
| 25 |
+
|
| 26 |
+
Trained using [qlora-pipe](https://github.com/tdrussell/qlora-pipe). Configs included in the `train`-subfolder.
|
| 27 |
+
|