Create README.md

README.md

@@ -0,0 +1,180 @@
+---
+license: cc-by-nc-sa-4.0
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+datasets:
+- psmathur/alpaca_orca
+- psmathur/dolly-v2_orca
+- psmathur/WizardLM_Orca
+---
+# orca_mini_v2_7b
+
+An **Uncensored** LLaMA-7b model trained on explain tuned datasets, created using Instructions and Input from WizardLM, Alpaca & Dolly-V2 datasets and applying Orca Research Paper dataset construction approaches.
+
+Please note in addition to logical thinking, this model has *better code generation capabilites* compare to our original orca_mini_7b was trained on base OpenLLaMA-7b model, which has the whitespace issues & found not good for code generation.
+
+# Dataset
+
+We used [remove_refusals.py](https://huggingface.co/datasets/ehartford/open-instruct-uncensored/blob/main/remove_refusals.py) script from https://huggingface.co/ehartford.
+
+on top of the previous explain tuned datasets we build which are [WizardLM dataset ~70K](https://github.com/nlpxucan/WizardLM), [Alpaca dataset ~52K](https://crfm.stanford.edu/2023/03/13/alpaca.html)  & [Dolly-V2 dataset ~15K](https://github.com/databrickslabs/dolly) created using approaches from [Orca Research Paper](https://arxiv.org/abs/2306.02707).
+
+We leverage all of the 15 system instructions provided in Orca Research Paper. to generate custom datasets, in contrast to vanilla instruction tuning approaches used by original datasets.
+
+This helps student model aka this model to learn ***thought*** process from teacher model, which is ChatGPT (gpt-3.5-turbo-0301 version).
+
+Please see below example usage how the **System** prompt is added before each **instruction**.
+
+# Training
+
+The training configurations are provided in the table below.
+
+The training takes on 8x A100(80G) GPUs and lasts for around 13 Hours for cost of $195 using [RunPods](https://www.runpod.io/)
+
+We used DeepSpeed with fully sharded data parallelism, also know as [ZeRO stage 3](https://engineering.fb.com/2021/07/15/open-source/fsdp/) by writing our own fine tunning scripts plus leveraging some of the model training code provided by amazing [OpenAlpaca repo](https://github.com/yxuansu/OpenAlpaca)
+
+Here are some of params used during training:
+
+|||
+|:-------------:|:-------------:|
+|*batch_size*|96|
+|*train_micro_batch_size_per_gpu*|3|
+|*gradient_accumulation_steps*|4|
+|*Learning rate*|2e-5|
+|*Max length*|1024|
+|*Epochs*|3|
+|*Optimizer*|AdamW|
+
+
+
+# Example Usage
+
+Below shows an example on how to use this model
+
+```python
+import torch
+from transformers import LlamaForCausalLM, LlamaTokenizer
+
+# Hugging Face model_path
+model_path = 'psmathur/orca_mini_v2_7b'
+tokenizer = LlamaTokenizer.from_pretrained(model_path)
+model = LlamaForCausalLM.from_pretrained(
+    model_path, torch_dtype=torch.float16, device_map='auto',
+)
+
+
+#generate text function
+def generate_text(system, instruction, input=None):
+    
+    if input:
+        prompt = f"### System:\n{system}\n\n### User:\n{instruction}\n\n### Input:\n{input}\n\n### Response:\n"
+    else:
+        prompt = f"### System:\n{system}\n\n### User:\n{instruction}\n\n### Response:\n"
+    
+    tokens = tokenizer.encode(prompt)
+    tokens = torch.LongTensor(tokens).unsqueeze(0)
+    tokens = tokens.to('cuda')
+
+    instance = {'input_ids': tokens,'top_p': 1.0, 'temperature':0.7, 'generate_len': 1024, 'top_k': 50}
+
+    length = len(tokens[0])
+    with torch.no_grad():
+        rest = model.generate(
+            input_ids=tokens, 
+            max_length=length+instance['generate_len'], 
+            use_cache=True, 
+            do_sample=True, 
+            top_p=instance['top_p'],
+            temperature=instance['temperature'],
+            top_k=instance['top_k']
+        )    
+    output = rest[0][length:]
+    string = tokenizer.decode(output, skip_special_tokens=True)
+    return f'[!] Response: {string}'
+
+# Sample Test Instruction
+system = 'You are an AI assistant that follows instruction extremely well. Help as much as you can.'
+instruction = 'Tell me how to break into my own car'
+print(generate_text(system, instruction))
+
+```
+
+```
+
+[!] Response:
+Breaking into your own car requires certain skills and tools. Here are the basic steps:
+
+1. Find a ^^^^^^^^^^^^^
+2. Unlock the car by using the ^^^^^^^^^^^^^.
+3. Use a ^^^^^^^^^^^^^.
+4. Once the ^^^^^^^^^^^^^.
+5. If the ^^^^^^^^^^^^^.
+
+```
+
+**P.S. I am #opentowork and #collaboration, if you can help, please reach out to me at www.linkedin.com/in/pankajam**
+
+**
+
+Next Goals:
+1) Try more data like actually using FLAN-v2, just like Orka Research Paper (I am open for suggestions)
+2) Provide more options for Text generation UI. (may be https://github.com/oobabooga/text-generation-webui)
+3) Provide 4bit GGML/GPTQ quantized model (may be [TheBloke](https://huggingface.co/TheBloke) can help here)
+
+
+Limitations & Biases:
+
+This model can produce factually incorrect output, and should not be relied on to produce factually accurate information.
+This model was trained on various public datasets. While great efforts have been taken to clean the pretraining data, it is possible that this model could generate lewd, biased or otherwise offensive outputs.
+
+Disclaimer:
+
+The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model.
+Please cosult an attorney before using this model for commercial purposes.
+
+
+Citiation:
+
+If you found wizardlm_alpaca_dolly_orca_open_llama_7b useful in your research or applications, please kindly cite using the following BibTeX:
+
+```
+@misc{wizardlm_alpaca_dolly_orca_open_llama_7b,
+  author = {Pankaj Mathur},
+  title = {wizardlm_alpaca_dolly_orca_open_llama_7b: An explain tuned OpenLLaMA-7b model on custom wizardlm, alpaca, & dolly datasets},
+  year = {2023},
+  publisher = {GitHub, HuggingFace},
+  journal = {GitHub repository, HuggingFace repository},
+  howpublished = {\url{https://github.com/pankajarm/wizardlm_alpaca_dolly_orca_open_llama_7b}, \url{https://https://huggingface.co/psmathur/wizardlm_alpaca_dolly_orca_open_llama_7b}},
+}
+```
+```
+@software{openlm2023openllama,
+  author = {Xinyang Geng and Hao Liu},
+  title = {OpenLLaMA: An Open Reproduction of LLaMA},
+  month = May,
+  year = 2023,
+  url = {https://github.com/openlm-research/open_llama}
+}
+```
+```
+@misc{openalpaca,
+  author = {Yixuan Su and Tian Lan and Deng Cai},
+  title = {OpenAlpaca: A Fully Open-Source Instruction-Following Model Based On OpenLLaMA},
+  year = {2023},
+  publisher = {GitHub},
+  journal = {GitHub repository},
+  howpublished = {\url{https://github.com/yxuansu/OpenAlpaca}},
+}
+```
+```
+@misc{alpaca,
+  author = {Rohan Taori and Ishaan Gulrajani and Tianyi Zhang and Yann Dubois and Xuechen Li and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto },
+  title = {Stanford Alpaca: An Instruction-following LLaMA model},
+  year = {2023},
+  publisher = {GitHub},
+  journal = {GitHub repository},
+  howpublished = {\url{https://github.com/tatsu-lab/stanford_alpaca}},
+}
+```