munish0838 commited on
Commit
e67d4ed
1 Parent(s): d5b9a51

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +262 -0
README.md ADDED
@@ -0,0 +1,262 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ datasets:
4
+ - tiiuae/falcon-refinedweb
5
+ - bigcode/the-stack-github-issues
6
+ - bigcode/commitpackft
7
+ - bigcode/starcoderdata
8
+ - EleutherAI/proof-pile-2
9
+ - meta-math/MetaMathQA
10
+ base_model: stabilityai/stable-code-3b
11
+ language:
12
+ - en
13
+ tags:
14
+ - causal-lm
15
+ - code
16
+ metrics:
17
+ - code_eval
18
+ library_name: transformers
19
+ model-index:
20
+ - name: stabilityai/stable-code-3b
21
+ results:
22
+ - task:
23
+ type: text-generation
24
+ dataset:
25
+ type: nuprl/MultiPL-E
26
+ name: MultiPL-HumanEval (Python)
27
+ metrics:
28
+ - name: pass@1
29
+ type: pass@1
30
+ value: 32.4
31
+ verified: false
32
+ - task:
33
+ type: text-generation
34
+ dataset:
35
+ type: nuprl/MultiPL-E
36
+ name: MultiPL-HumanEval (C++)
37
+ metrics:
38
+ - name: pass@1
39
+ type: pass@1
40
+ value: 30.9
41
+ verified: false
42
+ - task:
43
+ type: text-generation
44
+ dataset:
45
+ type: nuprl/MultiPL-E
46
+ name: MultiPL-HumanEval (Java)
47
+ metrics:
48
+ - name: pass@1
49
+ type: pass@1
50
+ value: 32.1
51
+ verified: false
52
+ - task:
53
+ type: text-generation
54
+ dataset:
55
+ type: nuprl/MultiPL-E
56
+ name: MultiPL-HumanEval (JavaScript)
57
+ metrics:
58
+ - name: pass@1
59
+ type: pass@1
60
+ value: 32.1
61
+ verified: false
62
+ - task:
63
+ type: text-generation
64
+ dataset:
65
+ type: nuprl/MultiPL-E
66
+ name: MultiPL-HumanEval (PHP)
67
+ metrics:
68
+ - name: pass@1
69
+ type: pass@1
70
+ value: 24.2
71
+ verified: false
72
+ - task:
73
+ type: text-generation
74
+ dataset:
75
+ type: nuprl/MultiPL-E
76
+ name: MultiPL-HumanEval (Rust)
77
+ metrics:
78
+ - name: pass@1
79
+ type: pass@1
80
+ value: 23
81
+ verified: false
82
+ pipeline_tag: text-generation
83
+ ---
84
+
85
+ # QuantFactory/stable-code-3b-GGUF
86
+ This is quantized version of [stabilityai/stable-code-3b](https://huggingface.co/stabilityai/stable-code-3b) created using llama.cpp
87
+
88
+ ## Model Description
89
+
90
+ Please note: For commercial use, please refer to https://stability.ai/license.
91
+
92
+ `stable-code-3b` is a 2.7B billion parameter decoder-only language model pre-trained on 1.3 trillion tokens of diverse textual and code datasets. `stable-code-3b` is trained on 18 programming languages (selected based on the 2023 StackOverflow Developer Survey) and demonstrates state-of-the-art performance (compared to models of similar size) on the MultiPL-E metrics across multiple programming languages tested using [BigCode's Evaluation Harness](https://github.com/bigcode-project/bigcode-evaluation-harness/tree/main).
93
+
94
+ ![spiderchart](stable_code_3b_spiderchart.svg)
95
+
96
+ | Model | Size | Python | C++ | Javascript | Java | PHP | Rust |
97
+ |------------------|------|--------|------|------------|------|------|------|
98
+ | **Stable Code** | 3B | 32.4% | 30.9%| 32.1% | 32.1%| 24.2%| 23.0%|
99
+ | CodeLLama | 7B | 30.0% | 28.2%| 32.5% | 31.1%| 25.7%| 26.3%|
100
+ | Deepseek Coder | 1.3B | 28.6% | 29.2%| 28.7% | 29.0%| 23.6%| 18.5%|
101
+ | Wizard Coder | 3B | 31.6% | 25.6%| 26.2% | 25.8%| 25.3%| 20.4%|
102
+ | StarCoder | 3B | 21.6% | 19.8%| 21.5% | 20.5%| 19.0%| 16.9%|
103
+ | Replit Code V1.5 | 3B | 23.0% | 25.9%| 26.2% | 23.6%| 23.2%| 21.5%|
104
+ | Deci Coder | 1B | 19.1% | 6.8% | 18.4% | 16.7%| 2.1% | 1.7% |
105
+
106
+ **Key Features**
107
+ * Fill in Middle Capability (FIM)
108
+ * Supports Long Context, trained with Sequences upto 16,384
109
+
110
+ ## Usage
111
+
112
+ Get started generating text with `stable-code-3b` by using the following code snippet:
113
+
114
+ ```python
115
+ import torch
116
+ from transformers import AutoModelForCausalLM, AutoTokenizer
117
+ tokenizer = AutoTokenizer.from_pretrained("stabilityai/stable-code-3b")
118
+ model = AutoModelForCausalLM.from_pretrained(
119
+ "stabilityai/stable-code-3b",
120
+ torch_dtype="auto",
121
+ )
122
+ model.cuda()
123
+ inputs = tokenizer("import torch\nimport torch.nn as nn", return_tensors="pt").to(model.device)
124
+ tokens = model.generate(
125
+ **inputs,
126
+ max_new_tokens=48,
127
+ temperature=0.2,
128
+ do_sample=True,
129
+ )
130
+ print(tokenizer.decode(tokens[0], skip_special_tokens=True))
131
+ ```
132
+
133
+ ### Run with Fill in Middle (FIM) ⚡️
134
+
135
+ <details>
136
+ <summary> Click to expand </summary>
137
+
138
+ ```python
139
+ from transformers import AutoModelForCausalLM, AutoTokenizer
140
+ tokenizer = AutoTokenizer.from_pretrained("stabilityai/stable-code-3b")
141
+ model = AutoModelForCausalLM.from_pretrained(
142
+ "stabilityai/stable-code-3b",
143
+ torch_dtype="auto",
144
+ attn_implementation="flash_attention_2",
145
+ )
146
+ model.cuda()
147
+ inputs = tokenizer("<fim_prefix>def fib(n):<fim_suffix> else:\n return fib(n - 2) + fib(n - 1)<fim_middle>", return_tensors="pt").to(model.device)
148
+ tokens = model.generate(
149
+ **inputs,
150
+ max_new_tokens=48,
151
+ temperature=0.2,
152
+ do_sample=True,
153
+ )
154
+ print(tokenizer.decode(tokens[0], skip_special_tokens=True))
155
+ ```
156
+
157
+ </details>
158
+
159
+ ### Run with Flash Attention 2 ⚡️
160
+
161
+ <details>
162
+ <summary> Click to expand </summary>
163
+
164
+ ```python
165
+ from transformers import AutoModelForCausalLM, AutoTokenizer
166
+ tokenizer = AutoTokenizer.from_pretrained("stabilityai/stable-code-3b", trust_remote_code=True)
167
+ model = AutoModelForCausalLM.from_pretrained(
168
+ "stabilityai/stable-code-3b",
169
+ trust_remote_code=True,
170
+ torch_dtype="auto",
171
+ + attn_implementation="flash_attention_2",
172
+ )
173
+ model.cuda()
174
+ inputs = tokenizer("import torch\nimport torch.nn as nn", return_tensors="pt").to(model.device)
175
+ tokens = model.generate(
176
+ **inputs,
177
+ max_new_tokens=48,
178
+ temperature=0.2,
179
+ do_sample=True,
180
+ )
181
+ print(tokenizer.decode(tokens[0], skip_special_tokens=True))
182
+ ```
183
+
184
+ </details>
185
+
186
+
187
+ ## Model Details
188
+
189
+ * **Developed by**: [Stability AI](https://stability.ai/)
190
+ * **Model type**: `stable-code-3b` models are auto-regressive language models based on the transformer decoder architecture.
191
+ * **Language(s)**: English, Code
192
+ * **Library**: [GPT-NeoX](https://github.com/EleutherAI/gpt-neox)
193
+ * **License**: Stability AI Community License.
194
+ * **Commercial License**: to use this model commercially, please refer to https://stability.ai/license
195
+ * **Contact**: For questions and comments about the model, please email `lm@stability.ai`
196
+
197
+ ### Model Architecture
198
+
199
+ The model is a decoder-only transformer similar to the LLaMA ([Touvron et al., 2023](https://arxiv.org/abs/2307.09288)) architecture with the following modifications:
200
+
201
+ | Parameters | Hidden Size | Layers | Heads | Sequence Length |
202
+ |----------------|-------------|--------|-------|-----------------|
203
+ | 2,796,431,360 | 2560 | 32 | 32 | 16384 |
204
+
205
+ * **Position Embeddings**: Rotary Position Embeddings ([Su et al., 2021](https://arxiv.org/abs/2104.09864)) applied to the first 25% of head embedding dimensions for improved throughput following [Black et al. (2022)](https://arxiv.org/pdf/2204.06745.pdf).
206
+ * **Tokenizer**: We use a modified version of the GPTNeoX Tokenizer.[`NeoX`](https://github.com/EleutherAI/gpt-neox). We add special tokens to train for Fill in the Middle (FIM) capabilities like `<FIM_PREFIX>` and `<FIM_SUFFIX>` along with other special tokens.
207
+
208
+ ## Training
209
+
210
+ ### Training Dataset
211
+
212
+ The dataset is comprised of a filtered mixture of open-source large-scale datasets available on the [HuggingFace Hub](https://huggingface.co/datasets): Falcon RefinedWeb extract ([Penedo et al., 2023](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)), along with [CommitPackFT](https://huggingface.co/datasets/bigcode/commitpackft) and [Github Issues](https://huggingface.co/datasets/bigcode/the-stack-github-issues) (BigCode., 2023), and StarCoder ([Li et al., 2023](https://arxiv.org/abs/2305.06161)). We further supplement our training with data from mathematical domains ([Azerbayev, Zhangir, et al., 2023](https://arxiv.org/abs/2310.10631) and, [Yu, Longhui, et al., 2023](https://arxiv.org/abs/2309.12284)).
213
+
214
+ Top 18 programming languages trained on:
215
+ - C
216
+ - CPP
217
+ - Java
218
+ - JavaScript
219
+ - CSS
220
+ - Go
221
+ - HTML
222
+ - Ruby
223
+ - Rust
224
+ - Markdown
225
+ - Shell
226
+ - Php
227
+ - Sql
228
+ - R
229
+ - Typescript
230
+ - Python
231
+ - Jupyter-Clean
232
+ - RestructuredText
233
+
234
+ ### Training Procedure
235
+
236
+ The model is pre-trained on the aforementioned datasets in `bfloat16` precision, optimized with AdamW.
237
+
238
+ ### Training Infrastructure
239
+
240
+ * **Hardware**: `stable-code-3b` was trained on the Stability AI cluster across 256 NVIDIA A100 40GB GPUs (AWS P4d instances).
241
+
242
+ * **Software**: We use a fork of `gpt-neox` ([EleutherAI, 2021](https://github.com/EleutherAI/gpt-neox)), train under 2D parallelism (Data and Tensor Parallel) with ZeRO-1 ([Rajbhandari et al., 2019](https://arxiv.org/abs/1910.02054v3)), and rely on flash-attention as well as SwiGLU and Rotary Embedding kernels from FlashAttention-2 ([Dao et al., 2023](https://tridao.me/publications/flash2/flash2.pdf))
243
+
244
+ ## Use and Limitations
245
+
246
+ ### Intended Use
247
+
248
+ The model is intended to be used as a foundational base model for application-specific fine-tuning. Developers must evaluate and fine-tune the model for safe performance in downstream applications. For commercial use, please refer to https://stability.ai/license.
249
+
250
+ ### Limitations and Bias
251
+
252
+ As a base model, this model may exhibit unreliable, unsafe, or other undesirable behaviors that must be corrected through evaluation and fine-tuning prior to deployment. The pre-training dataset may have contained offensive or inappropriate content, even after applying data cleansing filters, which can be reflected in the model-generated text. We recommend that users exercise caution when using these models in production systems. Do not use the models if they are unsuitable for your application, or for any applications that may cause deliberate or unintentional harm to others.
253
+
254
+ ## How to Cite Model
255
+
256
+ ```bibtex
257
+ @misc{stable-code-3b,
258
+ url={[https://huggingface.co/stabilityai/stable-code-3b](https://huggingface.co/stabilityai/stable-code-3b)},
259
+ title={Stable Code 3B},
260
+ author={Pinnaparaju, Nikhil and Adithyan, Reshinth and Phung, Duy and Tow, Jonathan and Baicoianu, James and Cooper, Nathan}
261
+ }
262
+ ```