Hugging Face's logo

JingyaoLi
/
MoTCoder-15B-v1.0

Text Generation
Transformers
Safetensors
gpt_bigcode
code
text-generation-inference
Inference Endpoints
Model card Files Community
JingyaoLi commited on
Commit
9c21642
•
1 Parent(s): 3e68ff7

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +63 -2
README.md CHANGED
@@ -7,7 +7,68 @@ tags:
7
  - code
8
  ---
9
 
10
- # MoTCoder
11
- Please refer to our official GitHub website for training and reference templates.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
12
 
13
 
 
7
  - code
8
  ---
9
 
10
+ # MoTCoder: Elevating Large Language Models with Modular Thought for Challenging Programming Tasks
11
+
12
+ This is the official model repository of MoTCoder: Elevating Large Language Models with Modular Thought for Challenging Programming Tasks.
13
+ ![MoTCoder Framework](./framework.png)
14
+
15
+ ## Performance
16
+
17
+ ![Performance on APPS](./impression.png)
18
+
19
+ **Performance on APPS**
20
+ | Model | Size | Pass@ | Introductory | Interview | Competition | All |
21
+ |---------------------------|------|-------|--------------|-----------|-------------|------|
22
+ | **GPT-Neo** | 2.7B | 1 | 3.90 | 0.57 | 0.00 | 1.12 |
23
+ | | | 5 | 5.50 | 0.80 | 0.00 | 1.58 |
24
+ | **Codex** | 12B | 1 | 4.14 | 0.14 | 0.02 | 0.92 |
25
+ | | | 5 | 9.65 | 0.51 | 0.09 | 2.25 |
26
+ | | | 1000 | 25.02 | 3.70 | 3.23 | 7.87 |
27
+ | **AlphaCode** | 1B | 1000 | 17.67 | 5.24 | 7.06 | 8.09 |
28
+ | **AlphaCode (Filtered 1k)**| | 5 | 14.36 | 5.63 | 4.58 | 7.17 |
29
+ | **AlphaCode (Filtered 10k)**| | 5 | 18.18 | 8.21 | 6.65 | 9.89 |
30
+ | **AlphaCode (Filtered 50k)**| | 5 | 20.36 | 9.66 | 7.75 | 11.42 |
31
+ | **StarCoder** | 15B | 1 | 7.25 | 6.89 | 4.08 | 6.40 |
32
+ | **WizardCoder** | 15B | 1 | 26.04 | 4.21 | 0.81 | 7.90 |
33
+ | **CodeLlama** | 7B | 5 | 10.76 | 2.01 | 0.77 | 3.51 |
34
+ | | | 10 | 15.59 | 3.12 | 1.41 | 5.27 |
35
+ | | | 100 | 33.52 | 9.40 | 7.13 | 13.77|
36
+ | | 13B | 5 | 23.74 | 5.63 | 2.05 | 8.54 |
37
+ | | | 10 | 30.19 | 8.12 | 3.35 | 11.58|
38
+ | | | 100 | 48.99 | 18.40 | 11.98 | 23.23|
39
+ | | 34B | 5 | 32.81 | 8.75 | 2.88 | 12.39|
40
+ | | | 10 | 38.97 | 12.16 | 4.69 | 16.03|
41
+ | | | 100 | 56.32 | 24.31 | 15.39 | 28.93|
42
+ | **CodeLlama-Python** | 7B | 5 | 12.72 | 4.18 | 1.31 | 5.31 |
43
+ | | | 10 | 18.50 | 6.25 | 2.24 | 7.90 |
44
+ | | | 100 | 38.26 | 14.94 | 9.12 | 18.44|
45
+ | | 13B | 5 | 26.33 | 7.06 | 2.79 | 10.06|
46
+ | | | 10 | 32.77 | 10.03 | 4.33 | 13.44|
47
+ | | | 100 | 51.60 | 21.46 | 14.60 | 26.12 |
48
+ | | 34B | 5 | 28.94 | 7.80 | 3.45 | 11.16 |
49
+ | | | 10 | 35.91 | 11.12 | 5.53 | 14.96 |
50
+ | | | 100 | 54.92 | 23.90 | 16.81 | 28.69 |
51
+ | **CodeLlama-Instruct** | 7B | 5 | 12.85 | 2.07 | 1.13 | 4.04 |
52
+ | | | 10 | 17.86 | 3.12 | 1.95 | 5.83 |
53
+ | | | 100 | 35.37 | 9.44 | 8.45 | 14.43 |
54
+ | | 13B | 5 | 24.01 | 6.93 | 2.39 | 9.44 |
55
+ | | | 10 | 30.27 | 9.58 | 3.83 | 12.57 |
56
+ | | | 100 | 48.73 | 19.55 | 13.12 | 24.10 |
57
+ | | 34B | 5 | 31.56 | 7.86 | 3.21 | 11.67 |
58
+ | | | 10 | 37.80 | 11.08 | 5.12 | 15.23 |
59
+ | | | 100 | 55.72 | 22.80 | 16.38 | 28.10 |
60
+ | **CoTCode** | 15B | 1 | 33.80 | 19.70 | 11.09 | 20.80 |
61
+ | **code-davinci-002** | - | 1 | 29.30 | 6.40 | 2.50 | 10.20 |
62
+ | **GPT3.5** | - | 1 | 48.00 | 19.42 | 5.42 | 22.33 |
63
+
64
+ **Performance on CodeContests**
65
+ | Model | Size | Revision | Val pass@1 | Val pass@5 | Test pass@1 | Test pass@5 | Average pass@1 | Average pass@5 |
66
+ |-------|------|----------|------------|------------|-------------|-------------|----------------|----------------|
67
+ | **code-davinci-002** | - | - | - | - | 1.00 | - | 1.00 | - |
68
+ | **code-davinci-002 + CodeT** | - | 5 | - | - | 3.20 | - | 3.20 | - |
69
+ | **WizardCoder** | 15B | - | 1.11 | 3.18 | 1.98 | 3.27 | 1.55 | 3.23 |
70
+ | **WizardCoder + CodeChain** | 15B | 5 | 2.35 | 3.29 | 2.48 | 3.30 | 2.42 | 3.30 |
71
+ | **CoTCode** | 15B | - | 2.39 | 7.69 | 6.18 | 12.73 | 4.29 | 10.21 |
72
+ | **GPT3.5** | - | - | 6.81 | 16.23 | 5.82 | 11.16 | 6.32 | 13.70 |
73
 
74