File size: 5,176 Bytes
8e1d9f0 9c21642 14c43a5 9c21642 8e1d9f0 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 |
---
license: bigscience-openrail-m
metrics:
- code_eval
library_name: transformers
tags:
- code
---
# MoTCoder: Elevating Large Language Models with Modular Thought for Challenging Programming Tasks
This is the official model repository of MoTCoder: Elevating Large Language Models with Modular Thought for Challenging Programming Tasks.
## Performance
**Performance on APPS**
| Model | Size | Pass@ | Introductory | Interview | Competition | All |
|---------------------------|------|-------|--------------|-----------|-------------|------|
| **GPT-Neo** | 2.7B | 1 | 3.90 | 0.57 | 0.00 | 1.12 |
| | | 5 | 5.50 | 0.80 | 0.00 | 1.58 |
| **Codex** | 12B | 1 | 4.14 | 0.14 | 0.02 | 0.92 |
| | | 5 | 9.65 | 0.51 | 0.09 | 2.25 |
| | | 1000 | 25.02 | 3.70 | 3.23 | 7.87 |
| **AlphaCode** | 1B | 1000 | 17.67 | 5.24 | 7.06 | 8.09 |
| **AlphaCode (Filtered 1k)**| | 5 | 14.36 | 5.63 | 4.58 | 7.17 |
| **AlphaCode (Filtered 10k)**| | 5 | 18.18 | 8.21 | 6.65 | 9.89 |
| **AlphaCode (Filtered 50k)**| | 5 | 20.36 | 9.66 | 7.75 | 11.42 |
| **StarCoder** | 15B | 1 | 7.25 | 6.89 | 4.08 | 6.40 |
| **WizardCoder** | 15B | 1 | 26.04 | 4.21 | 0.81 | 7.90 |
| **CodeLlama** | 7B | 5 | 10.76 | 2.01 | 0.77 | 3.51 |
| | | 10 | 15.59 | 3.12 | 1.41 | 5.27 |
| | | 100 | 33.52 | 9.40 | 7.13 | 13.77|
| | 13B | 5 | 23.74 | 5.63 | 2.05 | 8.54 |
| | | 10 | 30.19 | 8.12 | 3.35 | 11.58|
| | | 100 | 48.99 | 18.40 | 11.98 | 23.23|
| | 34B | 5 | 32.81 | 8.75 | 2.88 | 12.39|
| | | 10 | 38.97 | 12.16 | 4.69 | 16.03|
| | | 100 | 56.32 | 24.31 | 15.39 | 28.93|
| **CodeLlama-Python** | 7B | 5 | 12.72 | 4.18 | 1.31 | 5.31 |
| | | 10 | 18.50 | 6.25 | 2.24 | 7.90 |
| | | 100 | 38.26 | 14.94 | 9.12 | 18.44|
| | 13B | 5 | 26.33 | 7.06 | 2.79 | 10.06|
| | | 10 | 32.77 | 10.03 | 4.33 | 13.44|
| | | 100 | 51.60 | 21.46 | 14.60 | 26.12 |
| | 34B | 5 | 28.94 | 7.80 | 3.45 | 11.16 |
| | | 10 | 35.91 | 11.12 | 5.53 | 14.96 |
| | | 100 | 54.92 | 23.90 | 16.81 | 28.69 |
| **CodeLlama-Instruct** | 7B | 5 | 12.85 | 2.07 | 1.13 | 4.04 |
| | | 10 | 17.86 | 3.12 | 1.95 | 5.83 |
| | | 100 | 35.37 | 9.44 | 8.45 | 14.43 |
| | 13B | 5 | 24.01 | 6.93 | 2.39 | 9.44 |
| | | 10 | 30.27 | 9.58 | 3.83 | 12.57 |
| | | 100 | 48.73 | 19.55 | 13.12 | 24.10 |
| | 34B | 5 | 31.56 | 7.86 | 3.21 | 11.67 |
| | | 10 | 37.80 | 11.08 | 5.12 | 15.23 |
| | | 100 | 55.72 | 22.80 | 16.38 | 28.10 |
| **CoTCode** | 15B | 1 | 33.80 | 19.70 | 11.09 | 20.80 |
| **code-davinci-002** | - | 1 | 29.30 | 6.40 | 2.50 | 10.20 |
| **GPT3.5** | - | 1 | 48.00 | 19.42 | 5.42 | 22.33 |
**Performance on CodeContests**
| Model | Size | Revision | Val pass@1 | Val pass@5 | Test pass@1 | Test pass@5 | Average pass@1 | Average pass@5 |
|-------|------|----------|------------|------------|-------------|-------------|----------------|----------------|
| **code-davinci-002** | - | - | - | - | 1.00 | - | 1.00 | - |
| **code-davinci-002 + CodeT** | - | 5 | - | - | 3.20 | - | 3.20 | - |
| **WizardCoder** | 15B | - | 1.11 | 3.18 | 1.98 | 3.27 | 1.55 | 3.23 |
| **WizardCoder + CodeChain** | 15B | 5 | 2.35 | 3.29 | 2.48 | 3.30 | 2.42 | 3.30 |
| **CoTCode** | 15B | - | 2.39 | 7.69 | 6.18 | 12.73 | 4.29 | 10.21 |
| **GPT3.5** | - | - | 6.81 | 16.23 | 5.82 | 11.16 | 6.32 | 13.70 |
|