---
license: bigscience-openrail-m
metrics:
- code_eval
library_name: transformers
tags:
- code
---

# MoTCoder: Elevating Large Language Models with Modular Thought for Challenging Programming Tasks

This is the official model repository of MoTCoder: Elevating Large Language Models with Modular Thought for Challenging Programming Tasks.
![MoTCoder Framework](./framework.png)

## Performance

![Performance on APPS](./impression.png)

**Performance on APPS**
| Model                     | Size | Pass@ | Introductory | Interview | Competition | All  |
|---------------------------|------|-------|--------------|-----------|-------------|------|
| **GPT-Neo**               | 2.7B | 1     | 3.90         | 0.57      | 0.00        | 1.12 |
|                           |      | 5     | 5.50         | 0.80      | 0.00        | 1.58 |
| **Codex**                 | 12B  | 1     | 4.14         | 0.14      | 0.02        | 0.92 |
|                           |      | 5     | 9.65         | 0.51      | 0.09        | 2.25 |
|                           |      | 1000  | 25.02        | 3.70      | 3.23        | 7.87 |
| **AlphaCode**             | 1B   | 1000  | 17.67        | 5.24      | 7.06        | 8.09 |
| **AlphaCode (Filtered 1k)**|      | 5     | 14.36        | 5.63      | 4.58        | 7.17 |
| **AlphaCode (Filtered 10k)**|     | 5     | 18.18        | 8.21      | 6.65        | 9.89 |
| **AlphaCode (Filtered 50k)**|     | 5     | 20.36        | 9.66      | 7.75        | 11.42 |
| **StarCoder**             | 15B  | 1     | 7.25         | 6.89      | 4.08        | 6.40 |
| **WizardCoder**           | 15B  | 1     | 26.04        | 4.21      | 0.81        | 7.90 |
| **CodeLlama**             | 7B   | 5     | 10.76        | 2.01      | 0.77        | 3.51 |
|                           |      | 10    | 15.59        | 3.12      | 1.41        | 5.27 |
|                           |      | 100   | 33.52        | 9.40      | 7.13        | 13.77|
|                           | 13B  | 5     | 23.74        | 5.63      | 2.05        | 8.54 |
|                           |      | 10    | 30.19        | 8.12      | 3.35        | 11.58|
|                           |      | 100   | 48.99        | 18.40     | 11.98       | 23.23|
|                           | 34B  | 5     | 32.81        | 8.75      | 2.88        | 12.39|
|                           |      | 10    | 38.97        | 12.16     | 4.69        | 16.03|
|                           |      | 100   | 56.32        | 24.31     | 15.39       | 28.93|
| **CodeLlama-Python**      | 7B   | 5     | 12.72        | 4.18      | 1.31        | 5.31 |
|                           |      | 10    | 18.50        | 6.25      | 2.24        | 7.90 |
|                           |      | 100   | 38.26        | 14.94     | 9.12        | 18.44|
|                           | 13B  | 5     | 26.33        | 7.06      | 2.79        | 10.06|
|                           |      | 10    | 32.77        | 10.03     | 4.33        | 13.44|
|                           |      | 100   | 51.60        | 21.46     | 14.60       | 26.12 |
|                           | 34B  | 5     | 28.94        | 7.80      | 3.45        | 11.16 |
|                           |      | 10    | 35.91        | 11.12     | 5.53        | 14.96 |
|                           |      | 100   | 54.92        | 23.90     | 16.81       | 28.69 |
| **CodeLlama-Instruct**    | 7B   | 5     | 12.85        | 2.07      | 1.13        | 4.04  |
|                           |      | 10    | 17.86        | 3.12      | 1.95        | 5.83  |
|                           |      | 100   | 35.37        | 9.44      | 8.45        | 14.43 |
|                           | 13B  | 5     | 24.01        | 6.93      | 2.39        | 9.44  |
|                           |      | 10    | 30.27        | 9.58      | 3.83        | 12.57 |
|                           |      | 100   | 48.73        | 19.55     | 13.12       | 24.10 |
|                           | 34B  | 5     | 31.56        | 7.86      | 3.21        | 11.67 |
|                           |      | 10    | 37.80        | 11.08     | 5.12        | 15.23 |
|                           |      | 100   | 55.72        | 22.80     | 16.38       | 28.10 |
| **CoTCode**               | 15B  | 1     | 33.80        | 19.70     | 11.09       | 20.80 |
| **code-davinci-002**      | -    | 1     | 29.30        | 6.40      | 2.50        | 10.20 |
| **GPT3.5**                | -    | 1     | 48.00        | 19.42     | 5.42        | 22.33 |

**Performance on CodeContests**
| Model | Size | Revision | Val pass@1 | Val pass@5 | Test pass@1 | Test pass@5 | Average pass@1 | Average pass@5 |
|-------|------|----------|------------|------------|-------------|-------------|----------------|----------------|
| **code-davinci-002** | - | - | - | - | 1.00 | - | 1.00 | - |
| **code-davinci-002 + CodeT** | - | 5 | - | - | 3.20 | - | 3.20 | - |
| **WizardCoder** | 15B | - | 1.11 | 3.18 | 1.98 | 3.27 | 1.55 | 3.23 |
| **WizardCoder + CodeChain** | 15B | 5 | 2.35 | 3.29 | 2.48 | 3.30 | 2.42 | 3.30 |
| **CoTCode** | 15B | - | 2.39 | 7.69 | 6.18 | 12.73 | 4.29 | 10.21 |
| **GPT3.5** | - | - | 6.81 | 16.23 | 5.82 | 11.16 | 6.32 | 13.70 |