InternLM-Math commited on
Commit
994dece
1 Parent(s): 6c72704

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +88 -0
README.md ADDED
@@ -0,0 +1,88 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ pipeline_tag: text-generation
3
+ license: other
4
+ language:
5
+ - en
6
+ tags:
7
+ - math
8
+ ---
9
+
10
+ # InternLM-Step-Prover
11
+
12
+ <div align="center">
13
+
14
+ <img src="https://raw.githubusercontent.com/InternLM/InternLM/main/assets/logo.svg" width="200"/>
15
+ <div> </div>
16
+ <div align="center">
17
+ <b><font size="5">InternLM-Step-Prover</font></b>
18
+ <sup>
19
+ <a href="https://internlm.intern-ai.org.cn/">
20
+ <i>
21
+ </a>
22
+ </sup>
23
+ <div> </div>
24
+ </div>
25
+
26
+ A state-of-the-art LEAN4 step prover.
27
+
28
+ [💻 Github](https://github.com/InternLM/InternLM-Math)
29
+ </div>
30
+
31
+ # Dialogue Example
32
+ ### Input
33
+ DECL MyNat.mul_pow
34
+ GOAL a b n : N
35
+ ⊢ (a * b) ^ n = a ^ n * b ^ n
36
+ ### Output
37
+ PROOFSTEP induction n with t Ht
38
+
39
+
40
+ # Performance
41
+
42
+ ## MiniF2F
43
+ | Method | Model size | Pass | miniF2F-valid | miniF2F-test |
44
+ |--------|------------|------|---------------|--------------|
45
+ | **Whole-Proof Generation Methods** |
46
+ | GPT-4-turbo 0409 | - | 64 | 25.4% | 23.0% |
47
+ | DeepSeekMath-Base | 7B | 128 | 25.4% | 27.5% |
48
+ | DeepSeek-Prover | 7B | 1 | - | 30.0% |
49
+ | | | 64 | - | 46.3% |
50
+ | | | 128 | - | 46.3% |
51
+ | | | 8192 | - | 48.8% |
52
+ | | | 65536 | - | 50.0% |
53
+ | | | cumulative | *60.2%* | *52.0%* |
54
+ | TheoremLlama | - | cumulative | 36.5% | 33.6% |
55
+ | **Tree Search Methods** |
56
+ | COPRA (GPT-3.5) | - | 1 | - | 9.0% |
57
+ | COPRA (GPT-4) | - | 1 | - | 26.6% |
58
+ | DSP(Isabelle) | 540B | 100 | 42.6% | 38.9% |
59
+ | Proof Artifact Co-Training | 837M | 1 | 23.9% | 24.6% |
60
+ | | | 8 | 29.3% | 29.2% |
61
+ | ReProver | 229M | 1 | - | 25.0% |
62
+ | Llemma | 7B | 1 | 26.2% | 26.2% |
63
+ | Llemma | 34B | 1 | 27.9% | 25.8% |
64
+ | Curriculum Learning | 837M | 1 | 33.6% | 29.6% |
65
+ | | | 8 | 41.2% | 34.5% |
66
+ | | | 64 | 47.3% | 36.6% |
67
+ | Hypertree Proof Search | 600M | cumulative | 58.6% | - |
68
+ | | | 64 | - | 41.0% |
69
+ | Lean-STaR | 7B | 64 | - | 46.3% |
70
+ | InternLM2-Math | 7B | 1 | 29.9% | 30.3% |
71
+ | InternLM2-Math-Plus | 7B | 1 | - | 43.4% |
72
+ | InternLM2-Step-Prover | 7B | 1 | 59.8% | 48.8% |
73
+ | InternLM2-Step-Prover | 7B | 64 | **63.9%** | **54.5%** |
74
+
75
+ ## Proofnet & Putnam
76
+ | Method | Model size | Pass | result |
77
+ |--------|------------|------|--------|
78
+ | **ProofNet benchmark** |
79
+ | ReProver | 229M | 1 | 13.8% |
80
+ | InternLM2-Step-Prover | 7B | 1 | **18.1%** |
81
+ | **Putnam benchmark** |
82
+ | GPT-4 | - | 10 | 1/640 |
83
+ | COPRA (GPT-4) | - | 10 | 1/640 |
84
+ | DSP(Isabelle) | 540B | 10 | 4/640 |
85
+ | ReProver | 229M | 1 | 0/640 |
86
+ | InternLM2-Step-Prover | 7B | 1 | **5/640** |
87
+
88
+ # Citation and Tech Report