Text Generation
Transformers
Safetensors
English
llama
climate
conversational
text-generation-inference
Inference Endpoints
File size: 5,272 Bytes
2276ebf
 
 
e3c1318
 
 
 
 
 
 
 
 
 
 
88c371f
 
 
2276ebf
aa0bd15
2276ebf
aa0bd15
 
e3c1318
aa0bd15
e3c1318
b792fa5
e3c1318
83326de
25b2a7b
e3c1318
511bcfe
 
b792fa5
e3c1318
91fa02b
aa0bd15
e3c1318
 
 
88c371f
e3c1318
b792fa5
 
 
e3c1318
 
b792fa5
aa0bd15
e3c1318
 
 
b792fa5
aa0bd15
e3c1318
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
b792fa5
 
aa0bd15
 
e3c1318
 
 
 
88c371f
b792fa5
 
e3c1318
 
 
 
 
 
 
b792fa5
 
aa0bd15
 
 
 
 
 
 
 
88c371f
 
 
aa0bd15
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
---
language:
- en
datasets:
- OpenAssistant/oasst1
- databricks/databricks-dolly-15k
base_model: meta-llama/Llama-2-70b-hf
tags:
- climate
co2_eq_emissions:
  emissions: 40600
  training_type: "pre-training"
  geographical_location: "Washington, USA"
  hardware_used: "8x NVIDIA H100 HBM"
license: other
license_name: climategpt-community-license
license_link: https://huggingface.co/eci-io/climategpt-70b/blob/main/LICENSE.txt
---
# ClimateGPT-70B

ClimateGPT is a family of AI models designed to synthesize interdisciplinary research on climate change.
ClimateGPT-70B is a 70 billion parameter transformer decoder model that was adapted from Llama-2 to the domain of climate science using continuous pre-training on a collection of 4.2B tokens from curated climate documents.
The model is further instruction fine-tuned on a dataset of instruction-completion pairs manually collected by AppTek in cooperation with climate scientists.
[ClimateGPT-7B](https://huggingface.co/eci-io/climategpt-7b) outperforms Llama-2-70B Chat on our climate-specific benchmarks.
The model is designed to be used together with retrieval augmentation to extend the knowledge, and increase the factuality of the model and with cascaded machine translation to increase the language coverage.

## Model Details
Explore the model lineage [here](https://huggingface.co/spaces/EQTYLab/lineage-explorer?repo=https://huggingface.co/eci-io/climategpt-70b). 

- **Powered by:** [Erasmus AI](https://erasmus.ai)
- **Trained with:** [AppTek](https://apptek.com)
- **Authenticated by:** [EQTYLab](https://eqtylab.io)
- **Model type:** decoder-only Transformer
- **Language(s) (NLP):** English
- **License:** [ClimateGPT Community License](https://huggingface.co/eci-io/climategpt-70b/blob/main/LICENSE.txt)
- **Continued pre-trained from:** Llama-2-70B
- **Context length:** 4K tokens
- **Input:** Text-only data
- **Output:** Model generates text only
- **Paper:** [arXiv:2401.09646](https://arxiv.org/abs/2401.09646)
- **Website:** [eci.io](https://eci.io)

## Uses
- This model is intended to be directly used as a question answering model that is specialized in the climate domain.
- The model is aimed at providing useful feedback for decision makers, scientists and journalists involved in climate discussions.
- The model can also be used as a starting point for interested developers for further fine-tuning.
- The model is NOT intended to be a general-purpose chatbot (although it has chat capabilities).
- For the full system including cascaded MT, RAG, etc., we recommend the user to go to our demo website: [eci.io](https://eci.io)
- **Despite the efforts from the development team to eliminate them, as every other chat-capable LLMs, this model may generate biased, offensive or inaccurate responses.**

## Downstream Use

ClimateGPT-70B is an instruction-tuned model that can be directly used for climate-specific question-answering applications.
It was trained to perform well with retrieval augmentation and supports up to 5 references in context.

The model was trained using ChatML so the following format should be followed when prompting, including the  `<|im_start|>`, `<|im_end|>` tags, `system`, `user`, `context` and `assistant` identifiers and `[[0]]`, `[[1]]]` etc. tokens to indicate references.
    
    """
    <|im_start|>system
    {system_message}<|im_end|>
    <|im_start|>user
    {prompt}<|im_end|>
    <|im_start|>context
    [[0]] "{reference1_title}", {reference1_year}
    {reference1_text}
    [[1]] "{reference2_title}", {reference2_year}
    {reference2_text}
    [...]<|im_end|>
    <|im_start|>assistant
    """

## Training
- For the Llama-2 training data, we refer the user to https://huggingface.co/meta-llama/Llama-2-70b-hf.
- For continued pre-training, 4.2B climate-specific tokens (tokenized by the Llama tokenizer) are used.
- For instruction fine-tuning, about 272K instruction-completion pairs (both in the climate domain but also general domain) are used.

## Evaluation

Detailed evaluation results are presented in our [paper](https://arxiv.org/abs/2401.09646) on our model card website: [eci.io/model-card](https://eci.io/model-card)

## Environmental Impact
- **Hardware Type:** 8x NVIDIA H100 HBM
- **Power Consumption per GPU:** 775W
- **Hours used:** 2,182 hrs
- **Cloud Provider:** MLFoundry
- **Compute Region:** Washington, USA
- **Energy Mix:** 100% Hydro Power (24g CO2eq/kWh according to IPCC 2014)
- **Carbon Emitted:** 40.6kg CO2eq

## Citation

If you find ClimateGPT is useful in your work, please cite it with:

```
@misc{thulke2024climategpt,
      title={ClimateGPT: Towards AI Synthesizing Interdisciplinary Research on Climate Change}, 
      author={David Thulke and Yingbo Gao and Petrus Pelser and Rein Brune and Rricha Jalota and Floris Fok and Michael Ramos and Ian van Wyk and Abdallah Nasir and Hayden Goldstein and Taylor Tragemann and Katie Nguyen and Ariana Fowler and Andrew Stanco and Jon Gabriel and Jordan Taylor and Dean Moro and Evgenii Tsymbalov and Juliette de Waal and Evgeny Matusov and Mudar Yaghi and Mohammad Shihadah and Hermann Ney and Christian Dugast and Jonathan Dotan and Daniel Erasmus},
      year={2024},
      eprint={2401.09646},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}
```