Update README.md

README.md

@@ -7,7 +7,7 @@ tags:
 license: apache-2.0
 
 ---
-# Polyglot-Ko-3.8B
+# Polyglot-Ko-1.3B
 
 ## Model Description
 Polyglot-Ko is a series of large-scale Korean autoregressive language models made by the EleutherAI polyglot team. Polyglot-Ko-3.8B is the second one.
@@ -31,15 +31,38 @@ dimensions of each head. The model is trained with a tokenization vocabulary of
 
 ## Training data
 
-Polyglot-Ko was trained on 1.2TB Korean Dataset, a large-scale curated dataset created by [TUNiB](https://tunib.ai/).
+Polyglot-Ko was trained on 863 GB of Korean language data (1.2TB before processing), a large-scale dataset curated by [TUNiB](https://tunib.ai/). The data collection process has abided by South Korean laws. This dataset was collected for the purpose of training Polyglot-Ko models, so it will not be released for public use.  
+
+| Source                              |Size (GB) | Link                                  |
+|-------------------------------------|---------|------------------------------------------|
+| Korean blog posts                   | 682.3   | -                                        |
+| Korean news dataset                 | 87.0  | -                                        |
+| Modu corpus                         | 19.0  |corpus.korean.go.kr             |
+| Korean patent dataset               | 26.4  | -                                        |
+| Korean Q & A dataset                | 18.1  | -                                        |
+| KcBert dataset                      | 12.7  | github.com/Beomi/KcBERT          |
+| Korean fiction dataset              | 6.1   | -                                        |
+| Korean online comments              | 4.2   | -                                    |
+| Korean wikipedia                    | 1.4   | ko.wikipedia.org                 |
+| Clova Call                          | < 1.0   | github.com/clovaai/ClovaCall     |
+| Naver sentiment movie corpus        | < 1.0   | github.com/e9t/nsmc              |
+| Korean hate speech dataset          | < 1.0   | -                                        |
+| Open subtitles                      | < 1.0  | opus.nlpl.eu/OpenSubtitles.php   |
+| AIHub various tasks datasets        | < 1.0   |aihub.or.kr                 |
+| Standard Korean language dictionary | < 1.0  | stdict.korean.go.kr/main/main.do |
+                            
+Furthermore, in order to avoid the model memorizing and generating personally identifiable information (PII) in the training data, we masked out the following sensitive information in the pre-processing stage:
 
-## Training procedure
+* `<|acc|>` : bank account number
+* `<|rrn|>` : resident registration number
+* `<|tell|>` : phone number
 
-Polyglot-Ko was trained for 219 billion tokens over 105,000 steps on 256 * A100 GPUs with [GPT-NeoX framework](https://github.com/EleutherAI/gpt-neox). It was trained as an autoregressive language model, using cross-entropy loss to maximize the likelihood of predicting the next token correctly. 
+## Training procedure
+Polyglot-Ko-3.8B was trained for 219 billion tokens over 105,000 steps on 256 A100 GPUs with the [GPT-NeoX framework](https://github.com/EleutherAI/gpt-neox). It was trained as an autoregressive language model, using cross-entropy loss to maximize the likelihood of predicting the next token. 
 
 ## How to use
 
-This model can be easily loaded using the `AutoModelForCausalLM` functionality:
+This model can be easily loaded using the `AutoModelForCausalLM` class:
 
 ```python
 from transformers import AutoTokenizer, AutoModelForCausalLM
@@ -48,37 +71,16 @@ tokenizer = AutoTokenizer.from_pretrained("EleutherAI/polyglot-ko-3.8b")
 model = AutoModelForCausalLM.from_pretrained("EleutherAI/polyglot-ko-3.8b")
 ```
 
-## Data Risks
-
-Polyglot models learn an inner representation of the various languages that can be used to extract features useful for downstream tasks.
-The model is best at what it was pre-trained for, however, generating text from a prompt.
-
-### Privacy considerations
-General training algorithms for pre-trained language models have many hazards, that memorize personal information in training data. We added the following tokens to vocabulary to mitigate privacy problems and replaced much personal information with these tokens in data preprocessing steps.
-
-* `<|acc|>` : bank account number
-* `<|rrn|>` : resident registration number
-* `<|tell|>` : phone number
-
-### Limitations and Biases
-The core functionality of Polyglot is taking a string of text and predicting the next token. While language models are widely used for tasks other than this, there are a lot of unknowns with this work. When prompting Polyglot it is important to remember that the statistically most likely next token is often not the token that produces the most "accurate" text. Never depend upon Polyglot to produce factually accurate output. Depending upon the use case, Polyglot may produce socially unacceptable text.
+## Evaluation results
 
-As with all language models, it is hard to predict in advance how Polyglot will respond to particular prompts, and offensive content may occur without warning. We recommend having a human curate or filter the outputs before releasing them, both to censor undesirable content and to improve the quality of the results.
+We evaluate Polyglot-Ko-3.8B on [KOBEST dataset](https://arxiv.org/abs/2204.04541), a benchmark with 5 downstream tasks, against comparable models skt/ko-gpt-trinity-1.2B-v0.5, kakaobrain/kogpt and facebook/xglm-7.5B, using the prompts provided in the paper. 
 
-### Legal Restrictions
-Since there are laws in many countries related to data collection, we will collect data with due regard to the laws of those countries.
-Additionally, we plan to use the dataset to train our models, but we do not plan to make the dataset publicly available.
-
-## Evaluation results
-We used the [KOBEST dataset](https://arxiv.org/abs/2204.04541), which consists of five Korean downstream tasks, for evaluation.
-We added those tasks to [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) and utilized prompt templates described in the paper.
-We evaluted our model as well as two other Korean language models, i.e., skt/ko-gpt-trinity-1.2B-v0.5 and kakaobrain/kogpt for comparison.
-The following tables show the results when the number of few-shot examples differ. You can reproduce these results using [polyglot branch of lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness/tree/polyglot) and the following scripts.
+The following tables show the results when the number of few-shot examples differ. You can reproduce these results using the [polyglot branch of lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness/tree/polyglot) and the following scripts. For a fair comparison, all models were run under the same conditions and using the same prompts. In the tables, `n` refers to the number of few-shot examples. 
 
 ```console
 python main.py \
    --model gpt2 \
-   --model_args pretrained='EleutherAI/polyglot-ko-1.3b' \
+   --model_args pretrained='EleutherAI/polyglot-ko-3.8b' \
    --tasks kobest_copa,kobest_hellaswag \
    --num_fewshot $YOUR_NUM_FEWSHOT \
    --batch_size $YOUR_BATCH_SIZE \
@@ -86,35 +88,35 @@ python main.py \
    --output_path $/path/to/output/
 ```
 
-**We decided to show only COPA and HellaSwag from KOBEST because evaluated models performed similarly to random guesses or with high variance on other tasks.**
+**We show model performance on COPA and HellaSwag. On the other three tasks, the evaluated models all performed similarly close to random guessing.**
 
 ### COPA (F1)
 
-| Model                                                                                          | params | 0-shot | 5-shot | 10-shot | 50-shot |
-|------------------------------------------------------------------------------------------------|--------|--------|--------|---------|---------|
-| [facebook/xglm-7.5B](https://huggingface.co/facebook/xglm-7.5B)                                | 7.5B   | 0.6723 | 0.6731 | 0.6769  | 0.7119  |
-| [skt/ko-gpt-trinity-1.2B-v0.5](https://huggingface.co/skt/ko-gpt-trinity-1.2B-v0.5) &dagger;   | 1.2B   | 0.6696 | 0.6477 | 0.6419  | 0.6514  |
-| [kakaobrain/kogpt](https://huggingface.co/kakaobrain/kogpt) &ast;                              | 6.0B   | 0.7345 | 0.7287 | 0.7277  | 0.7479  |
-| [EleutherAI/polyglot-ko-1.3b](https://huggingface.co/EleutherAI/polyglot-ko-1.3b) (ours)       | 1.3B   | 0.7196 | 0.7193 | 0.7204  | 0.7206  |
+| Model                                                                                        | params | n=0 | n=5 | n=10 | n=50 |
+|----------------------------------------------------------------------------------------------|--------|--------|--------|---------|---------|
+| [skt/ko-gpt-trinity-1.2B-v0.5](https://huggingface.co/skt/ko-gpt-trinity-1.2B-v0.5)          | 1.2B   | 0.6696 | 0.6477 | 0.6419  | 0.6514  |
+| [kakaobrain/kogpt](https://huggingface.co/kakaobrain/kogpt)                                  | 6.0B   | 0.7345 | 0.7287 | 0.7277  | 0.7479  |
+| [facebook/xglm-7.5B](https://huggingface.co/facebook/xglm-7.5B)                              | 7.5B   | 0.6723 | 0.6731 | 0.6769  | 0.7119  |
+| [EleutherAI/polyglot-ko-1.3b](https://huggingface.co/EleutherAI/polyglot-ko-1.3b)            | 1.3B   | 0.7196 | 0.7193 | 0.7204  | 0.7206  |
 | **[EleutherAI/polyglot-ko-3.8b](https://huggingface.co/EleutherAI/polyglot-ko-3.8b) (this)** | **3.8B**   | **0.7595** | **0.7608** | **0.7638**  | **0.7788**  |
 
-<img src="https://user-images.githubusercontent.com/19511788/192492576-cdd80c5c-7c90-43e3-8a4b-7a8486878f23.png" width="800px">
+<img src="https://user-images.githubusercontent.com/19511788/192492558-d22d3cb0-dbbb-46fd-83e0-8bf9d81e4e9f.png" width="800px">
 
 ### HellaSwag (F1)
 
-| Model                                                                                          | params | 0-shot | 5-shot | 10-shot | 50-shot |
+| Model                                                                                          | params |n=0 | n=5 | n=10 | n=50 |
 |------------------------------------------------------------------------------------------------|--------|--------|--------|---------|---------|
+| [skt/ko-gpt-trinity-1.2B-v0.5](https://huggingface.co/skt/ko-gpt-trinity-1.2B-v0.5)            | 1.2B   | 0.4036 | 0.4    | 0.4011  | 0.4214  |
+| [kakaobrain/kogpt](https://huggingface.co/kakaobrain/kogpt)                                    | 6.0B   | 0.4599 | 0.456  | 0.4616  | 0.4754  |
 | [facebook/xglm-7.5B](https://huggingface.co/facebook/xglm-7.5B)                                | 7.5B   | 0.4261 | 0.437  | 0.4409  | 0.4517  |
-| [skt/ko-gpt-trinity-1.2B-v0.5](https://huggingface.co/skt/ko-gpt-trinity-1.2B-v0.5) &dagger;   | 1.2B   | 0.4036 | 0.4    | 0.4011  | 0.4214  |
-| [kakaobrain/kogpt](https://huggingface.co/kakaobrain/kogpt) &ast;                              | 6.0B   | 0.4599 | 0.456  | 0.4616  | 0.4754  |
-| [EleutherAI/polyglot-ko-1.3b](https://huggingface.co/EleutherAI/polyglot-ko-1.3b) (ours)       | 1.3B   | 0.4013 | 0.3984 | 0.417   | 0.4416  |
-| [EleutherAI/polyglot-ko-3.8b](https://huggingface.co/EleutherAI/polyglot-ko-3.8b) **(this)** | **3.8B**   | **0.4438** | **0.4786** | **0.4737**  | **0.4822**  |
+| [EleutherAI/polyglot-ko-1.3b](https://huggingface.co/EleutherAI/polyglot-ko-1.3b)              | 1.3B   | 0.4013 | 0.3984 | 0.417   | 0.4416  |
+| **[EleutherAI/polyglot-ko-3.8b](https://huggingface.co/EleutherAI/polyglot-ko-3.8b) (this)**   | **3.8B**   | **0.4438** | **0.4786** | **0.4737**  | **0.4822**  |
 
-<img src="https://user-images.githubusercontent.com/19511788/192492585-a976ee38-2967-446a-b577-94f219228f4d.png" width="800px">
+<img src="https://user-images.githubusercontent.com/19511788/192492568-f7114c8a-3967-41fa-a589-31fe07bad85d.png" width="800px">
 
-<p><strong>&dagger;</strong> The model card of this model provides evaluation results for the KOBEST dataset, but when we evaluated the model with the prompts described in the paper, we can't get similar results to it. Therefore, we checked the KOBEST paper and found that the results were similar to the fine-tuning results reported in the paper. Because we evaluated by prompt-based generation without fine-tuning the model, the results provided by the model card for the this model may differ.</p>
+## Limitations and Biases
 
-<p><strong>&ast;</strong> Since this model does not provide evaluation results with KOBEST dataset, we evaluated the model using lm-evaluation-harness ourselves. you can reproduce this result using the source code included in the polyglot branch of lm-evaluation-harness.</p>
+Polyglot has been trained to optimize next token prediction. Language models such as this are often used for a wide variety of tasks and it is important to be aware of possible unexpected outcomes. For instance, Polyglot will not always return the most factual or accurate response but the most statistically likely one. Polyglot may produce socially unacceptable or offensive content. We recommend having a human curator or other filtering mechanism to censor sensitive content. 
 
 ## Citation and Related Information
 ### BibTeX entry
@@ -146,7 +148,6 @@ See the License for the specific language governing permissions and
 limitations under the License.
 ```
 
-However, the model has the potential to generate unpredictable text as mentioned. Therefore, we are not responsible for any damages resulting from the use of the model.
-
 ### Acknowledgement
-This project would not have been possible without the computing resources provided by [Stability.ai](https://stability.ai). Thanks for providing a large amount of GPU resources. Furthermore, thanks to [TUNiB](https://tunib.ai) for providing a large-scale Korean dataset for this work.
+
+This project was made possible thanks to the computing resources from [Stability.ai](https://stability.ai), and thanks to [TUNiB](https://tunib.ai) for providing a large-scale Korean dataset for this work.