werty1248's picture
Update README.md
497e3e9 verified
metadata
library_name: transformers
license: llama3
base_model: beomi/Llama-3-Open-Ko-8B
datasets:
  - beomi/KoAlpaca-v1.1a
  - kyujinpy/OpenOrca-KO
  - nlpai-lab/openassistant-guanaco-ko
language:
  - ko
pipeline_tag: text-generation

Model Card

์•„์˜ค์ง€

Model Details

Model Description

beomi/Llama-3-Open-Ko-8B (์ตœ์‹  ๋ฒ„์ „)์˜ Instruction tuning ๋ฒ„์ „ ๋ชจ๋ธ์ž…๋‹ˆ๋‹ค.

Dataset

  • ์ƒ์—…์ ์œผ๋กœ ์ด์šฉ ๊ฐ€๋Šฅํ•œ ๋ฐ์ดํ„ฐ ์…‹์„ ์‚ฌ์šฉํ•˜์˜€์Šต๋‹ˆ๋‹ค.

  • ํ–ฅํ›„ ๋น„๊ต๋ฅผ ์œ„ํ•˜์—ฌ, ์ˆ˜ํ•™/์ฝ”๋”ฉ ๊ด€๋ จ ์งˆ๋ฌธ์ด ๋งŽ์€ ๋ฐ์ดํ„ฐ ์…‹(ex: kyujinpy/KOpen-platypus)์€ ์ œ์™ธํ•˜์˜€์Šต๋‹ˆ๋‹ค.

  • ๋ฉ€ํ‹ฐ ํ„ด ๋Œ€ํ™” ๋ฐ์ดํ„ฐ(nlpai-lab/openassistant-guanaco-ko)๋ฅผ ์ถ”๊ฐ€ํ•ด๋ณด์•˜์Šต๋‹ˆ๋‹ค.

  • beomi/KoAlpaca-v1.1a

    • 80% ํ™•๋ฅ ๋กœ instruction ์ถ”๊ฐ€("๋‹น์‹ ์€ ์ธ๊ณต์ง€๋Šฅ ๋น„์„œ์ž…๋‹ˆ๋‹ค. ์‚ฌ์šฉ์ž๊ฐ€ ๋‹ต์„ ์ดํ•ดํ•˜๊ธฐ ์œ„ํ•ด ์™ธ๋ถ€์—์„œ ๊ฒ€์ƒ‰ํ•  ํ•„์š”๊ฐ€ ์—†๋„๋ก ์ƒ์„ธํ•œ ๋‹ต๋ณ€์„ ์ œ๊ณตํ•˜์„ธ์š”.")
  • kyujinpy/OpenOrca-KO

    • ๋ณ€๊ฒฝ์‚ฌํ•ญ ์—†์Œ
  • nlpai-lab/openassistant-guanaco-ko

    • ๋ฉ€ํ‹ฐ ํ„ด ๋Œ€ํ™”์˜ ๊ฒฝ์šฐ, ๋งˆ์ง€๋ง‰ Assistant์˜ ๋‹ต๋ณ€์„ ๋ชฉํ‘œ๋กœ ํ•จ
    • Alpaca format์— ๋งž์ถฐ ์ด์ „ ๋Œ€ํ™” ๋‚ด์šฉ์— ๋Œ€ํ•œ ์ „์ฒ˜๋ฆฌ ์ˆ˜ํ–‰
    • 80% ํ™•๋ฅ ๋กœ instruction ์ถ”๊ฐ€ ("๋‹น์‹ ์€ ์ธ๊ณต์ง€๋Šฅ ๋น„์„œ์ž…๋‹ˆ๋‹ค. ์‚ฌ์šฉ์ž๊ฐ€ ๋‹ต์„ ์ดํ•ดํ•˜๊ธฐ ์œ„ํ•ด ์™ธ๋ถ€์—์„œ ๊ฒ€์ƒ‰ํ•  ํ•„์š”๊ฐ€ ์—†๋„๋ก ์ƒ์„ธํ•œ ๋‹ต๋ณ€์„ ์ œ๊ณตํ•˜์„ธ์š”.")

Training details

Training: Axolotl์„ ์ด์šฉํ•ด LoRA๋กœ 3epoch ํ•™์Šต ์‹œ์ผฐ์Šต๋‹ˆ๋‹ค.

  • lora_r: 32
  • lora_alpha: 32
  • lora_dropout: 0.05
  • gradient_accumulation_steps: 8
  • micro_batch_size: 4
  • num_epochs: 3
  • learning_rate: 0.0002 (2e-4)
  • lr_scheduler: cosine
  • warmup_steps: 50
  • sequence_len: 4096
  • bf16

ํ•™์Šต ์‹œ๊ฐ„: 1xA100, ์•ฝ 8์‹œ๊ฐ„

Evaluation

  • boolq๋ฅผ ์ œ์™ธํ•˜๋ฉด ํฐ ์ •ํ™•๋„ ํ–ฅ์ƒ์€ ์—†์—ˆ์Šต๋‹ˆ๋‹ค.
  • 5shot kobest (Accuracy)
Tasks werty1248/Llama-3-Ko-8B-OpenOrca beomi/Llama-3-Open-Ko-8B werty1248/Llama-3-Ko-8B-Instruct-AOG
kobest_boolq 0.7158ยฑ0.0120 0.7963ยฑ0.0108 0.8312ยฑ0.0100
kobest_copa 0.7620ยฑ0.0135 0.8110ยฑ0.0124 0.8120ยฑ0.0124
kobest_hellaswag 0.4740ยฑ0.0224 0.4780ยฑ0.0224 0.4700ยฑ0.0223
kobest_sentineg 0.9471ยฑ0.0112 0.9622ยฑ0.0096 0.9647ยฑ0.0093
kobest_wic 0.6079ยฑ0.0138 0.5778ยฑ0.0139 0.5937ยฑ0.0138

Format & Examples

  • Instruction - Input - Response ํ˜•ํƒœ๊ฐ€ ๊ถŒ์žฅ๋˜์ง€๋งŒ, ๊ฒฝํ—˜์ ์œผ๋กœ Question - Instruction - Response๋กœ ์‚ฌ์šฉํ•˜๋Š” ๊ฒƒ์ด ๋” ๋‚˜์€ ๊ฒƒ ๊ฐ™์Šต๋‹ˆ๋‹ค.

  • ์ž˜ ๋œ ๋‹ต๋ณ€ ์˜ˆ์‹œ

input = """### Question:
ํ”ผ๋ณด๋‚˜์น˜ ์ˆ˜์—ด์ด ๋ญ์•ผ? ๊ทธ๋ฆฌ๊ณ  ํ”ผ๋ณด๋‚˜์น˜ ์ˆ˜์—ด์— ๋Œ€ํ•ด ํŒŒ์ด์ฌ ์ฝ”๋“œ๋ฅผ ์งœ์ค˜๋ณผ๋ž˜?

### Instruction:
์นœ์ ˆํ•œ ์ฑ—๋ด‡์œผ๋กœ์„œ ์ƒ๋Œ€๋ฐฉ์˜ ์š”์ฒญ์— ์ตœ๋Œ€ํ•œ ์ž์„ธํ•˜๊ณ  ์นœ์ ˆํ•˜๊ฒŒ ๋‹ตํ•˜์ž.

### Response:
"""

ํ”ผ๋ณด๋‚˜์น˜ ์ˆ˜์—ด์€ 1์ฐจ์™€ 2์ฐจ ์ˆ˜๋ฅผ ์‚ฌ์šฉํ•ด ๋‹ค์Œ ์ˆ˜๋ฅผ ๊ตฌํ•˜๋Š” ๋ฐฉ์‹์œผ๋กœ ๋งŒ๋“ค์–ด์ง€๋Š” ์ˆœ์—ด์ž…๋‹ˆ๋‹ค. ์˜ˆ๋ฅผ๋“ค๋ฉด ํ”ผ๋ณด๋‚˜์น˜ ์ˆ˜์—ด์˜ ์ฒ˜์Œ 7๊ฐœ ์š”์†Œ๋Š” 1, 1, 2, 3, 5, 8, 13์ž…๋‹ˆ๋‹ค. ํ”ผ๋ณด๋‚˜์น˜ ์ˆ˜์—ด์˜ ๊ฐ ์œ„์น˜์—์„œ 2๊ฐœ ์ด์ „ ์œ„์น˜์˜ ์ˆ˜๋ฅผ ๋”ํ•˜์—ฌ ์ƒˆ๋กœ์šด ์š”์†Œ๊ฐ€ ๊ณ„์‚ฐ๋ฉ๋‹ˆ๋‹ค.

Python ์ฝ”๋“œ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.

def fibo(n):
    if n<2:
         return n
    else:
         return fibo(n-1)+fibo(n-2)

์ด๋ ‡๊ฒŒ ์ž‘์„ฑํ•จ์œผ๋กœ์จ ํ”ผ๋ณด๋‚˜์น˜ ์ˆ˜์—ด์˜ ์œ„์น˜ 0๋ถ€ํ„ฐ n๊นŒ์ง€ ๊ฐ ์š”์†Œ๊ฐ€ ๊ณ„์‚ฐ๋ฉ๋‹ˆ๋‹ค. 6์ด๋ผ๋Š” ์œ„์น˜๋กœ ๊ณ„์‚ฐํ•˜๋Š” ๊ฒฝ์šฐ ๋‹ค์Œ๊ณผ ๊ฐ™์€ ์ถœ๋ ฅ์ด ๋‚˜ํƒ€๋‚ฉ๋‹ˆ๋‹ค:

print(fibo(6))

fibonacci(6)==> 8<|end_of_text|>
  • ์ž˜๋ชป๋œ ๋‹ต๋ณ€(ํ• ๋ฃจ์‹œ๋„ค์ด์…˜) ์˜ˆ์‹œ
input = """### Question:
์„ธ์ข…๋Œ€์™•์˜ ์ง‘ํ˜„์ „ ๋งฅ๋ถ ๋˜์ง ์‚ฌ๊ฑด์— ๋Œ€ํ•ด์„œ ์•Œ๊ณ  ์žˆ์–ด?

### Instruction:
์นœ์ ˆํ•œ ์ฑ—๋ด‡์œผ๋กœ์„œ ์ƒ๋Œ€๋ฐฉ์˜ ์š”์ฒญ์— ์ตœ๋Œ€ํ•œ ์ž์„ธํ•˜๊ณ  ์นœ์ ˆํ•˜๊ฒŒ ๋‹ตํ•˜์ž.

### Response:
"""

๋„ค, 2007๋…„์— ์„ธ์ข…๋Œ€์™•์˜ ์ง‘ํ˜„์ „ ๋งฅ๋ถ ๋˜์ง„ ์‚ฌ๊ฑด์ด ์ผ์–ด๋‚ฌ์Šต๋‹ˆ๋‹ค. ๋™ํ™” ์ž‘๊ฐ€์ธ ์ด์ฃผ์˜์”จ๊ฐ€ ๊น€์†Œํฌ ์ž‘๊ฐ€์˜ ์ฑ…์„ ์ง‘ํ•„ํ•˜๋ฉด์„œ, ์ฑ… ๋‚ด์šฉ์ƒ ์›๋ž˜ ์žˆ์—ˆ๋˜ ์„ธ์ข…๋Œ€์™•์ด ๋งฅ๋ถ์„ ๋˜์กŒ๋‹ค๋Š” ๋‚ด์šฉ์„ ์ œ์žฌ๋กœ ํ•ด์„œ ์ถœํŒํ•˜์˜€๊ณ , ์ด์— ๊น€์†Œํฌ ์ž‘๊ฐ€๋Š” ์ง‘ํ˜„์ „์ด๋ž€ ์ด๋ฆ„์œผ๋กœ ์ด์ฃผ์˜ ์ž‘๊ฐ€์—๊ฒŒ ๋ฒ•์  ์†Œ์†ก์„ ๊ฑธ๊ฒŒ ๋˜์—ˆ์Šต๋‹ˆ๋‹ค. ๊ทธ๋Ÿฌ๋‚˜ ๊ฒฐ๋ก ์ ์œผ๋กœ, ๊น€์†Œํฌ ์ž‘๊ฐ€์˜ ์ง‘ํ˜„์ „์ด๋ž€ ๋‹จ์–ด๋Š” ํŠน์ • ๋‹จ์ฒด๋‚˜ ์žฅ์†Œ๋งŒ์„ ๋ช…๋ฐฑํ•˜๊ฒŒ ์‚ฌ์šฉํ•˜๋Š” 'ํ˜ธ์นญ'์˜ ์„ฑ๊ฒฉ๊ณผ ํ•จ๊ป˜ ๋‹ค์ˆ˜์˜ ์ง‘ํ˜„์ „ ์‚ฌ๋žŒ๋“ค์ด ๊ณต๋™ ์‚ฌ์šฉํ•˜๋Š” 'ํ‘œ์ค€'์˜ ์„ฑ๊ฒฉ์„ ๊ฐ€์กŒ์œผ๋ฏ€๋กœ, ์ด์ฃผ์˜ ์ž‘๊ฐ€์˜ ์ถœํŒ ํ–‰์œ„๊ฐ€ ๋ช…์˜ˆํ›ผ์†์— ํ•ด๋‹น๋˜๋Š”์ง€ ์—ฌ๋ถ€๋ฅผ ํŒ๋‹จํ•  ๋•Œ์—๋Š” ์ถœํŒ ๋‹น์‹œ ์ง‘ํ˜„์ „์ด ์ผ๋ฐ˜์ ์œผ๋กœ ์–ด๋–ค ์˜๋ฏธ๋ฅผ ๊ฐ€์ง€๊ณ  ์žˆ์—ˆ๋Š”์ง€๊ฐ€ ์ค‘์š”ํ•œ ์š”์†Œ๋กœ ๊ณ ๋ ค๋˜์—ˆ์Šต๋‹ˆ๋‹ค. ๋”ฐ๋ผ์„œ ๋‹จ์ˆœํžˆ ์ง‘ํ˜„์ „์ด ๊ฐ€๊ณ„์ƒ ํŠน์ • ๋‹จ์ฒด ๋ฐ ์žฅ์†Œ์™€ ๊ด€๋ จ๋œ ํ˜ธ์นญ์œผ๋กœ๋งŒ ์‚ฌ์šฉ๋˜์–ด์™”๋‹ค๋ฉด, ์ถœํŒ ํ–‰์œ„๋กœ ์ธํ•ด ๋ช…์˜ˆํ›ผ์†์„ ๋ฐ›์•˜๋‹ค๋Š” ๊น€์†Œํฌ ์ž‘๊ฐ€์˜ ์ฃผ์žฅ์€ ๋ฐ›์•„๋“ค์—ฌ์ง€์ง€ ์•Š์•˜์„ ์ˆ˜๋„ ์žˆ์Šต๋‹ˆ๋‹ค.<|end_of_text|>
  • ๋ฉ€ํ‹ฐํ„ด ๋Œ€ํ™” ์˜ˆ์‹œ (์‹ ๋ขฐ๋„ ๋‚ฎ์Œ)
"""### Question:
ํ”ผ๋ณด๋‚˜์น˜ ์ˆ˜์—ด์ด ๋ญ์•ผ? ๊ทธ๋ฆฌ๊ณ  ํ”ผ๋ณด๋‚˜์น˜ ์ˆ˜์—ด์— ๋Œ€ํ•ด ํŒŒ์ด์ฌ ์ฝ”๋“œ๋ฅผ ์งœ์ค˜๋ณผ๋ž˜?
### Response:
ํ”ผ๋ณด๋‚˜์น˜ ์ˆ˜์—ด์€ 1์ฐจ์™€ 2์ฐจ ์ˆ˜๋ฅผ ์‚ฌ์šฉํ•ด ๋‹ค์Œ ์ˆ˜๋ฅผ ๊ตฌํ•˜๋Š” ๋ฐฉ์‹์œผ๋กœ ๋งŒ๋“ค์–ด์ง€๋Š” ์ˆœ์—ด์ž…๋‹ˆ๋‹ค. ์˜ˆ๋ฅผ๋“ค๋ฉด ํ”ผ๋ณด๋‚˜์น˜ ์ˆ˜์—ด์˜ ์ฒ˜์Œ 7๊ฐœ ์š”์†Œ๋Š” 1, 1, 2, 3, 5, 8, 13์ž…๋‹ˆ๋‹ค. ํ”ผ๋ณด๋‚˜์น˜ ์ˆ˜์—ด์˜ ๊ฐ ์œ„์น˜์—์„œ 2๊ฐœ ์ด์ „ ์œ„์น˜์˜ ์ˆ˜๋ฅผ ๋”ํ•˜์—ฌ ์ƒˆ๋กœ์šด ์š”์†Œ๊ฐ€ ๊ณ„์‚ฐ๋ฉ๋‹ˆ๋‹ค.

Python ์ฝ”๋“œ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค.

def fibo(n):
    if n<2:
         return n
    else:
         return fibo(n-1)+fibo(n-2)

์ด๋ ‡๊ฒŒ ์ž‘์„ฑํ•จ์œผ๋กœ์จ ํ”ผ๋ณด๋‚˜์น˜ ์ˆ˜์—ด์˜ ์œ„์น˜ 0๋ถ€ํ„ฐ n๊นŒ์ง€ ๊ฐ ์š”์†Œ๊ฐ€ ๊ณ„์‚ฐ๋ฉ๋‹ˆ๋‹ค. 6์ด๋ผ๋Š” ์œ„์น˜๋กœ ๊ณ„์‚ฐํ•˜๋Š” ๊ฒฝ์šฐ ๋‹ค์Œ๊ณผ ๊ฐ™์€ ์ถœ๋ ฅ์ด ๋‚˜ํƒ€๋‚ฉ๋‹ˆ๋‹ค:

print(fibo(6))

fibonacci(6)==> 8
### Question:
๋” ๋น ๋ฅธ ์•Œ๊ณ ๋ฆฌ์ฆ˜์€ ์—†์„๊นŒ?

### Instruction:
์นœ์ ˆํ•œ ์ฑ—๋ด‡์œผ๋กœ์„œ ์ƒ๋Œ€๋ฐฉ์˜ ์š”์ฒญ์— ์ตœ๋Œ€ํ•œ ์ž์„ธํ•˜๊ณ  ์นœ์ ˆํ•˜๊ฒŒ ๋‹ตํ•˜์ž.

### Response:
"""

๋‹ค๋ฅธ Python ์ฝ”๋“œ๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์Šต๋‹ˆ๋‹ค:
def fibo(n):
    a, b = 0, 1
    for _ in range(n):
        a, b = b, a+b
    return a

์ด ์ฝ”๋“œ๋Š” ๋‘ ๊ฐœ์˜ ๋กœ์ปฌ ๋ณ€์ˆ˜ a ๋ฐ b๋ฅผ ์‚ฌ์šฉํ•˜๋Š” ์ดํ„ฐ๋ ˆ์ดํ„ฐ๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ๊ณ„์‚ฐ์„ ์ˆ˜ํ–‰ํ•ฉ๋‹ˆ๋‹ค. ์ดˆ๊ธฐํ™” ์‹œ a๋Š” ํ”ผ๋ณด๋‚˜์น˜ ์ˆ˜์—ด์˜ ์œ„์น˜ 0์˜ ๊ฐ’์œผ๋กœ ์„ค์ •๋˜๊ณ , b๋Š” ์ด์ „ ๊ณ„์‚ฐ์—์„œ ํ”ผ๋ด‡ ๊ณ„์—ด์˜ ์œ„์น˜ 0์˜ ๊ฐ’(์œ„์น˜ 0์„ ํฌํ•จํ•˜๋ฉด 1๋‹จ๊ณ„์—์„œ 1์ด๋ฉฐ, ์œ„์น˜ 1์„ ํฌํ•จํ•˜๋ฉด 2๋‹จ๊ณ„์—์„œ 1)์œผ๋กœ ์„ค์ •๋ฉ๋‹ˆ๋‹ค. ๋‹ค์Œ ๊ฐ ๋ฐ˜๋ณต์—์„œ a๋Š” ์ด์ „ ๋กœ์ปฌ ๋ณ€์ˆ˜ b์˜ ๊ฐ’๊ณผ ์ƒˆ๋กœ์šด ๊ฐ’์œผ๋กœ ๊ฐ’์ด ๋ฐ”๋€๋‹ˆ๋‹ค. ๋‘ ๋ฒˆ์งธ ํŒŒ๋ผ๋ฏธํ„ฐ __๋Š” ๊ฐ ๋ฐ˜๋ณต์—์„œ ๊ฐ’์ด ์ถœ๋ ฅ๋˜์ง€ ์•Š๋Š”๋‹ค๋Š”(๊ทธ๋Ÿฌ๋‚˜ ์—ฌ์ „ํžˆ ์‚ฌ์šฉ๋ฉ๋‹ˆ๋‹ค) ๊ฒƒ์„ ๋ฐ˜์˜ํ•˜๊ธฐ ์œ„ํ•ด ์„ ํƒ๋˜์—ˆ์Šต๋‹ˆ๋‹ค.<|end_of_text|>

License:

https://llama.meta.com/llama3/license