rwkv-4-raven / README.md
BlinkDL's picture
Update README.md
266f54e
|
raw
history blame
1.29 kB
metadata
language:
  - en
tags:
  - pytorch
  - text-generation
  - causal-lm
  - rwkv
license: apache-2.0
datasets:
  - the_pile

RWKV-4 "Raven"-series Models

Model Description

These are RWKV-4-Pile 1.5/3/7/14B models finetuned on Alpaca, CodeAlpaca, Guanaco, GPT4All, ShareGPT and more. Even the 1.5B model is surprisingly good for its size.

Gradio Demo: https://huggingface.co/spaces/BlinkDL/Raven-RWKV-7B

Use https://github.com/BlinkDL/ChatRWKV to run them.

See https://github.com/BlinkDL/RWKV-LM for details on the RWKV Language Model (100% RNN).

Best Prompt Format for Raven models, Bob is user, Alice is bot (NOTE: no space after final "Alice:"). You can use \n within xxxxxxxxxxx, but avoid \n\n.

Bob: xxxxxxxxxxxxxxxxxx\n\nAlice:
Bob: xxxxxxxxxxxxxxxxxx\n\nAlice: xxxxxxxxxxxxx\n\nBob: xxxxxxxxxxxxxxxx\n\nAlice:

New models will be named like Eng99%-Other1%, Eng86%-Chn10%-JpnEspKor2%-Other2%, etc. Language ratios determined by amount of ChatGPT data. Please share more ChatGPT data to increase the ratio of your language.

Old models:

  • RWKV-4-Raven-Eng : 99% English + 1% Multilang
  • RWKV-4-Raven-EngAndMore : 96% English + 2% Chn Jpn + 2% Multilang (More Jpn than v6 "EngChnJpn")
  • RWKV-4-Raven-ChnEng : 49% English + 50% Chinese + 1% Multilang

License: Apache 2.0