KhangHatto's picture
Update README.md
3f50938 verified
metadata
license: apache-2.0
datasets:
  - bigscience-data/roots_vi_binhvq_news_corpus
  - wikipedia
language:
  - vi
  - en
  - zh
library_name: transformers
tags:
  - t5
  - flant5
  - summarization
  - translation
  - question-answering
pipeline_tag: fill-mask

Extend vocabulary and Pretrain

We utilized SentencePiece to retrain a tokenizer for Vietnamese, English, and Chinese. This newly trained tokenizer's vocabulary was then combined with Flan-T5's original vocabulary, eliminating any duplicate tokens. The resulting merged vocabulary consists of 106611 tokens.

For a single-epoch continual pretraining, also referred to as incremental pretraining, we employed the Flan-T5-Large model. This pretraining was conducted on a diverse dataset exceeding 100 GB, incorporating the following sources:

  • NewsCorpus
  • Vietnamese Wikipedia
  • Vietnamese books
  • Vietnamese legal documents
  • Vietnamese legal text
  • English Wikipedia
  • Chinese Text

How to use

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("Hatto/HattoFlanT5-Large")  
model = AutoModelForSeq2SeqLM.from_pretrained("Hatto/HattoFlanT5-Large")
model.cuda()

Finetune and Benchmark

  • Wikilingua
  • Vietnews
  • Pho_NER
  • .....

Citation

  • Hatto
  • Ipcoms