bczhou commited on
Commit
56e222e
β€’
1 Parent(s): fe51d49

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +173 -19
README.md CHANGED
@@ -22,6 +22,8 @@ pipeline_tag: image-text-to-text
22
 
23
 
24
  ## 🎉 News
 
 
25
  * **[2024.02.25]** Update evaluation scripts and docs!
26
  * **[2024.02.25]** Data descriptions out. Release TinyLLaVA-1.5B and TinyLLaVA-2.0B!
27
  * **[2024.02.24]** Example code on inference and model loading added!
@@ -32,7 +34,8 @@ pipeline_tag: image-text-to-text
32
 
33
  ## ⌛ TODO
34
  - [ ] Add support for Ollama and llama.cpp.
35
- - [ ] Developers' guide / How to build demo locally.
 
36
  - [x] Model Zoo descriptions.
37
  - [x] Examples and inference.
38
  - [x] Release code for training.
@@ -45,22 +48,17 @@ pipeline_tag: image-text-to-text
45
 
46
  - Our best model, TinyLLaVA-3.1B, achieves better overall performance against existing 7B models such as LLaVA-1.5 and Qwen-VL.
47
 
48
- ## 🐳 Model Zoo
49
- ### Legacy Model
50
- - [tiny-llava-hf](https://huggingface.co/bczhou/tiny-llava-v1-hf)
51
-
52
- ### Pretrained Models
53
- - [TinyLLaVA-3.1B](https://huggingface.co/bczhou/TinyLLaVA-3.1B)
54
- - [TinyLLaVA-2.0B](https://huggingface.co/bczhou/TinyLLaVA-2.0B)
55
- - [TinyLLaVA-1.5B](https://huggingface.co/bczhou/TinyLLaVA-1.5B)
56
-
57
- ### Model Details
58
- | Name | LLM | Checkpoint | LLaVA-Bench-Wild | MME | MMBench | MM-Vet | SQA-image | VQA-v2 | GQA | TextVQA |
59
- |---------------|-------------------|------------------------------------------------|------------------|----------|---------|--------|-----------|--------|-------|---------|
60
- | TinyLLaVA-3.1B | Phi-2 | [TinyLLaVA-3.1B](https://huggingface.co/bczhou/TinyLLaVA-3.1B) | 75.8 | 1464.9 | 66.9 | 32.0 | 69.1 | 79.9 | 62.0 | 59.1 |
61
- | TinyLLaVA-2.0B | StableLM-2-1.6B | [TinyLLaVA-2.0B](https://huggingface.co/bczhou/TinyLLaVA-2.0B) | 66.4 | 1433.8 | 63.3 | 32.6 | 64.7 | 78.9 | 61.9 | 56.4 |
62
- | TinyLLaVA-1.5B | TinyLlama | [TinyLLaVA-1.5B](https://huggingface.co/bczhou/TinyLLaVA-1.5B) | 60.8 | 1276.5 | 55.2 | 25.8 | 60.3 | 76.9 | 60.3 | 51.7 |
63
 
 
 
 
 
 
 
 
 
 
64
 
65
 
66
  ## 🔧 Requirements and Installation
@@ -86,15 +84,51 @@ pip install -e .
86
  pip install -e ".[train]"
87
  pip install flash-attn --no-build-isolation
88
  ```
89
- ### Upgrade to latest code base
90
 
91
  ```Shell
92
  git pull
93
  pip install -e .
 
94
  # if you see some import errors when you upgrade, please try running the command below (without #)
95
  # pip install flash-attn --no-build-isolation --no-cache-dir
96
  ```
97
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
98
 
99
  ## 🔧 Quick Start
100
 
@@ -105,7 +139,9 @@ pip install -e .
105
  from tinyllava.model.builder import load_pretrained_model
106
  from tinyllava.mm_utils import get_model_name_from_path
107
  from tinyllava.eval.run_tiny_llava import eval_model
 
108
  model_path = "bczhou/TinyLLaVA-3.1B"
 
109
  tokenizer, model, image_processor, context_len = load_pretrained_model(
110
  model_path=model_path,
111
  model_base=None,
@@ -113,6 +149,7 @@ tokenizer, model, image_processor, context_len = load_pretrained_model(
113
  )
114
  ```
115
  </details>
 
116
  ## &#x1F527; Run Inference
117
  Here's an example of running inference with [TinyLLaVA-3.1B](https://huggingface.co/bczhou/TinyLLaVA-3.1B)
118
  <details>
@@ -122,6 +159,7 @@ Here's an example of running inference with [TinyLLaVA-3.1B](https://huggingface
122
  from tinyllava.model.builder import load_pretrained_model
123
  from tinyllava.mm_utils import get_model_name_from_path
124
  from tinyllava.eval.run_tiny_llava import eval_model
 
125
  model_path = "bczhou/TinyLLaVA-3.1B"
126
  prompt = "What are the things I should be cautious about when I visit here?"
127
  image_file = "https://llava-vl.github.io/static/images/view.jpg"
@@ -139,13 +177,15 @@ args = type('Args', (), {
139
  "num_beams": 1,
140
  "max_new_tokens": 512
141
  })()
 
142
  eval_model(args)
143
  ```
144
  </details>
 
145
  ### Important
146
  We use different `conv_mode` for different models. Replace the `conv_mode` in `args` according to this table:
147
  | model | conv_mode |
148
- |-------------------|---------------|
149
  | TinyLLaVA-3.1B | phi |
150
  | TinyLLaVA-2.0B | phi |
151
  | TinyLLaVA-1.5B | v1 |
@@ -155,6 +195,115 @@ To ensure the reproducibility, we evaluate the models with greedy decoding.
155
 
156
  See [Evaluation.md](https://github.com/DLCV-BUAA/TinyLLaVABench/blob/main/docs/Evaluation.md)
157
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
158
 
159
  ## &#x270F; Citation
160
 
@@ -169,4 +318,9 @@ If you find our paper and code useful in your research, please consider giving a
169
  archivePrefix={arXiv},
170
  primaryClass={cs.LG}
171
  }
172
- ```
 
 
 
 
 
 
22
 
23
 
24
  ## &#x1F389; News
25
+ * **[2024.03.10]** base recipe out!
26
+ * **[2024.03.10]** Finetune scripts out!
27
  * **[2024.02.25]** Update evaluation scripts and docs!
28
  * **[2024.02.25]** Data descriptions out. Release TinyLLaVA-1.5B and TinyLLaVA-2.0B!
29
  * **[2024.02.24]** Example code on inference and model loading added!
 
34
 
35
  ## &#x231B; TODO
36
  - [ ] Add support for Ollama and llama.cpp.
37
+ - [x] Developers' guide / How to build demo locally.
38
+ - [x] Training and custom finetuning docs.
39
  - [x] Model Zoo descriptions.
40
  - [x] Examples and inference.
41
  - [x] Release code for training.
 
48
 
49
  - Our best model, TinyLLaVA-3.1B, achieves better overall performance against existing 7B models such as LLaVA-1.5 and Qwen-VL.
50
 
51
+ ## Contents
 
 
 
 
 
 
 
 
 
 
 
 
 
 
52
 
53
+ - [Install](#x1f527-requirements-and-installation)
54
+ - [Model Zoo](#x1f433-model-zoo)
55
+ - [Demo](#Demo)
56
+ - [Quick Start](#x1f527-quick-start)
57
+ - [Run Inference](#x1f527-run-inference)
58
+ - [Evaluation](#evaluation)
59
+ - [Data](#data-preparation)
60
+ - [Train](#train)
61
+ - [Custom Finetune](#custom-finetune)
62
 
63
 
64
  ## &#x1F527; Requirements and Installation
 
84
  pip install -e ".[train]"
85
  pip install flash-attn --no-build-isolation
86
  ```
87
+ ### Upgrade to the latest code base
88
 
89
  ```Shell
90
  git pull
91
  pip install -e .
92
+
93
  # if you see some import errors when you upgrade, please try running the command below (without #)
94
  # pip install flash-attn --no-build-isolation --no-cache-dir
95
  ```
96
 
97
+ ## &#x1F433; Model Zoo
98
+ ### Legacy Model
99
+ - [tiny-llava-hf](https://huggingface.co/bczhou/tiny-llava-v1-hf)
100
+
101
+ ### Pretrained Models
102
+ - [TinyLLaVA-3.1B](https://huggingface.co/bczhou/TinyLLaVA-3.1B)
103
+ - [TinyLLaVA-2.0B](https://huggingface.co/bczhou/TinyLLaVA-2.0B)
104
+ - [TinyLLaVA-1.5B](https://huggingface.co/bczhou/TinyLLaVA-1.5B)
105
+
106
+ ### Model Details
107
+ | Name | LLM | Checkpoint | LLaVA-Bench-Wild | MME | MMBench | MM-Vet | SQA-image | VQA-v2 | GQA | TextVQA |
108
+ |---------------|-------------------|------------------------------------------------|------------------|----------|---------|--------|-----------|--------|-------|---------|
109
+ | TinyLLaVA-3.1B | Phi-2 | [TinyLLaVA-3.1B](https://huggingface.co/bczhou/TinyLLaVA-3.1B) | 75.8 | 1464.9 | 66.9 | 32.0 | 69.1 | 79.9 | 62.0 | 59.1 |
110
+ | TinyLLaVA-2.0B | StableLM-2-1.6B | [TinyLLaVA-2.0B](https://huggingface.co/bczhou/TinyLLaVA-2.0B) | 66.4 | 1433.8 | 63.3 | 32.6 | 64.7 | 78.9 | 61.9 | 56.4 |
111
+ | TinyLLaVA-1.5B | TinyLlama | [TinyLLaVA-1.5B](https://huggingface.co/bczhou/TinyLLaVA-1.5B) | 60.8 | 1276.5 | 55.2 | 25.8 | 60.3 | 76.9 | 60.3 | 51.7 |
112
+
113
+
114
+ ## Demo
115
+
116
+ ### Gradio Web Demo
117
+
118
+ Launch a local web demo by running:
119
+ ```shell
120
+ python tinyllava/serve/app.py --model-path bczhou/TinyLLaVA-3.1B --model-name TinyLLaVA-3.1B
121
+ ```
122
+
123
+ ### CLI Inference
124
+
125
+ We also support running inference with CLI. To use our model, run:
126
+ ```shell
127
+ python -m tinyllava.serve.cli \
128
+ --model-path bczhou/TinyLLaVA-3.1B \
129
+ --image-file "./tinyllava/serve/examples/extreme_ironing.jpg"
130
+ ```
131
+
132
 
133
  ## &#x1F527; Quick Start
134
 
 
139
  from tinyllava.model.builder import load_pretrained_model
140
  from tinyllava.mm_utils import get_model_name_from_path
141
  from tinyllava.eval.run_tiny_llava import eval_model
142
+
143
  model_path = "bczhou/TinyLLaVA-3.1B"
144
+
145
  tokenizer, model, image_processor, context_len = load_pretrained_model(
146
  model_path=model_path,
147
  model_base=None,
 
149
  )
150
  ```
151
  </details>
152
+
153
  ## &#x1F527; Run Inference
154
  Here's an example of running inference with [TinyLLaVA-3.1B](https://huggingface.co/bczhou/TinyLLaVA-3.1B)
155
  <details>
 
159
  from tinyllava.model.builder import load_pretrained_model
160
  from tinyllava.mm_utils import get_model_name_from_path
161
  from tinyllava.eval.run_tiny_llava import eval_model
162
+
163
  model_path = "bczhou/TinyLLaVA-3.1B"
164
  prompt = "What are the things I should be cautious about when I visit here?"
165
  image_file = "https://llava-vl.github.io/static/images/view.jpg"
 
177
  "num_beams": 1,
178
  "max_new_tokens": 512
179
  })()
180
+
181
  eval_model(args)
182
  ```
183
  </details>
184
+
185
  ### Important
186
  We use different `conv_mode` for different models. Replace the `conv_mode` in `args` according to this table:
187
  | model | conv_mode |
188
+ |---------------- |----------- |
189
  | TinyLLaVA-3.1B | phi |
190
  | TinyLLaVA-2.0B | phi |
191
  | TinyLLaVA-1.5B | v1 |
 
195
 
196
  See [Evaluation.md](https://github.com/DLCV-BUAA/TinyLLaVABench/blob/main/docs/Evaluation.md)
197
 
198
+ ## Data Preparation
199
+
200
+ In our paper, we used two different datasets: the [LLaVA dataset](https://github.com/haotian-liu/LLaVA?tab=readme-ov-file#pretrain-feature-alignment) and the [ShareGPT4V dataset](https://github.com/InternLM/InternLM-XComposer/blob/main/projects/ShareGPT4V/docs/Data.md), and compared their differences. In this section, we provide information on data preparation.
201
+
202
+ ### Pretraining Images
203
+ * LLaVA: The pretraining images of LLaVA is from the 558K subset of the LAION-CC-SBU dataset.
204
+ * ShareGPT4V: The pretraining images of ShareGPT4V is a mixture of 558K LAION-CC-SBU subset, SAM dataset, and COCO dataset.
205
+
206
+ ### Pretraining Annotations
207
+ * LLaVA: The pretraining annotations of LLaVA are [here](https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain).
208
+ * ShareGPT4V: The pretraining annotations of ShareGPT4V are [here](https://huggingface.co/datasets/Lin-Chen/ShareGPT4V/blob/main/share-captioner_coco_lcs_sam_1246k_1107.json).
209
+
210
+
211
+ ### SFT Images & Annotations
212
+ The majority of the two SFT datasets are the same, with the exception that the 23K detailed description data in LLaVA-1.5-SFT being replaced with detailed captions randomly sampled from the [100K ShareGPT4V data](https://huggingface.co/datasets/Lin-Chen/ShareGPT4V/blob/main/sharegpt4v_instruct_gpt4-vision_cap100k.json).
213
+
214
+ ### Download data
215
+
216
+ 1. Download relevant images
217
+
218
+ - LAION-CC-SBU-558K: [images.zip](https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain/blob/main/images.zip)
219
+ - COCO: This dataset is from the [COCO2017 challenge](https://cocodataset.org/). Download: [train2017](http://images.cocodataset.org/zips/train2017.zip)
220
+ - WebData: This dataset is curated by the [ShareGPT4V project](https://github.com/InternLM/InternLM-XComposer/tree/main/projects/ShareGPT4V). Download: [images](https://drive.google.com/drive/folders/1tCUQ-sq6vdshZVkF0ZeF3K4eztkXJgax?usp=sharing). Only for academic usage.
221
+ - SAM: This dataset is collected by [Meta](https://ai.meta.com/datasets/segment-anything-downloads/). Download: [images](https://ai.meta.com/datasets/segment-anything-downloads/). We only use 000000~000050.tar for now. If you just want to use ShareGPT4V for SFT, you can quickly download 9K images from [here](https://drive.google.com/file/d/1dKumdOKSXtV7lIXdrG7jsIK_z2vZv2gs/view?usp=drive_link).
222
+ - GQA: [GQA project page](https://cs.stanford.edu/people/dorarad/gqa/about.html). Download: [images](https://downloads.cs.stanford.edu/nlp/data/gqa/images.zip)
223
+ - OCR-VQA: [OCR-VQA project page](https://ocr-vqa.github.io/). Download: [download script](https://drive.google.com/drive/folders/1_GYPY5UkUy7HIcR0zq3ZCFgeZN7BAfm_?usp=sharing). We save all files as `.jpg`
224
+ - TextVQA: [TextVQA project page](https://textvqa.org/). Download: [trainvalimages](https://dl.fbaipublicfiles.com/textvqa/images/train_val_images.zip)
225
+ - VisualGenome: [VisualGenome project page](https://homes.cs.washington.edu/~ranjay/visualgenome/index.html). Download: [part1](https://cs.stanford.edu/people/rak248/VG_100K_2/images.zip), [part2](https://cs.stanford.edu/people/rak248/VG_100K_2/images2.zip)
226
+
227
+
228
+ 2. Download relevant annotations
229
+
230
+ - LLaVA's pretraining annotations: [blip_laion_cc_sbu_558k.json](https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain)
231
+ - LLaVA's SFT annotations: [llava_v1_5_mix665k.json](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K/blob/main/llava_v1_5_mix665k.json)
232
+ - ShareGPT4V's pretraining annotations: [share-captioner_coco_lcs_sam_1246k_1107.json](https://huggingface.co/datasets/Lin-Chen/ShareGPT4V/blob/main/share-captioner_coco_lcs_sam_1246k_1107.json)
233
+ - ShareGPT4V's SFT annotations: [sharegpt4v_mix665k_cap23k_coco-ap9k_lcs3k_sam9k_div2k.json](https://huggingface.co/datasets/Lin-Chen/ShareGPT4V/blob/main/sharegpt4v_mix665k_cap23k_coco-ap9k_lcs3k_sam9k_div2k.json)
234
+
235
+
236
+ ### Organize Data
237
+
238
+ Organize the image files and annotation files as follows in `path/to/your/data`:
239
+
240
+ ```none
241
+ data
242
+ β”œβ”€β”€ llava
243
+ β”‚ β”œβ”€β”€ llava_pretrain
244
+ β”‚ β”‚ β”œβ”€β”€ images
245
+ β”‚ β”‚ β”œβ”€β”€ blip_laion_cc_sbu_558k.json
246
+ β”œβ”€β”€ coco
247
+ β”‚ β”œβ”€β”€ train2017
248
+ β”œβ”€β”€ sam
249
+ β”‚ β”œβ”€β”€ images
250
+ β”œβ”€β”€ gqa
251
+ β”‚ β”œβ”€β”€ images
252
+ β”œβ”€β”€ ocr_vqa
253
+ β”‚ β”œβ”€β”€ images
254
+ β”œβ”€β”€ textvqa
255
+ β”‚ β”œβ”€β”€ train_images
256
+ β”œβ”€β”€ vg
257
+ β”‚ β”œβ”€β”€ VG_100K
258
+ β”‚ β”œβ”€β”€ VG_100K_2
259
+ β”œβ”€β”€ share_textvqa
260
+ β”‚ β”œβ”€β”€ images
261
+ β”œβ”€β”€ web-celebrity
262
+ β”‚ β”œβ”€β”€ images
263
+ β”œβ”€β”€ web-landmark
264
+ β”‚ β”œβ”€β”€ images
265
+ β”œβ”€β”€ wikiart
266
+ β”‚ β”œβ”€β”€ images
267
+ β”œβ”€β”€ text_files
268
+ β”‚ β”œβ”€β”€ llava_v1_5_mix665k.json
269
+ β”‚ β”œβ”€β”€ share-captioner_coco_lcs_sam_1246k_1107.json
270
+ β”‚ β”œβ”€β”€ sharegpt4v_mix665k_cap23k_coco-ap9k_lcs3k_sam9k_div2k.json
271
+ ```
272
+
273
+ ## Train
274
+
275
+ **This section we describe the base recipe.**
276
+ ### Hyperparameters
277
+ Both hyperparameters used in pretraining and finetuning are provided below.
278
+
279
+ 1. Pretraining
280
+
281
+ | Hyperparameter | Global Batch Size | Learning rate | Epochs | Max length | Weight decay |
282
+ |----------------| ---: | ---: | ---: |-----------:| ---: |
283
+ | TinyLLaVA-3.1B | 256 | 1e-3 | 1 | 3072 | 0 |
284
+
285
+ 2. Finetuning
286
+
287
+ | Hyperparameter | Global Batch Size | Learning rate | Epochs | Max length | Weight decay |
288
+ |----------------| ---: | ---: | ---: |-----------:| ---: |
289
+ | TinyLLaVA-3.1B | 128 | 2e-5 | 1 | 3072 | 0 |
290
+
291
+ ### Pretrain
292
+
293
+ **Replace paths to your paths**
294
+
295
+ Training script with DeepSpeed ZeRO-2: [`pretrain.sh`](https://github.com/DLCV-BUAA/TinyLLaVABench/blob/main/scripts/tiny_llava/pretrain.sh).
296
+
297
+ ### Finetune
298
+
299
+ **Replace paths to your paths**
300
+
301
+ Training script with DeepSpeed ZeRO-3: [`finetune.sh`](https://github.com/DLCV-BUAA/TinyLLaVABench/blob/main/scripts/tiny_llava/finetune.sh).
302
+
303
+ ## Custom-Finetune
304
+
305
+ Check out our custom finetune using LoRA [here](https://github.com/DLCV-BUAA/TinyLLaVABench/blob/dev/docs/CUTOM_FINETUNE.md).
306
+
307
 
308
  ## &#x270F; Citation
309
 
 
318
  archivePrefix={arXiv},
319
  primaryClass={cs.LG}
320
  }
321
+ ```
322
+
323
+
324
+ ## ❀️ Community efforts
325
+ * Our codebase is built upon the [LLaVA](https://github.com/haotian-liu/LLaVA) project. Great work!
326
+ * Our project uses data from the [ShareGPT4V](https://github.com/InternLM/InternLM-XComposer/tree/main/projects/ShareGPT4V) project. Great work!