5CD-AI
/

Vintern-1B-v2

@@ -1,199 +1,291 @@
 ---
 library_name: transformers
-tags: []
 ---
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
 ## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 ---
 library_name: transformers
+datasets:
+- 5CD-AI/Viet-OCR-VQA
+- 5CD-AI/Viet-Doc-VQA
+- 5CD-AI/Viet-Doc-VQA-II
+- Vi-VLM/Vista
+- 5CD-AI/Viet-Receipt-VQA
+- 5CD-AI/Viet-Sketches-VQA
+- 5CD-AI/Viet-Geometry-VQA
+- 5CD-AI/Viet-Wiki-Handwriting
+- 5CD-AI/Viet-ComputerScience-VQA
+- 5CD-AI/Viet-Handwriting-gemini-VQA
+- 5CD-AI/Viet-Menu-gemini-VQA
+- 5CD-AI/Viet-Vintext-gemini-VQA
+- 5CD-AI/Viet-OpenViVQA-gemini-VQA
+- 5CD-AI/Viet-Resume-VQA
+- 5CD-AI/Viet-ViTextVQA-gemini-VQA
+language:
+- vi
+- en
+pipeline_tag: visual-question-answering
 ---
+<div align="center">
+  <img src="Vintern_logo.png" width="700"/>
+</div>
+## Vintern-1B-v2 ❄️ (Viet-InternVL2-1B-v2) [\[🤗 HF Demo\]](https://huggingface.co/spaces/khang119966/Vintern) - The LLaVA 🌋 Challenger
+We are excited to introduce  **Vintern-1B-v2** the Vietnamese 🇻🇳 multimodal model that combines the advanced Vietnamese language model [Qwen2-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct)[1] with the latest visual model, [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px)[2], CVPR 2024. This model excels in tasks such as OCR-VQA, Doc-VQA, and Chart-VQA,... With only 1 billion parameters, it is **4096 context length** finetuned from the Viet-InternVL-1B model on over 3 million specialized image-question-answer pairs for optical character recognition 🔍, text recognition 🔤, document extraction 📑, and general QA. The model can be integrated into various on-device applications 📱, demonstrating its versatility and robust capabilities.
 ## Model Details
+|      Model Name      |                                     Vision Part                                     |                                        Language Part                                         |
+| :------------------: | :---------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------: |
+|      Vintern-1B-v2      |    [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px)    |            [Qwen2-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct)            |
+Vintern-1B-v2 is a multimodal large language model series, featuring models of various sizes. For each size, we release instruction-tuned models optimized for multimodal tasks. Vintern-1B-v2 consists of [InternViT-300M-448px](https://huggingface.co/OpenGVLab/InternViT-300M-448px), an MLP projector, and [Qwen2-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct).
+## Training details 📚
+The fine-tuning dataset was meticulously sampled in part from the following datasets:
+- [Viet-OCR-VQA](https://huggingface.co/datasets/5CD-AI/Viet-OCR-VQA) 📄
+- [Viet-Doc-VQA](https://huggingface.co/datasets/5CD-AI/Viet-Doc-VQA) 📚
+- [Viet-Doc-VQA-II](https://huggingface.co/datasets/5CD-AI/Viet-Doc-VQA-II) 📑
+- [Vista](https://huggingface.co/datasets/Vi-VLM/Vista) 🌄
+- [Viet-Receipt-VQA](https://huggingface.co/datasets/5CD-AI/Viet-Receipt-VQA) 🧾
+- [Viet-Sketches-VQA](https://huggingface.co/datasets/5CD-AI/Viet-Sketches-VQA) ✏️
+- [Viet-Geometry-VQA](https://huggingface.co/datasets/5CD-AI/Viet-Geometry-VQA) 📐
+- [Viet-Wiki-Handwriting](https://huggingface.co/datasets/5CD-AI/Viet-Wiki-Handwriting) 📝
+- [Viet-ComputerScience-VQA](https://huggingface.co/datasets/5CD-AI/Viet-ComputerScience-VQA) 💻
+- [Viet-Handwriting-gemini-VQA](https://huggingface.co/datasets/5CD-AI/Viet-Handwriting-gemini-VQA) ✍️
+- [Viet-Menu-gemini-VQA](https://huggingface.co/datasets/5CD-AI/Viet-Menu-gemini-VQA) 🍽️
+- [Viet-Vintext-gemini-VQA](https://huggingface.co/datasets/5CD-AI/Viet-Vintext-gemini-VQA) 🏷️
+- [Viet-OpenViVQA-gemini-VQA](https://huggingface.co/datasets/5CD-AI/Viet-OpenViVQA-gemini-VQA) 🔍
+- [Viet-Resume-VQA](https://huggingface.co/datasets/5CD-AI/Viet-Resume-VQA) 📃
+- [Viet-ViTextVQA-gemini-VQA](https://huggingface.co/datasets/5CD-AI/Viet-ViTextVQA-gemini-VQA) 📑
+## Benchmarks 📈
+Since there are still many different metrics that need to be tested, we chose a quick and simple metric first to guide the development of our model. Our metric is inspired by Lavy[4]. For the time being, we are using GPT-4 to evaluate the quality of answers on two datasets: OpenViVQA and ViTextVQA. Detailed results can be found at the provided . The inputs are images, questions, labels, and predicted answers. The model will return a score from 0 to 10 for the corresponding answer quality. The results table is shown below.
+<table border="1" cellspacing="0" cellpadding="5">
+    <tr align="center">
+        <td rowspan="2"><b>Model</b></td>
+        <td colspan="2"><b>Gpt4o-Score</b></td>
+    </tr>
+    <tr align="center">
+        <td><b>OpenViVQA-dev</b></td>
+        <td><b>ViTextVQA-dev</b></td>
+    </tr>
+    <tr align="center">
+        <td align="left">Vintern-1B</td>
+        <td>7.1/10</td>
+        <td>7.6/10</td>
+    </tr>
+    <tr align="center">
+        <td align="left">Vintern-1B-v2</td>
+        <td>....</td>
+        <td>....</td>
+    </tr>
+</table>
+We are still working on more detailed benchmarks.
+## Examples
+<div align="center">
+  <img src="ex_images/1.jpg" width="500"/>
+</div>
+```
+User: <image>
+Mô tả hình ảnh một cách chi tiết.
+Assistant: Bức ảnh là một giấy tờ có tiêu đề "SƠ YẾU LÝ LỊCH" được in trên nền màu xanh dương nhạt. Phía trên cùng bên trái là dòng chữ "GIÁO HỘI PHẬT GIÁO VIỆT NAM", bên phải là dòng chữ "CỘNG HÒA XÃ HỘI CHỦ NGHĨA VIỆT NAM". Dưới tiêu đề là thông tin cá nhân của người được sơ yếu lý lịch, bao gồm:
+- Họ và tên khai sinh: VƯƠNG TẤN VIỆT
+- Họ và tên gia pha: HỒ CHÍ VIỆT
+- Chân danh, Pháp hiệu: CHÂN QUANG
+- Ngày và nơi sinh: 09/12/1959 tại Buôn Mê Thuột
+- Giấy CMND số: 273038848... cấp tại CA Vũng Tàu ngày 04/05/2005.
+- Quê quán: Nghệ An
+- Hộ khẩu thường trú: Chùa Phật Quang - Chu Hai - Tân Hải - Tân Hải - Tân Thành - BR. VT
+- Phòng ở: Chùa Phật Quang - Chu Hai - Tân Hải - Tân Thành - BR. VT
+- Trình độ: + Văn hóa: Cử nhân Anh Văn
+- + Ngoại ngữ: + Phát học (chuyên môn): Diễn giảng
+- Ngày và nơi xuất trình: 15/01/1980 tại Thiên Viện Thường Chiểu
+```
+<div align="center">
+  <img src="ex_images/2.jpg" width="500"/>
+</div>
+```
+User: <image>
+Mô tả hình ảnh một cách chi tiết.
+Assistant: Bức ảnh chụp một hóa đơn của cửa hàng Circle K Việt Nam. Hóa đơn có tiêu đề "CIRCLE K VIETNAM" ở đầu trang, bên dưới là thông tin liên lạc của cửa hàng: "113 Trần Đại Nghĩa - Hà Nội", "Date: Oct 5 2019 06:39 PM SFB",
+"Cashier: 5360-Linh Luong Thi Thuy", "Description: 1 DASANI Bottled Water 1.5L+1PT 12.000", "Item(s) (VAT included): 12.000", "CASH", "CHANGE", "Chỉ xuất hóa đơn trong ngày", "Xin cảm ơn quy khách!"
+```
+<div align="center">
+  <img src="ex_images/3.jpg" width="400"/>
+</div>
+```
+User: <image>
+Mô tả hình ảnh một cách chi tiết.
+Assistant: Bức ảnh là một thông báo tuyển dụng công tác viên kinh doanh của FPT Telecom Bình Thuận. Bố cục được chia thành 4 phần chính:
+1. **Phần tiêu đề:** Gồm logo FPT Telecom Bình Thuận và dòng chữ "FPT TELECOM BÌNH THUẬN TUYỂN DỤNG CÔNG TÁC VIÊN KINH DOANH".
+2. **Phần lịch phỏng vấn:** Liệt kê thời gian và ngày tháng cụ thể cho lịch phỏng vấn (9h00 - 10h00 thứ Sáu hàng tuần).
+3. **Phần chi tiết công việc:** Liệt kê các nhiệm vụ liên quan đến công việc, bao gồm:
+    - Tìm kiếm thông tin, tiếp cận khách hàng tiềm năng.
+    - Tư vấn, giải thích cho khách hàng về dịch vụ của FPT Telecom.
+    - Thực hiện thủ tục ký kết hợp đồng với khách hàng.
+4. **Phần yêu cầu công việc:** Liệt kê các yêu cầu cần thiết cho ứng viên, bao gồm:
+    - Nam, nữ không giới hạn độ tuổi.
+    - Đam mê kinh doanh.
+    - Không cần kinh nghiệm.
+    - Phù hợp với Sinh viên, người cần kiếm thêm thu nhập, làm online,....
+5. **Phần quyền lợi:** Liệt kê các quyền lợi mà ứng viên sẽ nhận được, bao gồm:
+    - Thời gian linh động.
+    - Thu nhập hấp dẫn.
+    - Môi trường làm việc thân thiện, được hướng dẫn tận tình.
+    - Cơ hội trở thành nhân viên chính thức của FPT và được hưởng các chế độ, quyền lợi dành cho nhân viên chính thức.
+6. **Phần hồ sơ chuẩn bị:** Liệt kê các giấy tờ cần chuẩn bị cho ứng viên, bao gồm:
+    - 01 Bản sao hộ khẩu có xác thực của địa phương.
+    - 01 Bản sao CMND có xác thực của địa phương.
+7. **Phần thông tin:**
+    - Địa chỉ VP: 07 Tuyên Quang - Phan Thiết - Bình Thuận.
+    - Liên hệ: Ms.Lộc - 0908 391 305.
+8. **Phần lưu ý:** Nêu rõ FPT Telecom không thu bất kỳ khoản phí nào từ ứng viên.
+```
+## Quickstart
+Here provides a code snippet to show you how to load the tokenizer and model and how to generate contents.
+[**Quickstart with Google Colab📒 ⬅️ ⬅️**](https://colab.research.google.com/drive/1ZD1oB56PF0lF66RCuTVJYLTEV0tM3CFf?usp=sharing)
+```python
+import numpy as np
+import torch
+import torchvision.transforms as T
+# from decord import VideoReader, cpu
+from PIL import Image
+from torchvision.transforms.functional import InterpolationMode
+from transformers import AutoModel, AutoTokenizer
+IMAGENET_MEAN = (0.485, 0.456, 0.406)
+IMAGENET_STD = (0.229, 0.224, 0.225)
+def build_transform(input_size):
+    MEAN, STD = IMAGENET_MEAN, IMAGENET_STD
+    transform = T.Compose([
+        T.Lambda(lambda img: img.convert('RGB') if img.mode != 'RGB' else img),
+        T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC),
+        T.ToTensor(),
+        T.Normalize(mean=MEAN, std=STD)
+    ])
+    return transform
+def find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size):
+    best_ratio_diff = float('inf')
+    best_ratio = (1, 1)
+    area = width * height
+    for ratio in target_ratios:
+        target_aspect_ratio = ratio[0] / ratio[1]
+        ratio_diff = abs(aspect_ratio - target_aspect_ratio)
+        if ratio_diff < best_ratio_diff:
+            best_ratio_diff = ratio_diff
+            best_ratio = ratio
+        elif ratio_diff == best_ratio_diff:
+            if area > 0.5 * image_size * image_size * ratio[0] * ratio[1]:
+                best_ratio = ratio
+    return best_ratio
+def dynamic_preprocess(image, min_num=1, max_num=12, image_size=448, use_thumbnail=False):
+    orig_width, orig_height = image.size
+    aspect_ratio = orig_width / orig_height
+    # calculate the existing image aspect ratio
+    target_ratios = set(
+        (i, j) for n in range(min_num, max_num + 1) for i in range(1, n + 1) for j in range(1, n + 1) if
+        i * j <= max_num and i * j >= min_num)
+    target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1])
+    # find the closest aspect ratio to the target
+    target_aspect_ratio = find_closest_aspect_ratio(
+        aspect_ratio, target_ratios, orig_width, orig_height, image_size)
+    # calculate the target width and height
+    target_width = image_size * target_aspect_ratio[0]
+    target_height = image_size * target_aspect_ratio[1]
+    blocks = target_aspect_ratio[0] * target_aspect_ratio[1]
+    # resize the image
+    resized_img = image.resize((target_width, target_height))
+    processed_images = []
+    for i in range(blocks):
+        box = (
+            (i % (target_width // image_size)) * image_size,
+            (i // (target_width // image_size)) * image_size,
+            ((i % (target_width // image_size)) + 1) * image_size,
+            ((i // (target_width // image_size)) + 1) * image_size
+        )
+        # split the image
+        split_img = resized_img.crop(box)
+        processed_images.append(split_img)
+    assert len(processed_images) == blocks
+    if use_thumbnail and len(processed_images) != 1:
+        thumbnail_img = image.resize((image_size, image_size))
+        processed_images.append(thumbnail_img)
+    return processed_images
+def load_image(image_file, input_size=448, max_num=12):
+    image = Image.open(image_file).convert('RGB')
+    transform = build_transform(input_size=input_size)
+    images = dynamic_preprocess(image, image_size=input_size, use_thumbnail=True, max_num=max_num)
+    pixel_values = [transform(image) for image in images]
+    pixel_values = torch.stack(pixel_values)
+    return pixel_values
+model = AutoModel.from_pretrained(
+    "5CD-AI/Viet-InternVL2-1B",
+    torch_dtype=torch.bfloat16,
+    low_cpu_mem_usage=True,
+    trust_remote_code=True,
+).eval().cuda()
+tokenizer = AutoTokenizer.from_pretrained("5CD-AI/Viet-InternVL2-1B", trust_remote_code=True, use_fast=False)
+test_image = 'test-image.jpg'
+pixel_values = load_image(test_image, max_num=12).to(torch.bfloat16).cuda()
+generation_config = dict(max_new_tokens= 1024, do_sample=False, num_beams = 3, repetition_penalty=2.5)
+question = '<image>\nMô tả hình ảnh một cách chi tiết.'
+response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=None, return_history=True)
+print(f'User: {question}\nAssistant: {response}')
+#question = "Câu hỏi khác ......"
+#response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=history, return_history=True)
+#print(f'User: {question}\nAssistant: {response}')
+```
+## Citation
+```
+@article{vintern1b,
+  title={Vintern-1B: An Efficient Multimodal Model for Vietnamese},
+  author={Doan, Thanh-Khang and Huynh, Gia-Bao and Pham, Dinh-Thuc and Hoang, Tien-Dung and Pham, Huynh-Nhat},
+  year={2024},
+  url={https://huggingface.co/5CD-AI/Viet-InternVL2-1B}
+}
+```
+## References
+[1] Yang, An, et al. "Qwen2 technical report." arXiv preprint arXiv:2407.10671 (2024).
+[2] Chen, Zhe, et al. "Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
+[3] Chen, Zhe, et al. "How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites." arXiv preprint arXiv:2404.16821 (2024).
+[4] Tran, Chi, and Huong Le Thanh. "LaVy: Vietnamese Multimodal Large Language Model." arXiv preprint arXiv:2404.07922 (2024).