LLaVA-HR

🌋🌋 LLaVA-HR: High-Resolution Large Language-Vision Assistant 🌋🌋

✨Technical Report:

Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models
Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, Rongrong Ji

This repository contains the implementation of LLaVA-HR, a strong and efficient MLLM powered by our mixture-of-resolution adaptation. The features of LLaVA-HR include:

• High Image Resolutions: LLaVA-HR supports up to 1536 x 1536 image resolutions, which boosts the performance of fine-grained vision-language tasks, such as TextVQA.
• Remarkable Efficiency: LLaVA-HR maintains the similar training costs with LLaVA-1.5, e.g., ~20 hours on 8 A100s. Its inference speed is also fast as existing low-resolution MLLMs ! Check out our paper.
• Strong Performance: LLaVA-HR outperforms existing MLLMs on multiple benchmarks, e.g., 82.6 on VQAv2. LLaVA-HR is comparable to LLaVA-NexT using the training data of LLaVA-1.5 ! Check out our model zoo.
• Fair Comparison: LLaVA-HR adopts the same training data and configurations with LLaVA-1.5, which means that the performance gains all come from our mixture-of-resolution adaptation. We hope that LLaVA-HR can be a strong baseline for the community.

📣 News

• [2024.04.16] We fix the evaluation bug for SQA and MMVet. Now, LLaVA-HR-X can achieve 40.3 score in MMVet! checking our model zoo.

• [2024.03.06] 🔥🔥🔥 We release LLaVA-HR, a high-resolution MLLM with strong performance and remarkable efficiency. LLaVA-HR greatly outperforms LLaVA-1.5 on multiple benchmarks, checking our model zoo.

Table of Contents

• Install
• Model Zoo
• Training
• Evaluation
• Demo

Install

1. Clone this repository and navigate to LLaVA-HR folder

git clone https://github.com/luogen1996/LLaVA-HR.git
cd LLaVA-HR

2. Install Package

conda create -n llava-hr python=3.10 -y
conda activate llava-hr
pip install --upgrade pip  # enable PEP 660 support
pip install -e .

3. Install additional packages for training cases

pip install ninja
pip install flash-attn --no-build-isolation

Model Zoo

Training

Our training pipeline and datasets are directly borrowed from LLaVA-v1.5. The training consists of two stages:

• Low-resolution pretraining: train a projector on a subset of ∼558K image-text pairs to connect a frozen pretrained vision encoder and a frozen LLM.
• High-resolution instruction tuning: adopt our MR-Adaptation to accommodate high-resolution images, and fine-tune the whole MLLM with multimodal instruction data.

bash scripts/v1_5/pretrain_llava_hr.sh

├── coco
│   └── train2017
├── gqa
│   └── images
├── ocr_vqa
│   └── images
├── textvqa
│   └── train_images
└── vg
    ├── VG_100K
    └── VG_100K_2

bash scripts/v1_5/train_eval_llava_hr.sh

Evaluation

We follow LLaVA-v1.5 to conduct evaluations. you should download eval.zip and unzip it to ./playground/data/eval. Besides, we further implement the evaluation of coco-caption, refcoco, vizwiz,ocrvqa and okvqa. Please refer to Evaluation.md to prepare the data.

Then, your can run our evaluation script bash scripts/v1_5/eval.sh.

🤗 Demo

Gradio Web UI

Here are the steps to run the demo on your local devices.

python -m llava.serve.gradio_web_server --controller http://localhost:10000 --model-list-mode reload

python -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path ./checkpoints/llava-hr-7b-sft-1024

python -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port <different from 40000, say 40001> --worker http://localhost:<change accordingly, i.e. 40001> --model-path <ckpt2>

CUDA_VISIBLE_DEVICES=0,1 python -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path ./checkpoints/llava-hr-7b-sft-1024

CLI Inference

Here is the command for chatting with LLaVA-HR without the need of Gradio interface.

python -m llava.serve.cli \
    --model-path ./checkpoints/llava-hr-7b-sft-1024 \
    --image-file "./assets/example.jpg" 

👍 Acknowledgement

• LLaVA The codebase we built upon, and our baseline LLaVA-1.5 already has strong multimodal capabilities.

🔒 License

• The majority of this project is released under the Apache 2.0 license as found in the LICENSE file.
• The service is a research preview intended for non-commercial use only, subject to the model License of LLaMA and Terms of Use of the data generated by OpenAI. Please contact us if you find any potential violation.

✏️ Citation

If you find our paper and code useful in your research, please consider giving a star ⭐️ and citation 📝.

@article{luo2024feast,
  title={Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models},
  author={Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, Rongrong Ji},
  journal={arXiv preprint arXiv:2403.03003},
  year={2024}
}

✨ Star History

[![Star