Awesome-Open-Vocabulary-Semantic-Segmentation

If you find this project helpful, please consider giving it a star ⭐.

Contents

• Open-Vocabulary Semantic Segmentation(mainly updated by @liuqinying)

  • Fully-supervised Methods
  • Weakly-supervised Methods
  • Training-free Methods
  • Others

• Zero-shot Semantic Segmentation (mainly updated by @liuqinying)

• Referring-Image-Segmentation (mainly updated by @ghost-000)

  • Fully-Supervised Methods
  • Weakly-Supervised Methods

• Open-Vocabulary Object Detection (mainly updated by @tbh3223)

• Universal Semantic Segmentation

• Other Related Work

• Related Survey

Open-Vocabulary Semantic Segmentation

Fully-Supervised Open-Vocabulary Semantic Segmentation

The model is trained on fully-supervised semantic segmentation datasets with pixel-level annotations (e.g., COCO Stuff dataset).

1. [LSeg] | ICLR'22 | Language-driven Semantic Segmentation | [pdf] | [code]
2. [OpenSeg] | ECCV'22 | Scaling Open-vocabulary Image Segmentation with Image-level Labels | [pdf] | [code]
3. [Xu et al.] | ECCV'22 | A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model | [pdf] | [code]
4. [SegCLIP] | ICML'23 | SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentation | [pdf] | [code]
5. [MaskCLIP] | ICML'23 | Open-Vocabulary Universal Image Segmentation with MaskCLIP | [pdf] | [code]
6. [OVSeg] | CVPR'23 | Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP | [pdf] | [code]
7. [X-Decoder] | CVPR'23 | Generalized Decoding for Pixel, Image, and Language | [pdf] | [code]
8. [SAN] | CVPR'23(Highlight) | Side Adapter Network for Open-Vocabulary Semantic Segmentation | [pdf] | [code]
9. [SAN] | TAPMI'23 | SAN: Side Adapter Network for Open-vocabulary Semantic Segmentation | [pdf] | [code]
10. [ODISE] | CVPR'23 | Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models | [pdf] | [code]
11. [FreeSeg] | CVPR'23 | FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation | [pdf] | [code]
12. [OpenSeeD] | ICCV'23 | A Simple Framework for Open-Vocabulary Segmentation and Detection | [pdf] | [code]
13. [GKC] | ICCV'23 | Global Knowledge Calibration for Fast Open-Vocabulary Segmentation | [pdf]
14. [OPSNet] | ICCV'23 | Open-vocabulary Panoptic Segmentation with Embedding Modulation | [pdf] | [code]
15. [MasQCLIP] | ICCV'23 | MasQCLIP for Open-Vocabulary Universal Image Segmentation | [pdf]
16. [DeOP] | ICCV'23 | Open Vocabulary Semantic Segmentation with Decoupled One-Pass Network | [pdf] | [code]
17. [Li et al.] | ICCV'23 | Open-vocabulary Object Segmentation with Diffusion Models | [pdf] | [code]
18. [HIPIE] | NeurIPS'23 | Hierarchical Open-vocabulary Universal Image Segmentation | [pdf] | [code]
19. [FC-CLIP] | NeurIPS'23 | Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP | [pdf] | [code]
20. [MAFT] | NeurIPS'23 | Learning Mask-aware CLIP Representations for Zero-Shot Segmentation | [pdf] | [code]
21. [ADA] | NeurIPS'23 | Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregation | [pdf]
22. [Dao et al] | TMM | Class Enhancement Losses with Pseudo Labels for Open-Vocabulary Semantic Segmentation | [pdf]
23. [SELF-SEG] | Arixv'23.12 | Self-Guided Open-Vocabulary Semantic Segmentation | [pdf]
24. [OpenSD] | Arixv'23.12 | OpenSD: Unified Open-Vocabulary Segmentation and Detection | [pdf] | [code]
25. [RENOVATE] | Arixv'24.03 | Renovating Names in Open-Vocabulary Segmentation Benchmarks | [pdf]
26. [DreamCLIP] | ECCV'24 | DreamLIP: Language-Image Pre-training with Long Captions | [pdf] | [code]
27. [CAT-Seg] | CVPR'24 | CAT-Seg : Cost Aggregation for Open-Vocabulary Semantic Segmentation | [pdf] | [code]
28. [SED] | CVPR'24 | SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation | [pdf] | [code]
29. [SCAN] | CVPR'24 | Open-Vocabulary Segmentation with Semantic-Assisted Calibration | [pdf] | [code]
30. [OpenTrans] | CVPR'24 | Transferable and Principled Efficiency for Open-Vocabulary Segmentation | [pdf] | [code])
31. [H-CLIP] | Arixv'24.05 | Parameter-efficient Fine-tuning in Hyperspherical Space for Open-vocabulary Semantic Segmentation | [pdf]
32. [OpenDAS] | Arixv'24.05 | OpenDAS: Domain Adaptation for Open-Vocabulary Segmentation | [pdf]
33. [USE] | CVPR'24 | USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation | [pdf]
34. [EBSeg] | CVPR'24 | Open-Vocabulary Semantic Segmentation with Image Embedding Balancing | [pdf] | [code])
35. [MAFT+] | ECCV'24 | Collaborative Vision-Text Representation Optimizing for Open-Vocabulary Segmentation | [pdf] | [code])

Weakly-Supervised Open-Vocabulary Semantic Segmentation

[text-supervised/language-supervised] The model is trained on weakly supervised datasets with only image-level annotations/captions (e.g., CC12M dataset).

1. [GroupViT] | CVPR'22 | GroupViT: Semantic Segmentation Emerges from Text Supervision | [pdf] | [code]
2. [ViL-Seg] | ECCV'22 | Open-world Semantic Segmentation via Contrasting and Clustering Vision-Language Embedding | [pdf]
3. [MaskCLIP+] | ECCV'22(Oral) | Extract Free Dense Labels from CLIP | [pdf] | [code]
4. [ViewCo] | ICLR'23 | Viewco: Discovering Text-supervised Segmentation Masks via Multi-view Semantic Consistency | [pdf]
5. [SegCLIP] | ICML'23 | SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentation | [pdf] | [code]
6. [CLIP-S4] | CVPR'23 | CLIP-S4: Language-Guided Self-Supervised Semantic Segmentation | [pdf]
7. [PACL] | CVPR'23 | Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning | [pdf]
8. [OVSegmentor] | CVPR'23 | Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervision | [pdf] | [code]
9. [SimSeg] | CVPR'23 | A Simple Framework for Text-Supervised Semantic Segmentation | [pdf] | [code]
10. [TCL] | CVPR'23 | Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairs | [pdf] | [code]
11. [SimCon] | Arxiv'23.02 | SimCon Loss with Multiple Views for Text Supervised Semantic Segmentation | [pdf]
12. [Zhang et al.] |