In this repo, we expand Segment Anything Model (SAM) to support text prompt input. The text prompt could be object-level:full_moon: (eg, dog) and part-level:last_quarter_moon: (eg, dog head). Furthermore，we build a Visual ChatGPT-based dialogue system 🤖💬 that flexibly calls various segmentation models when receiving instructions in the form of natural language.

Beyond class-agnostic mask segmentation, this repo contains:

• Grounded segment anything at both object level and part level.
• Interacting with models in the form of natural language.

These abilities come from a series of models, including:

Q: When will VLPart paper be released ?

A: The paper is under internal review and it will be released in about one month. Please stay tuned.

Q: What is the difference between Grounded SAM and this project ?

A: Grounded SAM is Grounded DINO SAM, and this project is GLIP/VLPart SAM. We believe any open-vocabulary (text prompt) object detection model can be used to combine with SAM.

See installation instructions.

# prepare your private OpenAI key (for Linux)
export OPENAI_API_KEY={Your_Private_Openai_Key}
python chatbot.py --load "ImageCaptioning_cuda:0, SegmentAnything_cuda:1, PartPromptSegmentAnything_cuda:1, ObjectPromptSegmentAnything_cuda:0"

wget https://github.com/Cheems-Seminar/segment-anything-and-name-it/releases/download/v1.0/swinbase_part_0a0000.pth
wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth

python demo_vlpart_sam.py --input_image assets/twodogs.jpeg --output_dir outputs_demo --text_prompt "dog head"

Result:

wget https://github.com/Cheems-Seminar/segment-anything-and-name-it/releases/download/v1.0/glip_large.pth

python demo_glip_sam.py --input_image assets/demo2.jpeg --output_dir outputs_demo --text_prompt "frog"

Result:

For multiple prompts, seperate each prompt with ., for example, --text_prompt "dog head. dog nose"

• ViT-H SAM model
• VLPart Swin-Base
• GLIP Swin-Large

This project is under the CC-BY-NC 4.0 license. See LICENSE for details.

A large part of the code is borrowed from segment-anything, CLIP, GLIP, Grounded-Segment-Anything, Visual ChatGPT. Many thanks for their wonderful works.

If you find this project helpful for your research, please consider citing the following BibTeX entry.

@misc{segrec2023,
  title =        {Grounded Segment Anything: From Objects to Parts},
  author =       {Sun, Peize and Chen, Shoufa and Luo, Ping},
  howpublished = {\url{https://github.com/Cheems-Seminar/segment-anything-and-name-it}},
  year =         {2023}
}

@article{vlpart2023,
  title   =  {Going Denser with Open-Vocabulary Part Segmentation},
  author  =  {Sun, Peize and Chen, Shoufa and Zhu, Chenchen and Xiao, Fanyi and Luo, Ping and Xie, Saining and Yan, Zhicheng},
  journal =  {Under Review},
  year    =  {2023}
}