Use SAM and CLIP model to segment unique instances you want. You may use this repo to segment any instances in the picture with text prompts.
The main network architecture is as follows:
Clip Model Architecture
SAM Model Architecture
Install python packages via commands:
pip3 install -r requirements.txt
Download pretrained model weights
cd PROJECT_ROOT_DIR
bash scripts/download_pretrained_ckpt.sh
Instance segmentor first using sam model to get all obj's mask of the input image. Second using clip model to classify each mask with both image features and your text prompts features.
cd PROJECT_ROOT_DIR
export PYTHONPATH=$PWD:$PYTHONPATH
python tools/sam_clip_text_seg.py --input_image_path ./data/test_images/test_bear.jpg --text bear
Bear Instance Segmentation Result, Text Prompt: bear
Athelete Instance Segmentation Result, Text Prompt: athlete
Horse Instance Segmentation Result, Text Prompt: horse
Dog Instance Segmentation Result, Text Prompt: dog
Fish Instance Segmentation Result, Text Prompt: fish
Strawberry Instance Segmentaton Result, Text Prompt: strawberry
Glasses Instance Segmentaton Result, Text Prompt: glasses
Tv Instance Segmentaton Result, Text Prompt: television
Shoes Instance Segmentaton Result, Text Prompt: shoe
Bridge Instance Segmentaton Result, Text Prompt: bridge
Airplane Instance Segmentaton Result, Text Prompt: airplane
cd PROJECT_ROOT_DIR
export PYTHONPATH=$PWD:$PYTHONPATH
python tools/sam_clip_text_seg.py --input_image_path ./data/test_images/test_horse.jpg --text "horse,mountain,grass,sky,clouds,tree" --cls_score_thresh 0.5 --use_text_prefix
Horse Instance Segmentation Result, Text Prompt: horse,mountain,grass,sky,clouds,tree
Tv Instance Segmentaton Result, Text Prompt: television,audio system,tape recorder,box
Strawberry Instance Segmentaton Result, Text Prompt: strawberry,grapefruit,spoon,wolfberry,oatmeal
Frog Instance Segmentaton Result, Text Prompt: frog,turtle,snail,eye
Befor Optimize
After Optimize
Cluster first using sam model to get all obj's mask of the input image. Second using clip model to extract image features for each objects. Third calculate feature distance of every two object pairs. Finally using a similarity threshold to cluster source objects.
To test the cluster simply run
cd PROJECT_ROOT_DIR
export PYTHONPATH=$PWD:$PYTHONPATH
python tools/cluster_sam.py --input_image_path ./data/test_images/test_bear.jpg --simi_thresh 0.82
Bear Cluster Result
Horse Cluster Result
Each row represents source image, sam origin mask, ori masked image, clustered mask, cluster masked image
Integrate MobileSAM into the pipeline for lightweight and faster inference. If you want to use mobile-sam to segment your
image all you need to do is to modify ./config/sam.yaml file. Modify the model name field to vit_t and modify the
model weight file path to ./pretrained/sam/mobile_sam.pt
- Test different kinds of cluster method
- Using cluster result as input prompts to reseg the image via sam model
- Merge embedding feats of global image and masked image
Most of the repo's code borrows from opeai's clip repo and facebook's segment-anything repo:
