- We release the SegGPT model and inference code for segmentation everything, as well as some example images and videos.
git clone https://github.com/baaivision/Painter
cd Painter/SegGPT/SegGPT_inference && wget https://huggingface.co/BAAI/SegGPT/resolve/main/seggpt_vit_large.pth
pip install -r requirements.txt
Main process: 1: Using remap_pixel_values function in remap_pixel.py, the data set is converted to the corresponding pixel values according to the label 2: Call seggpt_inference.py function to generate mask 3: Use the remap_color_to_gray function in remap_pixel.py and turn the pixel value back.
tricks: 1: The original image generated by the segget code is the value of 255 pixels, and there is a map_to_color_class function in seggpt_engine.py, which can map 255 pixels to label pixels, and the scheme can be adjusted to improve segmentation performance. 2: When running seggpt_inference, the compare.py code will be called to automatically select the graph with the highest similarity to the target image as prompt
Everything in an image with a prompt.
python seggpt_inference.py \
--input_image examples/hmbb_2.jpg \
--prompt_image examples/hmbb_1.jpg \
--prompt_target examples/hmbb_1_target.png \
--output_dir ./
Everything in an image with multiple prompts.
python seggpt_inference.py \
--input_image examples/hmbb_3.jpg \
--prompt_image examples/hmbb_1.jpg examples/hmbb_2.jpg \
--prompt_target examples/hmbb_1_target.png examples/hmbb_2_target.png \
--output_dir ./
Everything in a video using a prompt image.
python seggpt_inference.py \
--input_video examples/video_1.mp4 \
--prompt_image examples/video_1.jpg \
--prompt_target examples/video_1_target.png \
--output_dir ./
Everything in a video using the first frame as the prompt.
python seggpt_inference.py \
--input_video examples/video_1.mp4 \
--prompt_target examples/video_1_target.png \
--output_dir ./
Processing a long video with prompts from both a target image and the predictions of the previous NUM_FRAMES frames.
NUM_FRAMES=4
python seggpt_inference.py \
--input_video examples/video_3.mp4 \
--prompt_target examples/video_3_target.png \
--num_frames $NUM_FRAMES \
--output_dir ./