Utilize Automatic Segmentation with SAM (segment-anything)

Autodetect elements in images and return images as possible greenscreen footage, the element-detected mask in full size of the fed image, a cropped version of the image where the element was detected, also with a separated mask, and a bbox list to later use the detected information in other workflow processes.

TODO: read this: facebookresearch/segment-anything#185

Arguments:

Every item marked with ( ) has been implemented, while those marked with (-) have been removed after testing. (discussion needed) indicates that we should discuss the relevance of these items.

Automatic Segmentations possible options:

( ) model (Sam): The SAM model to use for mask prediction.

( ) points_per_side (int or None): The number of points to be sampled along one side of the image. The total number of points is points_per_side**2. If None, 'point_grids' must provide explicit point sampling.

(-) points_per_batch (int): Sets the number of points run simultaneously by the model. Higher numbers may be faster but use more GPU memory.

(-) pred_iou_thresh (float): A filtering threshold in [0,1], using the model's predicted mask quality.

( ) stability_score_thresh (float): A filtering threshold in [0,1], using the stability of the mask under changes to the cutoff used to binarize the model's mask predictions.

(-) stability_score_offset (float): The amount to shift the cutoff when calculated the stability score.

( ) box_nms_thresh (float): The box IoU cutoff used by non-maximal suppression to filter duplicate masks.

(discussion needed)( ) crop_n_layers (int): If >0, mask prediction will be run again on crops of the image. Sets the number of layers to run, where each layer has 2**i_layer number of image crops.  

(discussion needed)( ) crop_nms_thresh (float): The box IoU cutoff used by non-maximal suppression to filter duplicate masks between different crops. 

(discussion needed)( ) crop_overlap_ratio (float): Sets the degree to which crops overlap. In the first crop layer, crops will overlap by this fraction of the image length. Later layers with more crops scale down this overlap.

(discussion needed)(-) crop_n_points_downscale_factor (int): The number of points-per-side sampled in layer n is scaled down by crop_n_points_downscale_factor**n.       

(discussion needed)(-) point_grids (list(np.ndarray) or None): A list over explicit grids  of points used for sampling, normalized to [0,1]. The nth grid in the  list is used in the nth crop layer. Exclusive with points_per_side.

( ) min_mask_region_area (int): If >0, postprocessing will be applied to remove disconnected regions and holes in masks with area smaller than min_mask_region_area. Requires opencv.

box_threshold

marks the threashold at which confidence the image features are filtered. lowering the threashold will result in more image feature. but be aware!! the lower the number the more vram will be consumed

two_pass

will run two passen on HQ models for hopefully better mask results. This has no effect on non-HQ models for now

multimask

When activated, the node will give you multiple mask and images stacked on the batch_size of the tensor. To make a selection later on please use the BatchSelector-Node until a selector inside this node is missing.

clean_mask_holes clean_mask_island

clean_mask_holes and clean_mask_island can take on very large values, as this seems to reflect the pixel density of the mask. 64 as default value is mainly used to remove small parts of the mask. A value of 0 will therefore not make any corrections to the mask.