Panoptic Segmentation

A segmentation task that labels every pixel with a category while also telling individual object instances apart.

Panoptic segmentation was proposed by Alexander Kirillov, Kaiming He, and colleagues in 2018 (published at CVPR 2019). It merges two separate tasks: semantic segmentation, which labels every pixel with a category but does not distinguish separate instances of the same category, and instance segmentation, which distinguishes individual object instances but ignores amorphous background categories such as sky or ground (which the paper calls “stuff”). Panoptic segmentation requires every pixel to get a category label, with an additional instance number for anything belonging to a countable object category (“things”), producing one complete, non-overlapping parse of the scene, evaluated with a newly proposed Panoptic Quality (PQ) metric. For robots, this simultaneously answers “where is the floor, where is the countertop” and “which cup is this, the first or the second,” and it is commonly used for scene understanding and semantic mapping.

ExampleIn a kitchen image, the floor, walls, and countertop are each labeled as one solid region by category, while three bowls on the counter are separately labeled bowl 1, bowl 2, and bowl 3.

Related

Semantic Segmentation · Instance Segmentation · Mask · Scene Understanding · Semantic Map · Mask R-CNN

Sources

Panoptic Segmentation (arXiv 1801.00868)