arxiv preprint – Segment Anything with Multiple Modalities

In this episode, we discuss Segment Anything with Multiple Modalities by Aoran Xiao, Weihao Xuan, Heli Qi, Yun Xing, Naoto Yokoya, Shijian Lu. The paper introduces MM-SAM, an extension of the Segment Anything Model (SAM) tailored for multi-modal data from various sensor suites, such as LiDAR plus RGB and thermal plus RGB. MM-SAM employs unsupervised cross-modal transfer and weakly-supervised multi-modal fusion to adapt efficiently to different sensor modalities. Extensive experiments validate that MM-SAM significantly outperforms the original SAM in robustness and segmentation accuracy across various sensors and modalities.