Addressing uncertainty in multi-modal fusion for improved object detection in dynamic environment

A major challenge in real world object detection for Video Surveillance (VS) is the dynamic nature of environmental conditions with respect to illumination, visibility, weather change, etc. With increasing availability of cameras and other sensor modalities beyond visible spectrum at lower cost, multi-modal VS systems involving visible, thermal infrared cameras, etc., are seen as a promising solution for reliable and robust operation in unfavorable environmental conditions like at night or dark situation. However, there are several research challenges to actually utilize the combined benefits of using different modalities. This paper addresses the uncertainty problem in the fusion of the information provided by complementary modalities like visible spectrum and thermal infrared video in a generic framework using evidence theory. A belief model is developed to determine the validity of a foreground region detected by each source for tracking. Fuzzy logic modeling is done for generating belief mass function from the sensor information by using two measurement features. A novel algorithm is developed for dynamic assessment of individual sensor reliability within the belief model. A generic approach to re-assign the conflicting mass is adopted for belief fusion to be done in a weighted manner depending on the context. Finally, the confirmed objects are tracked and the sensor measurements of their position, size, etc., are fused using a weighted Kalman filter fusion method. The approach is evaluated by using a pair of visible and thermal infrared sensors in real world challenging scenarios. Published by Elsevier B.V.