Retrieval From and Understanding of Large-Scale Multi-modal Medical Datasets: A Review

Content-based multimedia retrieval (CBMR) has been an active research domain since the mid 1990s. In medicine visual retrieval started later and has mostly remained a research instrument and less a clinical tool. The limited size of data sets due to privacy constraints is often mentioned as reason for these limitations. Nevertheless, much work has been done in CBMR, including the availability of increasingly large data sets and scientific challenges. Annotated data sets and clinical data for images have now become available and can be combined for multimodal retrieval. Much has been learned on user behavior and application scenarios. This text is motivated by the advances in medical image analysis and the availability of public large data sets that often include clinical data. It is a systematic review of recent work (concentrating on the period 2011-2017) on multimodal CBMR and image understanding in the medical domain, where image understanding includes techniques such as detection, localization, and classification for leveraging visual content. With the objective of summarizing the current state of research for multimedia researchers outside the medical field, the text provides ways to get data sets and identifies current limitations and promising research directions. The text highlights advances in the past six years and a trend to use larger scale training data and deep learning approaches that can replace/complement handcrafted features. Using images alone will likely only work in limited domains but combining multiple sources of data for multi-modal retrieval has the biggest chances of success, particularly for clinical impact.