Multi-device coverage testing of mobile applications

This paper evaluates the effectiveness of coverage approaches for selecting mobile devices (i.e., smartphones and tablets) to test mobile software applications. Due to the large number of such devices on the market and the variations in their characteristics, it is hard to guarantee that an application will work as intended on all devices. For this reason, multi-device testing is necessary. The goal of this research was to determine how many devices must be tested and which methods for device selection are best for revealing device-specific faults. We experimentally investigated a simple coverage of all values of each device's features separately and the each-choice coverage (i.e., the coverage of all device characteristics at the same time). To collect the experimental data, 15 Android applications were tested on 30 mobile devices and 24 device-specific faults were detected. Our research shows that a random selection of 13 devices achieved 100% effectiveness. However, coverage of device characteristics in the selection process yielded an acceptable 90% level of effectiveness with a set of only five devices. The most successful approaches were the coverage of different types of Android operating systems and the each-choice coverage. Our results include recommendations for increasing the effectiveness while decreasing the costs of mobile testing.