A Multimodal Social Signal Processing Approach to Team Interactions

Social signal processing develops automated approaches to detect, analyze, and synthesize social signals in human-human as well as human-machine interactions by means of machine learning and sensor data processing. Most works analyze individual or dyadic behavior, while the analysis of group or team interactions remains limited. We present a case study of an interdisciplinary work process for social signal processing that can develop automatized measures of complex team interaction dynamics, using team task and social cohesion as an example. In a field sample of 25 real project team meetings, we obtained sensor data from cameras, microphones, and a smart ID badge measuring acceleration. We demonstrate how fine-grained behavioral expressions of task and social cohesion in team meetings can be extracted and processed from sensor data by capturing dyadic coordination patterns that are then aggregated to the team level. The extracted patterns act as proxies for behavioral synchrony and mimicry of speech and body behavior which map onto verbal expressions of task and social cohesion in the observed team meetings. We reflect on opportunities for future interdisciplinary or collaboration that can move beyond a simple producer-consumer model.

Automated summary

Social signal processing is an automated approach that detects, analyzes, and synthesizes social signals in human-human and human-machine interactions. Most research focuses on individual or dyadic behavior, with limited analysis of group or team interactions. This case study demonstrates the development of automated measures for complex team interaction dynamics using team task and social cohesion as examples. Sensor data from cameras, microphones, and a smart ID badge were used to extract fine-grained behavioral expressions of task and social cohesion, which were then aggregated to the team level. These extracted patterns act as proxies for behavioral synchrony and mimicry, mapping onto verbal expressions of task and social cohesion in team meetings.