In human interactions, social communication situations build the framework around emotional interchange and the establishment of social coherence. Understanding of various communication channels and its integration seems central for reciprocal understanding and empathy. The following studies investigated the differential role of each of these communication channels in experimentally designed social interactions. This was first accomplished by means of behavioral and psychophysiological parameters, later with functional magnetic resonance imaging. Naturalistic dynamic stimulus material (validated short video clips) was applied in order to disentangle a combinatory influence and channel-specific correlates. Apart from a consistently emotional and a consistently neutral control condition, three partially emotional conditions transported emotionality via two channels only, while the third was held neutral (speech content, facial expressions, prosody). One last condition employed incomprehensive emotional content (foreign language). Behavioral measures included valence and intensity ratings of the presented and the own emotion. On a behavioral level, consistent emotional information appeared central for empathy - recognizing, understanding, and sharing the emotional state of the other. As soon as one channel was held neutral, empathic reactions as well as their intensity levels decreased. This effect was supported by a decrease of several galvanic skin response parameters. More specifically, neutral facial expressions mainly impaired correct ratings of the target emotion of the other while neutral speech content had a negative effect on one’s own affective response.

On a neural level, distinct neural activation patterns were associated with complete and partly emotional clips. Complete emotional stimuli activated thalamus and precuneus regions associated with multimodal integration and facilitative effects for empathy. Incomplete emotional clips elicited activation of the temporo-parietal junction, prefrontal cortex and middle temporal gyrus areas more related to conscious cognitive perspective-taking.

These studies are furthermore the basis for investigating diagnose-specific correlates of impaired emotion processing in psychiatric disorders. Behavioral and fMRI results of patients with depression and patients with schizophrenia suggest an interaction of diagnosis and communication condition in limbic regions but also on a level of evaluating emotionality in its context.