Multimodal Representation Learning and Its Application to Human Behavior Analysis

Book excerpt: "This thesis aims to learn the joint representation of text, acoustic and visual modalities to understand spoken language in face-to-face communications. Being able to mix and align those modalities appropriately helps humans to display sentiment, humor, and credible argument in daily conversations. The creative usage of these behaviors removes barriers in communication, grabs the attention of the audience, and even helps to build trust. Building algorithms for understanding these behavioral tasks is a difficult problem in AI. These tasks not only demand machine learning algorithms that create efficient fusion across modalities, incorporate world knowledge, and reasoning, but also require large complete datasets. To address these limitations, we design behavioral datasets and a series of multimodal machine learning algorithms. First, we present some key insights about credibility by analyzing the verbal and non-verbal features. The pre-trained facial expressions from baseline questions help to classify the relevant section as truth vs. bluff (70% accuracy ” 52% human accuracy). Analyzing interrogation answers in the context of facial expressions reveals interesting linguistic patterns of deceivers (e.g. less cognitively-inclined words, shorter answers). These patterns are absent when we analyze the language modality alone. Next, we develop UR-FUNNY - the first video dataset (16k instances, 19 hours) of humor detection. It is extracted from TedTalk videos using the laughter marker of the audience. We study the multimodal structure of humor and the importance of having a context story for building up the punchline. We design neural networks to detect multimodal humor and show the effectiveness of humor-centric features like ambiguity and superiority based on linguistic theories. To investigate the properties of high-quality arguments, we propose a set of features such as clarity, content variation, body movements, and pauses. These features are interpretable and can distinguish (p