Interpreting Multimodal Communicative Action

Matthew Stone, Rutgers


I give a short tutorial overview of the nonverbal actions that speakers use to enrich the meanings of their contributions to spoken face-to-face conversation. We will see the ways speakers not only refer to real space but construct virtual and abstract spaces to locate events; the ways they indicate geometry, represent objects and imitate actions; and the consistent patterns that link successive gestures into extended depictions. Time permitting, we will also explore the analogous ways speakers can recruit physical props and practical activity in the service of communication. Finally, we will look at some of the formal devices that have been used to model the compositional connections between language and gesture, including generalized conjunction, function-argument dependencies and the reconstruction of implicit inferential relationships. A key theme throughout the presentation will be the role of discourse coherence in organizing content across modalities into an integrated whole.