Discourse Structure of Text-Graphics Documents

In order to analyze, generate and use documents containing both text and graphics, it is important to have a theory of their structure. We argue that it is possible to develop a semantics for graphics, as well as text, and generate an integrated representation of text/graphics discourse, building on previous theories of text discourse. A major component of our theory is an integrated natural language / visual language lexicon that allows people to understand bimodal discourse. Another major component of text/graphics discourse is the role of the reader. That is, the reader constantly exercises choices to shift his or her attention between the text and graphics while reading/viewing. We demonstrate some computational approaches to the automation of building discourse structure at the level of syntax. This is followed by the construction of semantics via logical forms. The result of this work is a Text-Graphics Discourse Theory, TGDT.