From Naive Physics to Connotation: Learning about the World from How People Use Language
Margaret Jacks Hall, Greenberg Room (460-126)
Through a particular choice of a predicate, e.g., “X violated Y”, a writer can convey a range of implied sentiments and presupposed facts, including the intent of the author (X is an “villain” and Y is a “victim”), the mental state of the event participants (Y is probably “unhappy” toward X), and the presupposed value judgments (Y is something “valuable”). I will introduce Connotation Frames as a representation framework to organize these connotative implications associated with a predicate. As a case study, I will discuss the agency and power differences in the way characters of different genders are portrayed in modern films.
Next, I will present how the frame-centric representation of connotation can be extended to relative physical knowledge about actions and objects. Learning unspoken commonsense knowledge from language is nontrivial, as people rarely state the obvious, e.g., ``my house is bigger than me.'' However, this trivial everyday knowledge does influence the way people talk about the world, which provides indirect clues to reason about the world. For example, if ``Shanice entered her house’’, it must be that her house is bigger than her. Using this insight, I will show how we can reverse-engineer limited aspects of physical knowledge, such as size, weight, rigidness, strength, and speed, by inferring relative physical relations between objects while also reasoning about the physical implications of actions.
Last, I will discuss neural network approaches that complement the frame-centric approaches above. In particular, I will present Neural Checklist Models that can learn to compose a new recipe as if the network understands how the kitchen works. I will conclude the talk by discussing the challenges in current models and formalisms, pointing to avenues for future research.