Adventures in Flatland: Perceiving Social Interactions Under Physical DynamicsTianmin Shu1 (tshu@mit.edu) Marta Kryven1 (mkryven@mit.edu)

Tomer D. Ullman2 (tullman@fas.harvard.edu) Joshua B. Tenenbaum1 (jbt@mit.edu)1Department of Brain and Cognitive Sciences, MIT

2Department of Psychology, Harvard University

AbstractPeople make fast, spontaneous, and consistent judgementsof social situations, even in complex physical contexts withmultiple-body dynamics (e.g. pushing, lifting, carrying, etc.).What mental computations make such judgments possible? Dopeople rely on low-level perceptual cues, or on abstract con-cepts of agency, action, and force? We describe a new exper-imental paradigm, Flatland, for studying social inference inphysical environments, using automatically generated interac-tive scenarios. We show that human interpretations of events inFlatland can be explained by a computational model that com-bines inverse hierarchical planning with a physical simulationengine to reason about objects and agents. This model out-performs cue-based alternatives based on hand-coded (multi-nomial logistic regression) and learned (LSTM) features. Ourresults suggest that humans could use a combination of intu-itive physics and hierarchical planning to interpret complex in-teractive scenarios encountered in daily life.Keywords: social perception; theory of mind; intuitivephysics; Bayesian inverse planning; hierarchical planning

IntroductionWe can easily read the intentions of others in their phys-ical actions. As Oliver Wendell Holmes famously put it,“Even a dog knows the difference between being stumbledover and being kicked.” This ease belies the understandingof physics and psychology necessary to tell the difference.More broadly, when seeing others engage in social-physicalinteractions (e.g. watching a soccer game) we make intuitive,fast and consistent inferences about their actions from briefobservations, and without evaluative feedback. What mentalmechanisms support such multi-modal and varied inference?

On one hand, the speed of social attribution suggests that itmay be driven by low-level perceptual cues, such as facial ap-pearance (Todorov et al., 2005; Ambady & Rosenthal, 1993),or motion (van Buren et al., 2017; Shu et al., 2018). Yet, itsrichness suggests a reliance on theory of mind (ToM), or in-terpreting actions of others by joint inference over incentives,abilities and goals (Gelman et al., 1995; Hamlin et al., 2013).Reasoning about physical events has likewise been studied interms of perceptual cues (e.g.timing (Michotte, 1963) and ve-locity (Gilden & Proffitt, 1989)), and as driven by mentallysimulating the physical world, or intuitive physics (Forbus,2019; Battaglia et al., 2013).

Physical and social inferences are traditionally studied byseparate empirical paradigms, since they seem to rely on dif-ferent systems of knowledge (Carey, 2000), and engage dif-ferent neuro-cognitive domains (Fischer et al., 2016; Sliwa &

Entities(Agents or Objects)

WallsLandmarks

CA B

Figure 1: (A) Examples of real-life social interactions in physi-cal environments: a goalkeeper blocking a shot from an opponent;two persons carrying a couch. (B) The classic Heider-Simmel an-imation abstracts such real-life interactions in animated displays ofsimple geometric shapes. (C) Flatland captures social scenarios,and their physical dynamics, in a controlled, procedurally generatedenvironment. In this example subjects see three interacting circles,which represent agents and objects of different mass and size. Col-ored squares indicate landmarks (possible goal locations), and wallsare shown by black lines. Agents and objects cannot move throughwalls. An agent’s goal may be, for example, to move an object to aspecific landmark. Agents may have relationships with other agents,expressed as goals of helping or hindering the other.

Freiwald, 2017). However, in daily life both types of attribu-tions interact, with the interpretations of one domain relyingon the understanding of the other. For example, in the clas-sic (Heider & Simmel, 1944) experiment, the subjects’ nar-ratives illustrate both an understanding of the physical world,and of social relations and goals, such as: “The triangle isfrustrated that he cannot harm the circle,”. Any internal men-tal representation capable of accurately interpreting the natureof such multi-modal social interactions must integrate physi-cal and social representations. This integration is necessary todifferentiate between animate and physical events, and to seeagents simultaneously as objects, targets of physical actions,and as agents, enacting their goals.

In this work we study the mechanisms of social inferencesin dynamical physical scenes by introducing a new experi-mental paradigm, Flatland, inspired by Heider-Simmel ani-mations. Several computational and quantitative studies haveexamined social interactions and attributions in grid-worldenvironments (e.g. Baker et al., 2017; Kryven et al., 2016;Jara-Ettinger et al., 2015; Rabinowitz et al., 2018). Flatlandextends these studies to a continuous physical domain, closerto the original Heider-Simmel study, but with more controlover procedural stimulus generation and ground truth. Ourmethodology also builds on Shu et al. (2019), which used

deep reinforcement learning to generate simple social inter-actions in a 2D physics engine.

Flatland allows for a variety of goals, agent-to-agent re-lations, and physical properties of agents and objects (seeFigure 1A). We interpret human attributions of goals, rela-tions, and physical properties in this domain by a computa-tional model that combines a hierarchical planner (based onKaelbling & Lozano-Perez, 2011) with a physical simulationengine1. Many studies explored hierarchical planning in hu-man decision-making (e.g. Balaguer et al., 2016; Huys et al.,2015), and recently in social inference (Yildirim et al., 2019).We show that a combined hierarchical planning and physi-cal engine model outperforms cue-based alternatives, such asmultinomial logistic regression and LSTM, in predicting hu-man interpretations of ambiguous events. Our results suggestthe role of complex abstract physical and mentalistic conceptsin social inference.

Computational ModelingFlatlandFlatland is a 2D simulated physical environment with mul-tiple interacting agents and objects. Agents can have twotypes of goals: (1) a personal goal gi ∈ G , and (2) a socialgoal of helping or hindering another agent. Agents are sub-ject to physical forces, and can exert self-propelled forces tomove. Agents can also attach objects to their bodies and re-lease them later. In the current study, agents have accurateand explicit knowledge of the other agents’ goals. However,the Flatland environment can be extended to scenarios withincomplete information.

Formally, agents are represented by a decentralizedMulti-agent Markov Decision Process (Dec-MDP), i.e.,〈S ,A ,Ri,Ti〉, ∀i ∈ N, where N is the number of agents, Sand A are the state set and the action set shared by all agents,Ri : S×A→R is the agent’s reward function, Ti : S×A→ Sis the agents’ state transition probability. The amount of forcean agent can exert, fi, defines the agent’s strength, and shapesits dynamics in the physical environment. An agent’s statetransition probability Ti can be written as P(s′|s,a, fi,θi),where θi denotes the physical properties of the agent, otherthan its strength, (e.g. mass and shape). Assuming that allbodies have the same density, θi is easily observable based onvisual appearance, but this assumption can be relaxed.

An agent’s reward is jointly determined by: (1) the rewardof its own goal, (2) the reward of other agents’ goals, (3) itsrelationships with other agents, and (4) the cost of actions.Formally, we define an agent’s reward as:

Ri(s,a) = R(s,gi)+∑j 6=i

αi jR(s,g j)−C(a), (1)

where C(a) is the cost function; αi j indicates agent i’s re-lationship with agent j, including how much agent i caresabout agent j’s goal. For a friendly relationship, αi j > 0; for

1https://github.com/pybox2d/pybox2d

an adversarial one, αi j < 0; and αi j = 0 if the relationship isneutral.

Given this physical and social setup, we now consider howan agent could plan to achieve its goals in this environment.Interpreting an agent’s actions would then require the inver-sion of this planning process. A classic MDP-based approachwould prove exceedingly costly in the continuous physics ofFlatland, coupled with the agents’ composite rewards. Inthis work we deal with this complexity by incorporating ahierarchical planner inspired by the task and motion planning(TAMP) framework (Kaelbling & Lozano-Perez, 2011).

Hierarchical PlanningFigure 2 shows how the hierarchical planner (HP) works.Given an agent i’s goal, strength, relationships with otheragents, and the other agents’ goals, HP generates the bestaction to take at any given state. An agent can pursue itsown goal, or the goal of another agent. HP searches for plansΠi j = {aτ}t+T−1

τ=t with a finite horizon of T steps for all goalsg j, ∀ j ∈ N. Any g j such that j 6= i is a goal of anotheragent j. Each plan is simulated using the physics engine,and the agent’s cumulative reward following that plan is givenby a composite value function, V (Πi j) = ∑

T−1τ=0 Ri(st+τ,at+τ),

which incorporates the reward of its personal goal and theweighted rewards of other agents’ personal goals.

The plan with the highest cumulative reward is selected asthe final plan generated by the HP. To better adapt to otheragents’ plans, in the current implementation HP returns onlythe first action of the selected plan, and re-plans by searchingfor new plans at every step. So, the plans can be frequentlyadjusted according to the latest state.

To generate an optimal plan for each possible goal, theHP adopts a two-level architecture. First, a Symbolic Plan-ner (SP) prepares a sequence of sub-goals for a given goal.This entails generating symbolic states from physical states2,and creating a sequence of sub-goals that reaches the finalgoal. For example, a sub-goal could entail grabbing an ob-ject, blocking a door, or moving to a specific location. In thepresent study, SP used A∗ search to find the shortest path tothe goal in the space of symbolic states. Second, a MotionPlanner (MP) generates a sequence of actions3 that achieveseach sub-goals using Monte-Carlo Tree Search (MCTS) to-gether with a physics engine. Note that alternative implemen-tations of SP and MP may be suitable in different domains.

Formally, let πo(oti|st , fi,{g j} j∈N ,{αi j} j 6=i) ∝ eV (Πi j) be

the plan selection policy, where oti ∈ {gk}k∈N is the selected

goal, and let π(ati|st , fi,ot

i) be the policy for the selected goal,computed by MCTS. Then, the agent’s final policy is:

ati ∼ π(a

ti|st , fi,{g j} j∈N ,{αi j} j 6=i), (2)

2The symbolic states are predefined predicates: On(object, land-mark), Reachable(agent, object), Attached(agent, object) and theirnegation).

3Here the actions are forces that can be applied in eight possibledirections, grabbing or releasing an object, stopping, and no force.

Reachable( )

Reachable( )Reachable( )

On( )

Symbolic State

Basic Goalgi

Social GoalgjHelping

Symbolic Planner (SP)

Symbolic Planner (SP)

Get to

Get self to

Motion Planner (MP)

NotTouched( )

NotTouched( ) Goto

Grab

Push to

Goto ⇧ii

⇧ij

st

ati

Motion Planner (MP)

Left force

Hierarchical Planner(HP)

fi↵ij

Plan Selectionargmax⇧2{⇧ii,⇧ij}V (⇧)

fi, {gi}i2N , {↵ij}j 6=i

st

Help

Agent ’s Goal

⇧ij

Help Help Get self to Get self to Get self to

…

Time

ati

Agent ’s Planner

A

B

Help Help

#15 #20 #25 #30 #35 #40 #45

Figure 2: An illustration of an agent’s Hierarchical Planner (HP).(A) Using Symbolic Planner (SP) and Motion Planner (MP), HP optimizesa value function that combines a personal and a social goal (helping or hindering another agent). In this example the agent has a helpingsocial goal. (B) Each agent replans its actions every a few steps in response to a changing state of other agents and objects. An agent mayswitch goals, when the expected reward of the new goal outweighs that of the current goal. For example, the green agent first helps the redagent to bring the blue object to the pink landmark; once it has been reached, the green agent switches to pursuing its personal goal.

where

π(ati|st , fi,{g j} j∈N ,{αi j} j 6=i) =

∑ot

i∈{gk}k∈N

πa(at

i|st , fi,oti)π

o(oti|st , fi,{g j} j∈N ,{αi j} j 6=i).

(3)

Attributing Goals, Relationships, and StrengthsBy running the HP forward we can generate arbitrary inter-active scenarios in the Flatland environment. Such scenarioscould involve a number of objects of different sizes, shapes,and appearances, and a number of agents with personal andsocial goals. People viewing animations of this kind tend tointerpret them in terms of a narrative about the agents’ rela-tionships, incentives, and abilities (Heider & Simmel, 1944).Such interpretations may arise either from identifying spe-cific cues, or from applying a more structured, theory-likeunderstanding of objects and agents. We formalize these twoviews of human judgement using cue-based models as wellas a theory-based generative model that relies on Bayesianinverse planning enabled by the HP and physics simulation.

Generative Social and Physical Inference (GSPI) GSPIconducts Bayesian inference of latent variables (i.e., agents’goals, relationships, strengths) to describe an observed so-cial interaction through a generative model consisting of ourheirarchical planner and a physics engine. For each hypothe-sis of the latent variables, GSPI i) samples optimal plans w.r.t.Eq. 2, and ii) simulates entities’ trajectories in the physics en-

gine based on the hypothesis as well as the sampled plans.GSPI then defines the likelihood of the hypothesis by howmuch the simulated trajectories deviate from the observed tra-jectories. Combined with the priors of the latent variables,GSPI computes the posterior of the hypothesis using Bayes’srule:

P(gi,g j, fi, f j,αi j,α ji|s1:Ti ,s1:T

j ) ∝

P(s1:Ti ,s1:T

j |gi,g j, fi, f j,αi j,α ji) ·P(gi,g j, fi, f j,αi j,α ji).(4)

Given this principle, we show how GSPI can be used forinferring agents’ goal selection, relationships, and comparingtheir relative strengths in details as fellows.

First we can calculate the posterior probability of theagents’ goals, given observations and given the agents’ socialand physical properties:

Pi j(oti,o

tj) = P(ot

i,otj|st ,st+1,gi,g j, fi, f j,αi j,α ji) ∝

∑at

i ,atj

P(st+1|st ,ati,a

tj)π

a(ati|st , fi,ot

i)πo(ot

i|st , fi,{gi,g j},αi j)

·πa(atj|st , f j,ot

j)πo(ot

j|st , f j,{gi,g j},α ji)P(ati)P(a

tj)P(o

ti)P(o

tj),

(5)where P(st+1|st ,at

i,atj) = e−β||st+1−st+1||, with st+1 being the

predicted next state after taking ati based on physics simu-

lation. Here β controls the agent’s proximity to the optimalplans generated by the HP. A large β means that the agent willfollow the optimal plan; as β becomes smaller, it is increas-ingly likely to deviate from the optimal plan. P(o) and P(a)

are uniform priors.The probability that both agents are pursuing their personal

goals in the last T steps is given by:4

P(oi = gi,o j = g j|s1:T ) =

∑fi, f j ,αi j ,α ji

∏t

Pi j(gi,g j)P( fi)P( f j)P(αi j)P(α ji). (6)

The probability that agent i is pursing a social goal (helpingor hindering another agent) is given by:

P(oi = g j,o j = g j|s1:T ) =

∑gi, fi, f j ,αi j ,α ji

∏t

Pi j(g j,g j)P(gi)P( fi)P( f j)P(αi)P(α j).

(7)Here we discretize f and α for making the computationstractable. We assume uniform priors for goals and strengths.For α, we assume that P(α = 0) = P(α > 0) = P(α < 0) =1/3, so that there is no bias toward any type of relationship.

To infer the relationship between two agents, we first de-rive the posterior probabilities of αi j and α ji.

P(αi j,α ji|s1:T ) =

∏t

∑gi,g j , fi, f j

∑ot

i ,otj

Pi j(oti,o

tj)P(gi)P(g j)P( fi)P( f j)P(αi j)P(α ji).

(8)Based on Eq. 8, we can derive the posterior probability ofspecific relationships. E.g.,

P(Adversarial|s1:T ) = ∑αi j≤0,α ji<0

∑αi j<0,α ji≤0

P(αi j,α ji|s1:T ).

(9)Finally, the expected strength difference between two

agents is given by:

E[ fi− f j|s1:T ] = ∑fi

∑f j

( fi− f j)P( fi, f j|s1:T ), (10)

where

P( fi, f j|s1:T ) =

∏t

∑gi,g j ,αi j ,α ji

∑ot

i ,otj

Pi j(oti,o

tj)P(gi)P(g j)P( fi)P( f j)P(αi j)P(α ji).

(11)

Cue-based models We compared the GSPI model withthree cue-based alternatives: Cue-based-1: Multinomial lo-gistic regression based on feature statistics of the wholevideo; Cue-based-2: Multinomial logistic regression basedon concatenated feature statistics of chunks of the video; Cue-based-3: Long short-term memory (LSTM). Each cue-basedmodel was trained on 400 stimuli, not used in the experi-ment. Following (Ullman et al., 2010), we used the followingcues for each agent: (1) coordinates, (2) velocity, (3) acceler-ation, (4) relative velocity w.r.t. other entities and landmarks,

4Note that the equations here are constrained to two-agent sce-narios for simplicity and readability, but they can be easily extendedto more general cases.

(5) distance to other entities and landmarks, (6) whether theagent is touching another entity. The LSTM model acceptsthe sequence of these cues as input and learns motion fea-tures by itself. For logistic regression models, we encode thecue sequences as statistics (mean, minimum, maximum, stan-dard deviation) to obtain motion features. Cue-based-1 usedthe statistics over the whole video as input, and Cue-based-2 concatenated statistics of short chunks of a video as input.To train these models, we generated 400 training videos byrandomly sampling agents’ goals, relations, and strengths aswell as the environment layout, sizes and initial positions ofentities. Note that these 400 training videos were not shownin the human experiment.

Methods

Flatland is a simple but rich environment, capable of gen-erating many visually distinct scenes from a relatively smallnumber of underlying physical and social variables. Flatlandscenarios allow us to quantitatively test alternative accountsof human physical and social reasoning. We have describedtwo such basic alternatives – i) a theory-like inference that re-quires forward planning models and physical simulation forthe physical and psychology of agents in Flatland, and ii)a cue-based alternative that relies on many separate visualcues to map between observed social interactions and agents’goals, relationships and strengths. We next describe an empir-ical study of human inferences in Flatland, in order to assessthe fit of these two different models.

Procedure

The experiment5 was presented in a web browser using psi-Turk (Gureckis et al., 2016). The instructions explained howFlatland works. After reading instructions, subjects com-pleted 3 comprehension quizzes for judging goals, relation-ship, and strengths respectively. Subjects who failed to accu-rately respond to all quizzes were asked to read the instruc-tions again until they correctly responded to all quizzes. Next,subjects responded to two practice stimuli, similar to the stim-uli presented in the main experiment, the responses to whichwere not included in the analysis. After completing the prac-tice, subjects saw 6 stimuli and reported: (1) the goals of eachagent, (2) the relationship between agents, and (3) the relativestrength of each agent. The responses were given by selectingthe appropriate items from a multiple-choice list.

Subjects

120 subjects (mean age = 38.4; 45 female) were recruitedon Amazon Mechanical Turk and paid $1.60 for 12 minutes.Subjects gave informed consent. The study was approved bythe MIT Institutional Review Board.

5The exact experimental setup (screenshots) canbe found at https://osf.io/25nsr/?view only=ce34eb376d0c4f3dbf3a095bd7dafb60

0.20.40.60.81.01.21.41.6

Human

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Mod

el

Entropy of Goal Judgment

0.0 0.5

Human

0.0

0.2

0.4

0.6

0.8

1.0

Mod

el

r = 0.84

Goal

0.0 0.5 1.0

Human

0.00

0.25

0.50

0.75

1.00

Mod

elr = 0.90

Relationship

�0.5 0.0 0.5

Human

�0.5

0.0

0.5

Mod

el

r = 0.75

Strength

E F G

A B C D

Figure 3: Comparing the GSPI model’s inferences to human re-sponses. (A) Probabilities of each of the possible agent goals givenby the model, plotted against the averaged human responses. (B)Probabilities of each type of relationship (Neutral, Friendly, Ad-versarial) given by the model plotted against the averaged humanresponses. (C) The model’s estimate of the strength difference be-tween the two agents against the averaged human response. (D) En-tropy (in bits) of human goal judgements plotted against the entropyof the model’s goal judgments. Each data point represents one stim-ulus, and the error bars indicate the bootstrapped 95% confidenceintervals. Notably, stimuli that exhibited higher entropy (more am-biguity) in humans were also harder for the model. (E-G) Humanand model’s inferred agents’ relationships, given ground-truth. Er-ror bars show 95% confidence intervals. Both humans and modelcorrectly identified the relationships in most of the stimuli, and hada higher degree of confusion when the ground-truth relationship wasneutral.

StimuliStimuli were 30 Flatland animations6, which always con-tained three interacting bodies (shown as circles) and fourlandmarks (squares placed at the four corners of the screen,as shown in Figure 1C). Two of the interacting bodies werealways agents and one was an object. Subjects were informedwhich of the circles were agents, and which was the object.Objects varied in mass, and agents varied in their relativestrength. Each agent always had one personal goal, whichcould be one of the following: (1) moving itself to a spe-cific landmark, (2) approaching another entity, or (3) movingthe object to a specific landmark. In addition to their per-sonal goals, some of the agents also had social goals of either(1) helping the other agent achieve its goal, or (2) hinderingthe other agent. All animations were 10 seconds long with aframerate of 30.

We generated a large number of stimuli using our hierar-chical planner with randomized parameter settings, including(1) entities’ sizes and initial positions, (2) the environmentlayout, (3) agents’ strengths, goals, and their relationship. Wemanually selected 30 representative examples.7

ResultsThe comparison of the cue-based models and the GSPI modelis summarized in Table 1. Importantly, human responses

6Stimuli can be viewed at https://www.youtube.com/playlist?list=PL0ygI9h8RqG yypVml0xMl8Lkcd5hbuxk

7We aimed to sample a variety of enacted scenarios, and pre-ferred animations that could be interpreted with ambiguity to elicita distribution of responses over possible interpretations.

GSPI Cue1 Cue2 Cue3 GTGoal .84 .06 .09 .09 .73Relation .90 .21 .32 .01 .77Strength .75 .60 .63 .06 .71

Table 1: Correlations of average human responses with the modelsand with ground truth.

were closer to the predictions of the GSPI model than tothe ground truth. The bootstrapped inter-subject correlationswere r = .83 (SD= .02) for goals, r = .88 (SD= .03) for rela-tionships, and r = .68 (SD = .09) for strengths, which showsthat humans made highly consistent inferences of goals andrelationships, but less consistent inferences of strengths.

We calculated goal judgements as the marginalized prob-ability of a goal being reported in a given stimulus. Humanresponses to the strength question were recorded as (-1, 0,1), corresponding to (“weaker”, “same”, and “stronger”). Wefound that all cue-based models performed well on the train-ing data, but poorly on the testing stimuli. Notably, Cue-based-1 and Cue-based-2 produced good estimates of theagents’ relative strength, suggesting that simple cues couldbe useful in judging physical properties. A detailed summarycomparing the GSPI model to human responses is given inFigure 3, showing a close match between the models’ and thehumans’ judgements. As shown in Figure 3D, human andmodel also agree on which stimuli were easy (low-entropy)and which were hard (high-entropy). The correlation betweenmodel and human entropy was r = .41 (p = .02).

Figure 4 shows four representative stimuli along with thecorresponding human and model inferences. The model notonly recognized the ground-truth with high confidence inmost cases, but also shared similar confusion with humansover goals and relations when the agents’ behaviors were hardto interpret (e.g. both humans and the model were all uncer-tain about the goals and the relationship in Figure 4E). No-tably, subjects sometimes failed to recognize that an agentalso had a personal goal in addition to its social goal (Fig-ure 4D). In contrast, in such cases the model generated highconfidence inferences over both goals.

DiscussionOur results show that human interpretations of complex so-cial and physical multi-agent interactions can be described bya hierarchical planner combined with a physical simulationengine. Notably, the proposed GSPI model matches humanpredictions on a number of important dimensions: (1) it canaccurately predict human responses, even in cases where sev-eral interpretations are plausible, (2) it makes mistakes simi-lar to human mistakes in cases when ground truth is unclear,and (3) unlike alternative cue-based models, our GSPI modelrequires few observations of behavior to reach an inference.In contrast, cue-based models not only require a large corpusof training data, but are also constrained to more simple inter-active scenarios (for example, inferring an agent’s strength),and produce less generalizable results.

https://youtu.be/hFzejE5cOlI

https://youtu.be/Htlhf2WJK3w

https://youtu.be/fylivfYKs-A

A

B

C

D https://youtu.be/hCDoNBxAkNg

Adversarial Neutral Friendly0.0

0.2

0.4

0.6

0.8

1.0Relationship

Hinder Obj2SE Self2Obj0.0

0.2

0.4

0.6

0.8

1.0Green’s Goal(s)

Obj2SE Hinder Self2Obj0.0

0.2

0.4

0.6

0.8

1.0Red’s Goal(s)

Help Self2SE Obj2NW0.0

0.2

0.4

0.6

0.8

1.0Green’s Goal(s)

Obj2NW Help Hinder0.0

0.2

0.4

0.6

0.8

1.0Red’s Goal(s)

Adversarial Neutral Friendly0.0

0.2

0.4

0.6

0.8

1.0Relationship

Obj2NW Hinder Help0.0

0.2

0.4

0.6

0.8

1.0Green’s Goal(s)

Self2SW Obj2SW Hinder0.0

0.2

0.4

0.6

0.8

1.0Red’s Goal(s)

Adversarial Neutral Friendly0.0

0.2

0.4

0.6

0.8

1.0Relationship

HumanModel

Self2SE Obj2SE Help0.0

0.2

0.4

0.6

0.8

1.0Green’s Goal(s)

Obj2SW Help Self2Obj0.0

0.2

0.4

0.6

0.8

1.0Red’s Goal(s)

Adversarial Neutral Friendly0.0

0.2

0.4

0.6

0.8

1.0Relationship

Self2 Obj2 Obj2Hinder Hinder Help

HinderObj2 Self2Obj Self2ObjHinder Obj2

Obj2 Self2ObjHelp Self2 Obj2 Help

Obj2 HinderHelp Self2Help Obj2E https://youtu.be/DM2BFgJSDN4

Obj2Purple Obj2Pink Self2Black0.0

0.2

0.4

0.6

0.8

1.0Green’s Goal(s)

Hinder Help Obj2Purple0.0

0.2

0.4

0.6

0.8

1.0Red’s Goal(s)

Adversarial Neutral Friendly0.0

0.2

0.4

0.6

0.8

1.0Relationship

HelpHinder Obj2 Obj2 Obj2 Self2

Figure 4: Representative stimuli (left) and their corresponding human and model judgment on goals and relationships (right). The red andgreen circles are agents and the blue circle is an object. Here, we show the top 3 goals out of all 12 goals for each agent based on humanresponses. Personal goals are coded as “X2Y”, which indicates “get entity X to location Y”; colored squares represent landmarks; “Self”indicates the agent itself; “Obj” means the object entity. Note that the probabilities of goals do not sum up to 1 since an agent could have 1or 2 goals in a video. The ground-truth is highlighted by red underscore bars (the agents’ ground-truth goals in E was not among the top 3human responses.). We include the URL links for viewing the stimuli.

The Flatland paradigm offers a convenient, automated,and controllable way of generating a variety of social andphysical stimuli. While the present study considered two-agent Flatland scenarios with limited goals and properties,the framework can be extended to multiple agents, alternativeworld layouts, and different physical engines. Together withthe GSPI inference engine, Flatland improves on the currenttools for studying physical inference and social attribution,and allows researchers to study both of these phenomena atthe same time.

The model and humans sometimes disagree about the pos-sible agents’ goals. Some disagreements occur when themodel’s confidence is high, but human confidence is low (seethe top-left dots in Figure 3A). Individual inspection of thestimuli in question revealed three common cases for disagree-ment. First, human interpretations of agents’ actions are lessaccurate when agents are weak, leading to noisy estimates ofgoals. Second, humans sometimes report one of the agent’ssub-goal as the final goal, and miss to notice the other goals.Third, humans sometimes fail to recognize the personal goalsof a helping or hindering agent. Future work could study sub-

goal attribution in more detail, by asking subjects to reportall possible goals, along with their probability. Future workcould also investigate the richness of human judgements inambiguous stimuli, by asking subjects to informally describethe reasoning behind their inferences. For example, in highlyambiguous scenarios humans could rely on a library of ab-stract structures in social situations8, in order to generate ex-planations outside the space admissible by our model.

Disagreements may also happen when humans’ confidenceis high, but the model confidence is low (the bottom-centerdots in Figure 3A). Such scenarios are interesting becausethey may reveal non-uniform priors that humans bring to thetable. For example, humans might assume that the agents arefriendly or adversarial by default, leading to a biased goalinference. Alternatively, humans might place higher priorson certain types of goals in preference to other types. Suchpriors may also vary between subjects, with different subjectsexhibiting different kinds of non-uniform priors, depending

8For example, such abstract social structures could include: jeal-ousy, game-play, sport, flirtation, bluff, disappointment, etc.

on recent experience, context, or personality. We intend toinvestigate these phenomena in future work.

Lastly, in the current work we assume that both the sub-jects, and the model, know which entities are agents or ob-jects. This allows us to constrain the inference to goals, rela-tionships, and strengths, for simplicity of analyzing and pre-senting results. At the same time, telling apart agents fromobjects (i.e., animacy detection), as humans do easily andintuitively, is an interesting modelling challenge on its own.Judging animacy may require a complex interplay of appear-ance cues, ability to move on its own, producing the kinds ofmovement expected of animate agents (e.g. breathing, shiv-ering), as well as the interpretation of actions as intentionaland directed toward a goal. Future studies could investigatethe mental representations of such inferences, and our exper-imental paradigm could provide an empirical and computa-tional platform toward supporting this investigation.

A wider implication of our work, is that it demonstratesthe computational synergy between intuitive physics, hierar-chical planning, and theory of mind. While individual cog-nitive phenomena in these domains are traditionally consid-ered separate domains, much of cognition likely share this hi-erarchical, interdependent and multi-sensory structure. Thismeans that inferences informed by different sensory modali-ties and mental representations produced by different cogni-tive domains are available to each other. Our work takes astep toward a computational approach of studying the multi-sensory nature of the mind. We show that it is possible tocomputationally model how humans interpret complex socialand physical scenarios.

AcknowledgementThis work was supported by NSF STC award CCF-1231216,ONR MURI N00014-13-1-0333, and the ONR Science of AIprogram.

ReferencesAmbady, N., & Rosenthal, R. (1993). Half a minute: Predicting

teacher evaluations from thin slices of nonverbal behavior andphysical attractiveness. Journal of personality and social psy-chology, 64(3), 431.

Baker, C. L., Jara-Ettinger, J., Saxe, R., & Tenenbaum, J. B. (2017).Rational quantitative attribution of beliefs, desires and percepts inhuman mentalizing. Nature Human Behaviour, 1, 0064.

Balaguer, J., Spiers, H., Hassabis, D., & Summerfield, C. (2016).Neural mechanisms of hierarchical planning in a virtual subwaynetwork. Neuron, 90(4), 893–903.

Battaglia, P. W., Hamrick, J. B., & Tenenbaum, J. B. (2013). Simu-lation as an engine of physical scene understanding. Proceedingsof the National Academy of Sciences, 110(45), 18327–18332.

Carey, S. (2000). The origin of concepts. Journal of Cognition andDevelopment, 1(1), 37–41.

Fischer, J., Mikhael, J. G., Tenenbaum, J. B., & Kanwisher, N.(2016). Functional neuroanatomy of intuitive physical infer-ence. Proceedings of the national academy of sciences, 113(34),E5072–E5081.

Forbus, K. D. (2019). Qualitative representations: How peoplereason and learn about the continuous world. MIT Press.

Gelman, R., Durgin, F., & Kaufman, L. (1995). Distinguishingbetween animates and inanimates: Not by motion alone. Causalcognition: A multidisciplinary debate, 150–184.

Gilden, D. L., & Proffitt, D. R. (1989). Understanding collision dy-namics. Journal of Experimental Psychology: Human Perceptionand Performance, 15(2), 372.

Gureckis, T. M., Martin, J., McDonnell, J., Rich, A. S., Markant,D., Coenen, A., . . . Chan, P. (2016). psiturk: An open-sourceframework for conducting replicable behavioral experiments on-line. Behavior research methods, 48(3), 829–842.

Hamlin, K., Ullman, T., Tenenbaum, J., Goodman, N., & Baker, C.(2013). The mentalistic basis of core social cognition: Experi-ments in preverbal infants and a computational model. Develop-mental science, 16(2), 209–226.

Heider, F., & Simmel, M. (1944). An experimental study of apparentbehavior. The American Journal of Psychology, 243–259.

Huys, Q. J., Lally, N., Faulkner, P., Eshel, N., Seifritz, E., Gersh-man, S. J., . . . Roiser, J. P. (2015). Interplay of approximateplanning strategies. Proceedings of the National Academy of Sci-ences, 112(10), 3098–3103.

Jara-Ettinger, J., Schulz, L., & Tenenbaum, J. B. (2015). The naıveutility calculus: Joint inferences about the costs and rewards ofactions. In Cogsci.

Kaelbling, L. P., & Lozano-Perez, T. (2011). Hierarchical task andmotion planning in the now. In Ieee international conference onrobotics and automation.

Kryven, M., Ullman, T., Cowan, W., & Tenenbaum, J. B. (2016).Outcome or strategy? a bayesian model of intelligence attribu-tion. In Proceedings of the thirty-eighth annual conference of thecognitive science society.

Michotte, A. (1963). The perception of causality. Basic Books.Rabinowitz, N. C., Perbet, F., Song, H. F., Zhang, C., Eslami, S. A.,

& Botvinick, M. (2018). Machine theory of mind. arXiv preprintarXiv:1802.07740.

Shu, T., Peng, Y., Fan, L., Lu, H., & Zhu, S.-C. (2018). Perceptionof human interaction based on motion trajectories: From aerialvideos to decontextualized animations. Topics in Cognitive Sci-ence, 10(1), 225–241.

Shu, T., Peng, Y., Lu, H., & Zhu, S.-C. (2019). Partitioning theperception of physical and social events within a unified psycho-logical space. In Proceedings of the forty-first annual conferenceof the cognitive science society.

Sliwa, J., & Freiwald, W. A. (2017). A dedicated network for socialinteraction processing in the primate brain. Science, 356(6339),745–749.

Todorov, A., Mandisodza, A. N., Goren, A., & Hall, C. C. (2005).Inferences of competence from faces predict election outcomes.Science, 308(5728), 1623–1626.

Ullman, T. D., Baker, C. L., Macindoe, O., Evans, O., Goodman,N., & Tenenbaum, J. B. (2010). Help or hinder: Bayesian modelsof social goal inference. In Proceedings of advances in neuralinformation processing systems (p. 1874-1882).

van Buren, B., Gao, T., & Scholl, B. J. (2017). What are the underly-ing units of perceived animacy? chasing detection is intrinsicallyobject-based. Psychonomic bulletin & review, 24(5), 1604–1610.

Yildirim, I., Saeed, B., Bennett-Pierre, G., Gerstenberg, T., Tenen-baum, J., & Gweon, H. (2019). Explaining intuitive difficultyjudgments by modeling physical effort and risk. arXiv preprintarXiv:1905.04445.