Supplementary Material for Rel3D: A MinimallyContrastive Benchmark for Grounding Spatial

Relations in 3D

Ankit Goyal†, Kaiyu Yang†, Dawei Yang†‡, Jia Deng†‡University of Michigan, Ann Arbor, MI

†Princeton University, Princeton, NJ{agoyal, kaiyuy, daweiy, jiadeng}@princeton.edu

1 Predicate Vocabulary

There are in total 27336 images in Rel3D. Figure. 1 plots the number of images per predicate.

0 250 500 750 1000 1250 1500 1750Number Images

onunder

overon top of

to the side ofin

to the left of (wrt you)to the right of (wrt you)

faces awaybelow

faces towardsin front of

behindaround

leaning againstpoints away

neartouching

behind (wrt you)aligned to

points towardsinside

coveringin front of (wrt you)

to the right ofto the left of

far frompassing through

to the side of (wrt you)outside

Figure 1: Number of images per predicate in Rel3D

34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada.

2 Object Vocabulary

Table 1: Object categories in Rel3D along with their source.ShapeNetSem Only YCB Only Both in YCB and ShapeNetSem SimpleWordSpeaker Can Cereal Box BuildingComputer Fruits Bottle MountainPicture Frame Cup GlassesDesk Block RadioTV Ball StickMirror StonePerson HousePicture WheelBook ShoeChair DoorRug TireBed TreeCamera WallMedia Storage BagTablePlantGunCellPhonePlateCatControllerKnifeCapAnimalClockLadderCarWasherBusBoatFishRingTruckBirdTeapotAirplaneBowlForkSpoonCouchChild BedVaseToiletSinkSuitcaseBike

Table 2: Mapping between categories in ShapeNetSem and Rel3D. All categories in ShapeNetSemnot present in Rel3D either have few shapes or have spatial relations similar to some category inRel3D.

2

ShapeNetSem Cat. In Rel3D < 5 shapes Rel3D Cat. Similar Cat. in Rel3DWallArt Yes PictureChair,Recliner Yes ChairSpeaker Yes SpeakerLamp,DeskLamp No PlantChair,OfficeChair Yes ChairChair,rankChairs,OfficeSideChair Yes ChairComputer Yes ComputerChair,rankChairs Yes ChairTable,RoundTable,AccentTable Yes TablePicture Frame Yes Picture FrameCouch Yes CouchCandle No BottleDresser,ChestOfDrawers Yes Media StorageDesk Yes DeskHelicopter Yes AirplaneBottle,WineBottle Yes BottleClock,StandingClock Yes ClockBench Yes TableTV Yes TVMirror Yes MirrorPerson Yes PersonPicture Yes PictureBottle Yes BottleShelves,Bookcase Yes Media StorageTrashBin No CupBed,LoftBed No BedBooks Yes BookChair,OfficeSideChair Yes ChairBook Yes BookChair Yes ChairLamp,FloorLamp No PlantLaptop,Computer Yes ComputerFoodItem,MilkCarton Yes Cereal BoxOven,Counter No Yes WasherFireplace NoLamp,TableLamp No PlantRug Yes RugArcadeMachine No YesMonitor Yes TVLightSwitch No BlockOutdoorTable Yes TableBed Yes BedChair,rankChairs,OfficeChair Yes ChairTable,AccentTable Yes TableCamera Yes CameraMedia Storage Yes Media StorageTable Yes TablePlant Yes PlantGun Yes GunDresser,ChestOfDrawers,DresserWithMirror Yes Media StorageChild Bed Yes Child BedCabinet Yes Media StorageVase Yes VaseCabinet,Sideboard Yes Media StorageChair,Recliner,rankChairs Yes Chair

3

Lamp,WallLamp No PlantTvStand,Media Storage Yes Media StorageWindow No PictureStapler No BlockMicrowave No WasherViolin No KnifeTvStand,MediaStorage Yes Media StorageTable,DiningTable Yes TableTable,RoundTable Yes TableCellPhone Yes CellPhoneChair,SideChair Yes ChairPlate Yes PlateTeapot Yes TeapotCat Yes CatLamp,CeilingLamp No PlantController Yes ControllerArmoire,ChestOfDrawers Yes Media StorageChair,ChairWithOttoman Yes ChairLamp,LampPost No PlantArmoire,Dresser,Wardrobe,ChestOfDrawers Yes Media StorageArmoire,Wardrobe,ChestOfDrawers Yes Media StoragePicture Frame Yes Picture FrameBookcase Yes Media StorageOven No WasherRefrigerator No WasherTable,EndTable Yes TableTable,RoundTable,DiningTable Yes TableLamp,DeskLamp,FloorLamp No PlantPiano No TableBathtub No YesStool Yes TableWhiteboard No PictureKnife Yes KnifeLamp No PlantDonut,FoodItem No RingCap Yes CapGameTable Yes TableChair,Couch Yes ChairChestOfDrawers Yes Media StorageTable,EndTable,RoundTable Yes TableTable,CoffeeTable Yes TableMediaPlayer No WasherMousePad No YesAnimal Yes AnimalPicture,Painting Yes PictureAirplane Yes AirplaneToilet Yes ToiletChair,AccentChair Yes ChairFoodItem No YesComputer Yes ComputerPrinter No WasherDartBoard No WallArtCurtain No DoorBooth No YesSpaceShip Yes AirplaneHorse Yes Animal

4

Nightstand,ChestOfDrawers Yes Media StorageTank No TruckFileCabinet Yes Media StorageBed,PosterBed Yes BedCabinet,CurioCabinet Yes Media StorageTable,RoundTable,CoffeeTable Yes TableSink Yes SinkIpod Yes CellPhoneBed,SingleBed Yes BedAnimal,PlushToy Yes AnimalToyFigure No Block, PersonBed,KingBed Yes BedTvStand No YesNotepad Yes BookDryer Yes WasherWallArt,Painting Yes PictureBottle,BeerBottle Yes BottleNintendoDS,VideoGameController Yes ControllerBarTable Yes TableBowl Yes BowlChair,Chaise,rankChairs Yes ChairClock Yes ClockStool,Barstool Yes TableToy No YesClock,WallClock Yes ClockRock Yes StoneBottle,DrinkBottle Yes BottleDart No YesFork Yes ForkCabinet,FileCabinet No Media StorageCoffeeMaker No SinkDresserWithMirror Yes MirrorLadder Yes LadderChair,Chaise Yes ChairPoster,WallArt Yes PictureCouch,Loveseat Yes CouchChair,BeanBag Yes ChairSuitcase Yes SuitcaseWii,VideoGameConsole Yes ControllerDog Yes AnimalPianoKeyboard No Controller, BlockPSP,VideoGameController Yes ControllerSnowman No YesCabinet,Sideboard,CurioCabinet No YesDishwasher Yes WasherSpoon Yes SpoonHorse,ToyFigure Yes AnimalRadio Yes RadioCar Yes CarFoodItem,Cereal Box Yes Cereal BoxPainting Yes PictureWasher Yes WasherBus Yes BusBed,DoubleBed Yes BedCereal Box Yes Cereal BoxComputerMouse Yes

5

ToyFigure,SpaceShip NoBowl,FoodItem No BowlSodaCan Yes CanBoat Yes BoatFish Yes FishGameTable,RoundTable Yes TableCoin No Plate, RingPicture,WallArt,Painting Yes PictureAirplane,ToyFigure,rankAirplanes Yes AirplaneMediaDiscs No YesIpad Yes CellPhoneRing Yes RingPowerSocket No YesIpod,Cabling No Yes_StanfordSceneDBModels NoCouch,Sectional Yes CouchBackpack Yes BagTruck Yes TruckLamp,DeskLamp,TableLamp No YesElephant Yes AnimalBed Yes BedDesk,DraftingTable Yes DeskBed,RoundBed Yes BedPen No YesChair,SideChair,rankChairs Yes ChairBoard Yes PictureBowl,FruitBowl Yes BowlKeyboard No Block, BookDoor Yes DoorChair,rankChairs,AccentChair Yes ChairChestOfDrawers,DresserWithMirror Yes Media StorageIpod,Headphones No YesDiscCase No YesTank,ToyFigure No YesHeadphones YesLDesk No DeskCounter No TableNightstand No YesRack,Shelves,CoatRack Yes Media StorageSword Yes KnifeShip Yes BoatHanger No YesEasel No YesBed,CanopyBed,PosterBed No BedArmoire,Wardrobe No YesVanity No YesSkateBoard No YesBird Yes BirdBedWithNightstand Yes BedToy,Clock Yes ClockCar,ToyFigure Yes CarCow Yes AnimalShelves Yes Media StorageCamera,DSLRCamera Yes CameraBike Yes BikeBicycle Yes Bike

6

Thumbtack No YesPicture,WallArt No PictureShirt No Yes_OIMwhitelist No YesRack No YesFries,FoodItem No YesHat Yes CapBowl,FoodItem,FruitBowl Yes Yes BowlArmoire Yes Media StoragePowerStrip No YesChair,KneelingChair Yes ChairHarp No YesRack,Shelves Yes Media StorageArmoire,Dresser,ChestOfDrawers Yes Media StorageTable,ChestOfDrawers Yes TableToy,Horse Yes Yes AnimalKnife,Utensils Yes KnifeIpod,CellPhone Yes CellPhoneGamecube,VideoGameConsole Yes ControllerStaplerWithStaples No YesCabinet,ChestOfDrawers Yes Media StorageToy,Truck Yes TruckBowl,Plate Yes BowlFaucet Yes Yes SinkKettle Yes TeapotToy,Animal,PlushToy Yes AnimalVideoGameConsole Yes ControllerBed,Crib Yes Child BedToiletPaper NoWasherDryerSet Yes WasherDonkey Yes AnimalTissueBox No YesRiceCooker No YesCredenza No YesBed,KingBed Yes BedPlate,FoodItem,FoodPlate Yes PlatePoster Yes PictureShower No YesLectern No YesCassette No YesWallArtWithFigurine Yes PictureGlasses Yes GlassesTvStand,RoundTable,MediaStorage Yes TableXbox,VideoGameConsole No YesGuitarStand No Yes_BAD No YesTelephone No YesBed,Cradle Yes Child BedToy,Elephant,PlushToy,ToyFigure Yes AnimalBidet No YesDeskLamp No PlantTV,TvStand No YesRadio,Clock Yes ClockBed,Dresser,DoubleBed,ChestOfDrawers Yes BedRoom NoCalculator No Block, Controller

7

Toy,Airplane No Airplane

Table 3: Mapping between objects in YCB and Rel3D.Object ID Rel3D Category Reason for not including in Rel3D001_chips_can Can002_master_chef_can Can003_cracker_box Cereal Box004_sugar_box Cereal Box005_tomato_soup_can Can006_mustard_bottle Bottle007_tuna_fish_can Can008_pudding_box Cereal Box009_gelatin_box Cereal Box010_potted_meat_can Can011_banana Fruits012_strawberry Fruits013_apple Fruits014_lemon Fruits015_peach Bad reconstruction016_pear Fruits017_orange Fruits018_plum Bad reconstuction019_pitcher_base Bad reconstuction021_bleach_cleanser Bottle022_windex_bottle Largely distorted023_wine_glass Missing Object024_bowl Bad reconstuction025_mug Cup026_sponge Block027_skillet Bad reconstuction028_skillet_lid Missing object029_plate Bad reconstuction030_fork Bad reconstuction031_spoon Bad reconstuction032_knife Bad reconstuction033_spatula Missing object035_power_drill Only 1 shape036_wood_block Block037_scissors Only 1 shape038_padlock Bad reconstruction039_key Missing object040_large_marker Only 1 shape041_small_marker Bad reconstuction042_adjustable_wrench Bad reconstuction043_phillips_screwdriver Bad reconstuction044_flat_screwdriver Bad reconstuction048_hammer Only 1 shape049_small_clamp Missing .obj file050_medium_clamp Bad reconstuction051_large_clamp Bad reconstuction052_extra_large_clamp Only 1 Shape053_mini_soccer_ball Bad reconstuction054_softball Ball055_baseball Ball056_tennis_ball Ball

8

057_racquetball Ball058_golf_ball Ball059_chain Bad reconstruction061_foam_brick Block062_dice Bad reconstruction063-a_marbles Bad reconstruction063-b_marbles Missing objects063-c_marbles Missing objects063-d_marbles Missing objects063-e_marbles Missing objects063-f_marbles Missing objects065-a_cups Bad reconstruction, looks like a blob065-b_cups Bad reconstruction065-c_cups Bad reconstruction065-d_cups Cup065-e_cups Cup065-f_cups Cup065-g_cups Cup065-h_cups Bad reconstruction065-i_cups Cup065-j_cups Cup070-a_colored_wood_blocks Bad reconstruction070-b_colored_wood_blocks Missing objects071_nine_hole_peg_test Block072-a_toy_airplane Small Object Part072-b_toy_airplane Small Object Part072-c_toy_airplane Small Object Part072-d_toy_airplane Small Object Part072-e_toy_airplane Small Object Part072-f_toy_airplane Small Object Part072-g_toy_airplane Small Object Part072-h_toy_airplane Small Object Part072-i_toy_airplane Small Object Part072-j_toy_airplane Small Object Part072-k_toy_airplane Small Object Part073-a_lego_duplo Small Object Part073-b_lego_duplo Small Object Part073-c_lego_duplo Small Object Part073-d_lego_duplo Small Object Part073-e_lego_duplo Small Object Part073-f_lego_duplo Small Object Part073-g_lego_duplo Small Object Part073-h_lego_duplo Small Object Part073-i_lego_duplo Small Object Part073-j_lego_duplo Small Object Part073-k_lego_duplo Small Object Part073-l_lego_duplo Small Object Part073-m_lego_duplo Small Object Part076_timer Only 1 shape077_rubiks_cube Block078_tshirt No obj model

9

3 Relation Plots

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

aligned topositive

negative

(a) aligned to

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

aroundpositive

negative

(b) around

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

behind (wrt you)positive

negative

(c) behind (wrt you)

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

behindpositive

negative

(d) behind

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

coveringpositive

negative

(e) covering

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

faces awaypositive

negative

(f) faces away

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

faces towardspositive

negative

(g) faces towards

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

far frompositive

negative

(h) far from

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

in front of (wrt you)positive

negative

(i) in front of (wrt you)

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

in front ofpositive

negative

(j) in front of

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

inpositive

negative

(k) in

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

insidepositive

negative

(l) inside

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

leaning againstpositive

negative

(m) leaning against

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

nearpositive

negative

(n) near

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

on top ofpositive

negative

(o) on top of

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

onpositive

negative

(p) on

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

outsidepositive

negative

(q) outside

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

overpositive

negative

(r) over

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

passing throughpositive

negative

(s) passing through

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

points awaypositive

negative

(t) points away

10

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

points towardspositive

negative

(u) points towards

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

to the left of (wrt you)positive

negative

(v) to the left of (wrtyou)

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

to the left ofpositive

negative

(w) to the left of

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

to the right of (wrt you)positive

negative

(x) to the right of (wrtyou)

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

to the right ofpositive

negative

(y) to the right of

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

to the side of (wrt you)positive

negative

(z) to the side of (wrtyou)

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

to the side ofpositive

negative

(aa) to the side of

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

touchingpositive

negative

(ab) touching

x

−3 −2 −10

12

3

z

−3

−2

−1

0

1

2

3

y

−3

−2

−1

0

1

2

3

underpositive

negative

(ac) underFigure 2: Each dot represents a scene in our dataset (blue means positive examples and red fornegative). The location of the dot represent the relative position of the object w.r.t. to the subject inthe frame of reference of the observer.

11

4 Model Hyperparameters

Table 4: All the hyper-parameters we used to tune the baseline models in Rel3D. The values in boldrepresent the hyperparameter choice that performed the best on the validation set.

Model l2 regularization Feature dim Roi size Back-bone

2D 0, 1e-4, 1e-3 64, 128, 256, 512VtranE 0, 1e-6, 1e-4, 1e-2 128, 256, 512, 1024, 2048, 4096 1, 3, 5 resnet18VipCNN 0, 1e-6, 1e-4 3, 5, 7, 9 resnet18, resnet101DRNet 0, 1e-6, 1e-4, 1e-3 64, 128, 256, 512PPFRCN 0, 1e-6, 1e-4 3 resnet18, resnet101

5 Results Predicate-wise

Model alig

ned

to

arou

nd

behi

nd

behi

nd(w

rtyo

u)be

low

cove

ring

face

s aw

ayfa

ces t

owar

dsfa

r fro

m

in infro

ntof

infro

ntof

(wrt

you)

insid

e

lean

ing

agai

nst

near

on onto

pof

outsi

de

over

pass

ing

thro

ugh

poin

tsaw

ay

poin

tsto

war

dsto

the l

eft o

fto

the l

eft o

f (w

rtyo

u)to

the r

ight

ofto

the r

ight

of(w

rtyo

u)to

the s

ide o

fto

the s

ide o

f (w

rtyo

u)to

uchi

ng

2d 0.578 0.817 0.576 0.945 0.845 0.873 0.513 0.5 0.872 0.762 0.625 0.931 0.717 0.757 0.916 0.845 0.831 0.667 0.798 0.67 0.516 0.551 0.717 0.948 0.517 0.935 0.661 0.889 0.674

VtranE 0.612 0.768 0.53 0.961 0.818 0.822 0.526 0.55 0.723 0.766 0.578 0.944 0.692 0.716 0.878 0.849 0.838 0.667 0.805 0.717 0.548 0.543 0.667 0.933 0.542 0.952 0.641 0.792 0.667

VipCNN 0.647 0.774 0.606 0.914 0.75 0.803 0.545 0.567 0.872 0.703 0.602 0.889 0.725 0.662 0.882 0.797 0.77 0.583 0.743 0.594 0.492 0.623 0.5 0.881 0.592 0.917 0.713 0.889 0.727

DRNet 0.612 0.848 0.576 0.961 0.777 0.866 0.539 0.558 0.723 0.752 0.625 0.958 0.75 0.676 0.849 0.832 0.858 0.708 0.775 0.689 0.5 0.529 0.6 0.952 0.475 0.957 0.67 0.875 0.667

PPFRCN 0.647 0.811 0.606 0.844 0.777 0.793 0.513 0.567 0.83 0.752 0.633 0.889 0.7 0.635 0.857 0.828 0.791 0.542 0.741 0.66 0.516 0.601 0.817 0.913 0.558 0.939 0.69 0.903 0.667

MLP (Aligned Absolute) 0.672 0.945 0.803 0.977 0.946 0.904 0.955 0.783 0.787 0.841 0.891 0.944 0.817 0.797 0.929 0.909 0.824 0.792 0.869 0.698 0.913 0.703 0.817 0.956 0.808 0.965 0.796 0.889 0.674

MLP (Raw Absolute) 0.672 0.915 0.682 0.977 0.946 0.889 0.818 0.608 0.862 0.831 0.742 0.903 0.85 0.743 0.933 0.888 0.845 0.833 0.874 0.708 0.786 0.652 0.717 0.94 0.567 0.987 0.716 0.917 0.659

Human 0.888 0.982 0.97 0.961 0.946 0.965 0.968 0.925 0.777 0.955 0.938 0.958 0.933 0.932 0.966 0.983 0.973 0.75 0.982 0.972 0.9682 0.986 0.95 0.988 0.958 0.983 0.951 0.986 0.932

12