Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for...
-
Upload
christian-chapman -
Category
Documents
-
view
214 -
download
0
Transcript of Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for...
![Page 1: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/1.jpg)
Image Statistics and the Image Statistics and the Perception of 3D ShapePerception of 3D ShapeRoland W. FlemingMax Planck Institute
for Biological Cybernetics
Yuanzhen Li
Edward H. AdelsonMassachusetts Institute of Technology
![Page 2: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/2.jpg)
Matte
Glossy
Mirrored
![Page 3: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/3.jpg)
Hen
ry M
oore
![Page 4: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/4.jpg)
Visual system estimates surface orientation from image intensity
Classical Classical Shape from ShadingShape from Shading
reflectance mapimage
![Page 5: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/5.jpg)
Visual system estimates surface orientation from image intensity
Classical Classical Shape from ShadingShape from Shading
reflectance map
Problems: Intensities are ambiguous
Reflectance map is unknown
No principled way to predict successes vs. failures of shape perception
![Page 6: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/6.jpg)
Surface reflectanceSurface reflectance
A parametric space of glossy plastic materials (using Ward model)
Diffuse Reflectance, dDiffuse Reflectance, d
Sp
ecu
lar
Reflect
an
ce,
sS
pecu
lar
Reflect
an
ce,
s
![Page 7: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/7.jpg)
Don’t use image intensity ! Use the kinds of image measurements the visual
system employs at the front end
Alternative approachAlternative approach
reflectance mapimage
![Page 8: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/8.jpg)
Don’t use image intensity ! Use the kinds of image measurements the visual
system employs at the front end
Alternative approachAlternative approach
image
What can these measurements tell us about 3D shape ?
Can filter responses predict human shape perception ?
![Page 9: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/9.jpg)
highly curved
Curvatures determineCurvatures determinedistortionsdistortions
![Page 10: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/10.jpg)
slightlycurved
Anisotropies in surface curvature lead to powerful distortions of the reflected world
Curvatures determineCurvatures determinedistortionsdistortions
![Page 11: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/11.jpg)
Population codesPopulation codes
![Page 12: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/12.jpg)
Population codesPopulation codes
![Page 13: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/13.jpg)
Population codesPopulation codes
![Page 14: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/14.jpg)
Population codesPopulation codes
![Page 15: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/15.jpg)
StatisticsStatisticsIlluminations
Sh
ap
es
Render many images: 50 Shapes 12 Illuminations 5 Reflectances
Measure the distribution of orientations (i.e. filter population response) for every point in every image
Look for regularities
![Page 16: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/16.jpg)
Orientation fieldsOrientation fields
Ground truth
![Page 17: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/17.jpg)
Orientation fieldsOrientation fields
Error (estimate - ground truth)
![Page 18: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/18.jpg)
Surface reflectanceSurface reflectance
Diffuse Reflectance, dDiffuse Reflectance, d
Sp
ecu
lar
Reflect
an
ce,
sS
pecu
lar
Reflect
an
ce,
s
![Page 19: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/19.jpg)
If the visual system relies on these measurements then:
1: Shape perception should be stable across changes that do not affect these measurements
2: Perceived shape should vary systematically when scene or image modifications do affect these measurements
PredictionsPredictions
![Page 20: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/20.jpg)
Perceived shape should be extremely stable across changes in surface glossiness.
Prediction 1Prediction 1
Nefs, Koenderink & Kappers, 2006
“We found no evidence that the perceived shapes of glossy objects are different from the perceived shapes of matte objects...”
![Page 21: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/21.jpg)
ExperimentExperiment
Specular reflectionSpecular reflection Diffuse reflectionDiffuse reflection
![Page 22: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/22.jpg)
ExperimentExperiment
Specular reflectionSpecular reflection Diffuse reflectionDiffuse reflection
![Page 23: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/23.jpg)
Orientation fieldsOrientation fields
ground truth
![Page 24: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/24.jpg)
Orientation fieldsOrientation fields
ground truth
![Page 25: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/25.jpg)
For shaded surfaces, perceived shape should undergo (subtle) changes across variations in illumination
Prediction 2Prediction 2
![Page 26: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/26.jpg)
For shaded surfaces, perceived shape should undergo (subtle) changes across variations in illumination
Prediction 2Prediction 2
![Page 27: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/27.jpg)
For shaded surfaces, perceived shape should undergo (subtle) changes across variations in illumination
Prediction 2Prediction 2
Todd, Norman, Koenderink & Kappers (1997) report little effect of illumination. But that was with additional cues.
Koenderink, van Doorn, Christou & Lappin (1996)
Nefs, Koenderink & Kappers, 2006
![Page 28: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/28.jpg)
For shaded surfaces, perceived shape should undergo (subtle) changes across variations in illumination
Prediction 2Prediction 2
Caniard & Fleming, 2007
![Page 29: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/29.jpg)
If the visual system relies on these measurements then:
1: Shape perception should be stable across changes that do not affect these measurements. Even when these changes are not natural.
2: Perceived shape should vary systematically when scene or image modifications do affect these measurements
PredictionsPredictions
![Page 30: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/30.jpg)
Test improbable combination of lighting and reflectance
Decouple intensity from image orientation
non-linear intensity transfer function
normal shadingnormal shading ‘‘weird’ shadingweird’ shading
““Weird” ShadingWeird” Shading
![Page 31: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/31.jpg)
normal shadingnormal shading‘‘weird’ shadingweird’ shading
““Weird” shadingWeird” shading
![Page 32: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/32.jpg)
normal shadingnormal shading‘‘weird’ shadingweird’ shading
““Weird” shadingWeird” shading
perceived tiltperceived tilt
perceived slantperceived slant
normal shadingnormal shading
““ weir
d”
shadin
gw
eir
d”
shad
ing
S1
S1
![Page 33: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/33.jpg)
normal shadingnormal shading‘‘weird’ shadingweird’ shading
““Weird” shadingWeird” shading
perceived tiltperceived tilt
perceived slantperceived slant
normal shadingnormal shading
““ weir
d”
shadin
gw
eir
d”
shad
ing
S2
S2
![Page 34: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/34.jpg)
normal shadingnormal shading
‘‘ weir
d’
shad
ing
weir
d’
shad
ing
““Weird” shadingWeird” shading
Pooled data across 6 shapes
tilttilt slantslant
normal shadingnormal shading
r2 = 0.93 r2 = 0.88
![Page 35: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/35.jpg)
Affine TransformationAffine Transformation
Shear:- does affect first derivatives- does NOT affect second derivatives
![Page 36: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/36.jpg)
Shear:- does affect first derivatives- does NOT affect second derivatives
Affine TransformationAffine Transformation
![Page 37: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/37.jpg)
Shear:- does affect first derivatives- does NOT affect second derivatives
Affine TransformationAffine Transformation
![Page 38: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/38.jpg)
Shear:- does affect first derivatives- does NOT affect second derivatives
Affine TransformationAffine Transformation
![Page 39: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/39.jpg)
Matching TaskMatching Task
Subject adjusts shear of match until it appears to be same shape as test
test match
![Page 40: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/40.jpg)
Matching TaskMatching Task
Subject adjusts shear of match until it appears to be same shape as test
test match
![Page 41: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/41.jpg)
Matching TaskMatching Task
Subject adjusts shear of match until it appears to be same shape as test
test match
![Page 42: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/42.jpg)
Matching TaskMatching Task
Subject adjusts shear of match until it appears to be same shape as test
test match
![Page 43: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/43.jpg)
PredictionsPredictions
test shear
matc
h s
hear ve
ridi
cal
image statisticsprediction
![Page 44: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/44.jpg)
ResultsResults
test shear
matc
h s
hear
![Page 45: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/45.jpg)
If the visual system relies on these measurements then:
1: Shape perception should be stable across changes that do not affect these measurements.
2: Perceived shape should vary systematically when scene or image modifications do affect these measurements. Even when these changes are not natural.
PredictionsPredictions
![Page 46: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/46.jpg)
Illusory distortionsIllusory distortionsof shapeof shape
Inspired by Todd & Thaler VSS 05
![Page 47: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/47.jpg)
Illusory distortionsIllusory distortionsof shapeof shape
Inspired by Todd & Thaler VSS 05
![Page 48: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/48.jpg)
Illusory distortionsIllusory distortionsof shapeof shape
Inspired by Todd & Thaler VSS 05
![Page 49: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/49.jpg)
Illusory distortionsIllusory distortionsof shapeof shape
![Page 50: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/50.jpg)
Illusory distortionsIllusory distortionsof shapeof shape
![Page 51: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/51.jpg)
Experiment
![Page 52: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/52.jpg)
ResultsResultsveridicalstimulus
![Page 53: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/53.jpg)
ResultsResultspredictedstimulus
![Page 54: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/54.jpg)
ResultsResultsresultsstimulus
![Page 55: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/55.jpg)
Dot product between subject’s data and predictions
ResultsResults
![Page 56: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/56.jpg)
Dot product between subject’s data and predictions
ResultsResults
![Page 57: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/57.jpg)
Dot product between subject’s data and predictions
ResultsResults
“veridical”prediction
“ori
en
tati
on
field
”p
red
icti
on
![Page 58: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/58.jpg)
Dot product between subject’s data and predictions
ResultsResults
“veridical”prediction
“ori
en
tati
on
field
”p
red
icti
on
![Page 59: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/59.jpg)
ConclusionsConclusions
Useful shape cues can be derived from relatively simple image measurements at the front end of vision
In some cases these measurements are surprisingly robust across variations in other scene properties (e.g. illumination, reflectance).
Scale and orientation measurements can predict certain successes and failures of human 3D shape perception across a range of natural and unnatural stimuli.
![Page 60: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/60.jpg)
Thank youThank youFunding
RF supported byDFG FL 624/1-1
![Page 61: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/61.jpg)
Generative space of all possible combinations of surface curvature and local orientation in the reflectance map
Expected errorsExpected errors
![Page 62: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/62.jpg)
Reflectance as IlluminationReflectance as Illumination
a(f) = 1 / f
= 0 = 0.4 = 0.8 = 1.2
= 1.6 = 2.0 = 4.0 = 8.0
![Page 63: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/63.jpg)
Cues to 3D ShapeCues to 3D Shape
specularities shading texture
Conventional wisdom: different cues have different physical causes must be processed differently by visual system (‘modules’)
![Page 64: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/64.jpg)
specularities shading texture
Goal: Find commonalities between cues.
Cues to 3D ShapeCues to 3D Shape
![Page 65: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/65.jpg)
Cues to 3D ShapeCues to 3D Shape
![Page 66: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/66.jpg)
Cues to 3D ShapeCues to 3D Shape
Fleming, Torralba, Adelson
Todd and colleagues
Mingolla and Grossberg
Koenderink and van Doorn
Zucker and colleagues
Zaidi and Li
Malik and Rosenholtz
![Page 67: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/67.jpg)
Rendering withRendering withReflectance mapsReflectance maps
Reflectance map is a lookup-table that specifies image intensity for all surface normals Surface normals are indices for accessing values from the reflectance map
Within a local patch of surface, the normal changes smoothly This maps a small patch of the reflectance map “texture” into the image The rate at which the indices sweep through the reflectance map determines the warping
transformation that is applied to the texture patch during the mapping
![Page 68: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/68.jpg)
Hierarchy of shapeHierarchy of shapeattributesattributes
We often refer to “stereo” or “texture”, or “shading” as “cues” to shape.
Traditional definition of shape cue: a physical property that can inform us about shape, e.g. “stereo”, or “texture”, or “shading”
New definition of cue: a specific image measurement that provides statistically reliable information about a specific property of the scene.
Any given cue on its own may be highly ambiguous, specifying some abstract, high level scene property that does not uniquely specify the object
![Page 69: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/69.jpg)
Hierarchy of shapeHierarchy of shapeattributesattributes
Easily measurable image statistics that can inform us about any property of shape
![Page 70: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/70.jpg)
It is remarkable that we can recover 3D shape:
No motion No stereo No shading No texture
image consists of nothing more than a distorted reflection of the world surrounding the object
Ideal mirrored surface
Fleming et al. (2004). JOV
Shape from SpecularitiesShape from Specularities
![Page 71: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/71.jpg)
As the object moves from scene to scene, the image changes dramatically.
Yet, somehow we are able to recover the 3D shape.
Shape from SpecularitiesShape from Specularities
![Page 72: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/72.jpg)
Approach IApproach I::inverse opticsinverse optics
Estimate shape by inverting the physics of mirror reflections.
Image from Savarese and Perona
Make an explicit model of the environment
Make assumptions about specific environmental features (e.g. ‘lines are straight’)
![Page 73: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/73.jpg)
Estimate shape directly from the image Collect image measurements that are reliable
across ‘typical’ environments
Approach IIApproach II::direct perceptiondirect perception
No need to estimate the environment
ust use the pattern of distortions in the image
![Page 74: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/74.jpg)
Pattern of compressions and rarefactions across the image indicates something about the 3D shape.
Shape from TextureShape from Texture
![Page 75: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/75.jpg)
Real-world illumination is highly structured Specular reflections of the real world are a bit like texture Can we solve the 3D shape of mirrors using shape-from-
texture ?
Shape from TextureShape from Texture
??
![Page 76: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/76.jpg)
Slant distorts texture but not reflections
Image distortionsImage distortions
![Page 77: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/77.jpg)
Image distortionsImage distortions
![Page 78: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/78.jpg)
Image distortionsImage distortions
![Page 79: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/79.jpg)
Curvature distorts reflections but not texture
Image distortionsImage distortions
![Page 80: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/80.jpg)
Shape-from-textureShape-from-textureandand
shape-from-specularityshape-from-specularityfollow different rulesfollow different rules
For texture, image compression depends on surface slant
first derivative of surface
For reflections, image compression depends on surface curvature properties
second derivatives of surface
![Page 81: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/81.jpg)
Local analysis: Local analysis: banding patternsbanding patterns
![Page 82: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/82.jpg)
Gauge Figure TaskGauge Figure Task
Subject adjusts 3D orientation of “gauge figure” to match local orientation of surface
![Page 83: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/83.jpg)
Slant and TiltSlant and Tilt
Image from Palmer, 1999
![Page 84: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/84.jpg)
Results IResults I
objective tilt
subje
ctiv
e t
ilt
TiltTilt
objective slant
subje
ctiv
e s
lant
SlantSlant
objective tilt
subje
ctiv
e t
ilt
objective slant
subje
ctiv
e s
lant
TiltTilt SlantSlant
![Page 85: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/85.jpg)
Results IIResults II
objective tilt
subje
ctiv
e t
ilt
TiltTilt
objective slant
subje
ctiv
e s
lant
SlantSlant
objective tilt
subje
ctiv
e t
ilt
objective slant
subje
ctiv
e s
lant
TiltTilt SlantSlant
![Page 86: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/86.jpg)
Is it just the occluding contour?Is it just the occluding contour?
No, it is not
![Page 87: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/87.jpg)
Interpreting distortedInterpreting distortedreflectionsreflections
![Page 88: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/88.jpg)
Effects of Effects of compressioncompression
![Page 89: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/89.jpg)
3D shape appears to be conveyed by the continuously varying patterns of orientation across the image of a surface
![Page 90: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/90.jpg)
Beyond specularityBeyond specularity
Specular reflectionSpecular reflection Diffuse reflectionDiffuse reflection
![Page 91: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/91.jpg)
Differences betweenDifferences betweendiffuse and specular reflectiondiffuse and specular reflection
![Page 92: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/92.jpg)
Differences betweenDifferences betweendiffuse and specular reflectiondiffuse and specular reflection
![Page 93: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/93.jpg)
Differences betweenDifferences betweendiffuse and specular reflectiondiffuse and specular reflection
![Page 94: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/94.jpg)
ShinyShiny
Painted Painted
![Page 95: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/95.jpg)
Beyond specularityBeyond specularity
Specular reflectionSpecular reflection Diffuse reflectionDiffuse reflection
![Page 96: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/96.jpg)
Latent orientationLatent orientationstructurestructure
![Page 97: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/97.jpg)
Orientation fieldsOrientation fieldsin shadingin shading
![Page 98: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/98.jpg)
Orientation fieldsOrientation fieldsin shadingin shading
![Page 99: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/99.jpg)
highly curved
![Page 100: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/100.jpg)
slightlycurved
Anisotropies in surface curvature lead to anisotropies in the image.
![Page 101: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/101.jpg)
TextureTexture
Anisotropic compression of texture depends on surface slant
![Page 102: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/102.jpg)
TextureTexture
Anisotropic compression of texture depends on surface slant
![Page 103: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/103.jpg)
Orientation fieldsOrientation fieldsin texturein texture
![Page 104: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/104.jpg)
Orientation fieldsOrientation fieldsin texturein texture
![Page 105: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/105.jpg)
Orientation fieldsOrientation fieldsin texturein texture
![Page 106: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/106.jpg)
No need for visual system to estimate reflectance or illumination explicitly.
Classical shape from shading uses the reflectance map to estimate surface normals from image intensities
Reflectance map is usually unknown and ambiguous
Potential of Potential of Orientation FieldsOrientation Fields
![Page 107: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/107.jpg)
Visual system estimates surface orientation from image intensity
Classical Classical Shape from ShadingShape from Shading
reflectance mapimage
![Page 108: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/108.jpg)
Stable across albedo discontinuities.
Breton and Zucker (1996), Huggins and Zucker (2001)
Potential of Potential of Orientation FieldsOrientation Fields
![Page 109: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/109.jpg)
Uses biologically plausible measurements
Orientation selectivity maps in primary visual cortex of tree shrew. After Bosking et al. (1997).
Potential of Potential of Orientation FieldsOrientation Fields
![Page 110: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/110.jpg)
May explain how images with no obvious BRDF interpretation nevertheless yield 3D percepts
Potential of Potential of Orientation FieldsOrientation Fields
Ohad Ben-Shahar
![Page 111: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/111.jpg)
Converting between cuesConverting between cues
input imageinput image
Todd & Oomes 2004
( )2
Latent shadingLatent shading
![Page 112: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/112.jpg)
( )2
Converting between cuesConverting between cues
input imageinput image
Todd & Oomes 2004
Latent shadingLatent shading
![Page 113: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/113.jpg)
Matte vs. ShinyMatte vs. Shiny Same generative statistics, different mappings
Mapped Mapped as as texturetexture
Mapped as Mapped as reflectionreflection
Mapped Mapped as as texturetexture
Mapped as Mapped as reflectionreflection
![Page 114: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/114.jpg)
Texture vs. ReflectanceTexture vs. Reflectance
![Page 115: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/115.jpg)
Texture vs. ReflectanceTexture vs. Reflectance
![Page 116: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/116.jpg)
Texture vs. ReflectanceTexture vs. Reflectance
![Page 117: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/117.jpg)
Texture vs. ReflectanceTexture vs. Reflectance
![Page 118: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/118.jpg)
ConclusionsConclusions
Orientation fields are potentially a very powerful source of information about 3D shape
For the early stages of 3D shape processing, seemingly different cues may have more in common than previously thought
![Page 119: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/119.jpg)
Todd’s BlobsTodd’s Blobs
![Page 120: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/120.jpg)
Todd’s BlobsTodd’s Blobs
![Page 121: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/121.jpg)
What still needs to be explained?What still needs to be explained?
For Lambertian materials (or blurry illuminations), the reflectance map is so smooth that it is significantly anisotropic.
Therefore shading orientation fields vary considerably with changes in illumination.
sidefront top
![Page 122: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/122.jpg)
What still needs to be explained?What still needs to be explained?
Note analogy to textures of different orientations
Todd et al. (2004)
![Page 123: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/123.jpg)
Two possibilitiesTwo possibilities
I. Change in orientation field predicts (subtle) changes in perceived 3D shape
II. There are higher-order invariants in the orientation fields
sidefront top
![Page 124: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/124.jpg)
Eigenvectors of Hessian matrix
Intrinsic principal curvatures
![Page 125: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/125.jpg)
![Page 126: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/126.jpg)
Matte dark grey
Rough metal
Glossy light grey
![Page 127: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/127.jpg)
PlasticsPlastics
(a) Mirror (b) Smooth plastic (c) Rough plastic
![Page 128: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/128.jpg)
When the world is anisotropicWhen the world is anisotropic
Brushed horizontally Brushed vertically
![Page 129: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/129.jpg)
Stripy worldStripy world
![Page 130: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/130.jpg)
Matte vs. ShinyMatte vs. Shiny Same generative statistics, different mappings
Mapped as texture
Mapped as reflection
Mapped as texture
Mapped as reflection
![Page 131: Image Statistics and the Perception of 3D Shape Roland W. Fleming Max Planck Institute for Biological Cybernetics Yuanzhen Li Edward H. Adelson Massachusetts.](https://reader038.fdocuments.net/reader038/viewer/2022110207/56649d835503460f94a69bfd/html5/thumbnails/131.jpg)
Hypothesis: the way the reflections are distorted is systematically related to properties of the 3D shape
Shape from SpecularitiesShape from Specularities