Post on 01-Nov-2019
1
COMPSCI 773• Lecturers: Dr Patrice Delmas (303.389) Lecturers: Dr Patrice Delmas (303.389)
A. Prof. Georgy Gimel’farb (303.391) • Lecture time: W 1pm-2pm F 12pm-2pm• Marking:
– 40% Final exam – 60% Lab work (30% group work, 20% individual assessment, 10%
oral assessment)oral assessment)• 3 assignments
March 2, 2010 1COMPSCI 773 S1T
COMPSCI 7731st assignment: camera calibration synchronisation of USB 1 assignment: camera calibration, synchronisation of USB
cameras for image acquisition, image pairs rectifications –03.04.09
2nd assignment: image matching / stereo matching algorithms implementation, depth map construction, 3D visualisation (OpenGL/CUDA, optional) – 08.05.09
3rd assignment (06.06.09):– 2+3D Face mapping/recognition (tentative)– Whole system testing (live demo)
March 2, 2010 2COMPSCI 773 S1T
2
COMPSCI 773• Individual assessment: (either or both)Individual assessment: (either or both)
• Oral interview (with Patrice)• Research paper presentation during final 3 lectures (time permitting)• Overall involvement in group work
• Assignment reports (what’s in):• A group report:
• Who did whatWho did what• What solution has been chosen and why
• An individual report• Detailing parts each student did• Presenting OWN solution and results (if any)
March 2, 2010 3COMPSCI 773 S1T
COMPSCI 773• The report should look like a research report with referencesThe report should look like a research report with references
– Justified explanations on the chosen solutions, graphs, resultsand – Critical assessment of the outcomes
• Programming• Windows C, C++ • OpenGl, CUDA, Gtkp• You are allowed to use external libraries but you have to state it• You may be asked to pass your code to other groups for the next
assignment• If you use another group’s codes you MUST state it in your report
March 2, 2010 4COMPSCI 773 S1T
3
The project: Advanced biometrics: 2D/3D mapping/face recognition
• Part 1:– Camera calibration– Image acquisition, camera synchronisation (GUI)– Calibration assessment– Rectification
• Part 2:– Improve calibration accuracy/rectification– Matching/stereo matching– Camera synchronisationCamera synchronisation– Database acquisition
• Part 3: Face authentication– Goal: Identify faces from images using 2/3D data– 2D/3D Statistical analysis– Live face recognition
• Each group will have to do BOTH partsMarch 2, 2010 5COMPSCI 773 S1T
What is available and what you will have to do
US: 3D USB 2 0 web-camerasUS: 3D USB 2.0 web cameras• Calibration object• 3D scanner• PCs-Windows
YOU: A calibration object (if you do not like ours)• Code in C, C++ (avoid Java) real-time• Create a proper GUI integrating the different parts of the project• Setup drivers if necessarySetup drivers if necessary• A very strong team/personal effort
OUTCOME• You will undertake work at the top-edge of today’s research• You will gain a unique experience of Applied Computer Vision
March 2, 2010 6COMPSCI 773 S1T
4
What is expectedProjects at the edge of research development: neither easy nor simpleProjects at the edge of research development: neither easy nor simple• A great deal of work is required but: (1) you will learn a lot; (2) this
could count as work experience in an expending IT area; (3) you can show what you are worth without having to fear too much about the final exam
• Still 2 students failed because of exam in 2005, none since but some ended up with B-/C+ final grade
• The exam will encompass all that will be lectured + project-related questions• It is not a good idea to concentrate on a very restricted part of the project as
this will penalise you by the end• If you like the projects you can continue towards the same directions for
COMPSCI 780 / MSc studies• If you think COMPSCI 773 is too hard for you we can offer a COMPSCI 780
project along the same directions
March 2, 2010 7COMPSCI 773 S1T
What you should know or learn very quickly
• C C++ programming• C, C++ programming• Confidence in mathematical skills (linear algebra)• Basics of Image Processing• To learn quickly (by yourself)• GUI design (just the basics)• A bit of CUDA for matching computation• Camera control and synchronization• The rest we will teach you…March 2, 2010 8COMPSCI 773 S1T
5
Course contents• Introductory lecture • Image matching
• Real-time image processing• Image matching
• Stereo image matching• 2D and 3D vision
geometry• Camera calibration• Stereo calibration
processing• Rectification of stereo
pairs• Colour discrimination • Features classification:
PCA-LDA• Segmenting binary images• Feature extraction• 3D scene description /
understanding
• Higher level statisticalapproaches
• Course overview and final demo
March 2, 2010 COMPSCI 773 S1T 9
Project details: Part 1• Acquire synchronised still streams from 3D webcams• Acquire synchronised still streams from 3D webcams
– Acquisition of face images• Calibrate cameras• Triangulate corresponding points to obtain their 3D
positions• Rectification• Demo
March 2, 2010 COMPSCI 773 S1T 10
6
Project details: Part 2• Improve calibration/rectification accuracy• Matching / stereo matching
– Implement a simple window-based correlation-based matching algorithm
• Acquire face database with 3D webcam• Extract faces from images• Normalize database images• Analyse 2D database images• Compare new faces versus faces within databasep• Create a GUI to interact with database• Perform a real-time demo
March 2, 2010 COMPSCI 773 S1T 11
Project details: Part 3• Matching / stereo matchingMatching / stereo matching
– Implement a complex matching algorithm• Analyse 3D database images• Compare new faces versus faces within database• Repeat with 2+3D data• Perform a real-time demo• Write a conference submission like report• Write a conference submission-like report
March 2, 2010 COMPSCI 773 S1T 12
7
A Possible Tentative FrameworkRGB.ppm or video stream
Morphological filtering
Hand colour segmentation
Clustering -labelling
Hand normalisation
PCAei ei+
1
... en
w1 w2 ... wnInput pose
e e e
Trained informationRepresentative weightings
March 2, 2010 COMPSCI 773 S1T 13
Classificationei ei+1 ... en
w11 w21 ... wn1
w12 w22 ... wn2
w13 w23 ... wn3
w14 w24 ... wn4
p g g
Class 1: OpenClass 2: GoClass 3: StopClass 4: Left
Mean image
Classified pose: Class 1
Questions ?• First thing firstFirst thing first
– Get access to the 773 open space-level 3 (one swipe card per group)– Create groups
• Next– I will do the first 3 weeks of the semester; Georgy - the 6 weeks following,
Patrice will do the last 3 weeks– I am very keen to help and answer questions:
• Better ask from Wednesday after lectures to Friday• Better ask from Wednesday after lectures to Friday• This is a research: I am keen to learn from you!• My advice: read research articles, respect assignments requirements,
but feel very welcome to explore alternative solutions
March 2, 2010 COMPSCI 773 S1T 14
Slide 14
CD1 celine, 23/02/2009
8
3D vision at CS3D vision at CS--UoAUoA
January 05, 2010 Intelligent Vision Systems: Dept. of Computer Science
15
People– Dr. P. Delmas, A/Prof. G. Gimel’farb– 6 PhDs and 4 MSc students– 2 postgraduate courses 773, 760
Di it l h t t & t ti l t• Digital photogrammetry & computational stereo– Camera calibration; forming epipolar stereo pairs– Fast 3D reconstruction: our own algorithms (faster, more
accurate)– symmetric dynamic programming stereo (SDPS);
symmetric concurrent stereo matching (SCSM)– Interactive editing of 3D models to form 3D surfaces – 3D face reconstruction and animation, 2/3D face
recognition– RT 3D vision (from acquisition to features then to fast
prototyping and modelling)
March 2, 2010 COMPSCI 773 S1T 16
9
3D vision Lab (city campus)• New Lab located on the 5th floor CS building
– Lab supervisor: PhD candidate A. GastelumF d d b CS F S t ff h f d FRST (300k NZ$)– Funded by CS, FoS staff research fund, FRST (300k NZ$)
• 3D acquisition capabilities:– 3D scanner (sub-mm accuracy on a 0.5m3 volume)– 3D stereo vision (active, passive, still or video up to 80 fps)
• Synchronised USB video cameras (80fps)• Synchronised Firewire video cameras (30fps)
– 3D display panel (Nov 09)November 30 2009• Projects
– 3D face animation (RT acquisition, 3d face features extraction, automatic prototyping, High res multi-layer SPH model)
– Real-time 3D face features– Stereo algorithms benchmarking– 3D characterisation of solids deformation
November 30, 2009
March 2, 2010 COMPSCI 773 S1T 17
Photogrammetry Laboratory(2)
• Optical ‘Range’.Integrated– Integrated.
– Multiple systems view a common scene.
• Stereo bench.– Sideways for face
capture!• Projector for structured
Example benchmark Data:
3D scanner
• Projector for structured lighting.
• Commercial 3D scanner.• Solutionix Rexcan 400.
Depth map Perspective visualisation
March 2, 2010 COMPSCI 773 S1T 18
10
3D Shape Reconstructionfrom stereo images (SDPS)
Binocular stereo
March 2, 2010 COMPSCI 773 S1T 19
2D face feature extraction (1996-2002)
(Geodesic) Active Contours for face feature extraction
C. Petitjean
With M. LievinMarch 2, 2010 COMPSCI 773 S1T 20
11
3D faceRendering using 2D images (2004)Idea: map 3d skin texture to CT-scan skull surfaceApplication: dental scanning
+3D model
3D face
2D image
Mapping program
March 2, 2010 COMPSCI 773 S1T 21
Automatic adjustment of the 3D generic model to patient features
AAM (active appearance models)AAM (active appearance models) • Set of points describing face features• Generic point location trained on a database of faces:
• Using Gabor jets (bunch of filters of varying amplitude, frequency and orientation) computed on manually selected face features
• Node points respective locationNode points respective location• Combines shape and texture in a PCA space• Searches a new image iteratively by using the texture error (pixel
wise) to drive the model parameters• Converge well if initialize close to solution
March 2, 2010 COMPSCI 773 S1T 22
12
AAM results (automatic extraction of face features)
Interface between Raw Data and Generic Model (2005)
2 modes: semi-automatic, automaticAutomatic: AAM on colour image, correspondences to 3D values on depth map, mapping of fiducial points
Model with
p p, pp g pto animated 3D model
animation system
Correspondences made and mapped via RBF with a final nearest point map and texture projection
Depth map
Results in a custom face with animation system in place
March 2, 2010 COMPSCI 773 S1T 24
13
Static face reconstruction Static face reconstruction evaluation (2004)evaluation (2004)
• Evaluated approaches:– Active/passive binocular stereo:
Active structured lightingActive structured lighting– Active photometric stereo
• 12 algorithms in total. Ground truth data: 3D scanner• Aims:
– Evaluate effectiveness• Accuracy, time complexity
– Database of faces.
25
• Alternative test set• Focus on stereo vision algorithms
– Determine best approach for dynamic 3D reconstruction system
March 2, 2010 COMPSCI 773 S1T 25
Images
Stereo pair Stereo pair withcolour code projection
March 2, 2010 COMPSCI 773 S1T 26
Ground truth
14
Statistical resultsStatistical resultsBP BP +
GradBP + Strip
DPM DPM + Grad
DPM + Strip
CM CM + Grad
CM + Strip
SAD SDPSGC SAD +Grad.
SAD +Strip
SDPS +Grad.
SDPS +Strip
GC + Grad.
GC + Strip
Grad. Strip Grad. StripGrad. Strip73% 77% 89% 88% 89% 92% 79% 84% 92%
77% 83% 92% 80% 85% 92% 89% 90% 93%
27Ground truth
Gray codeFCV Four Path Shapelet
77% 83% 92% 80% 85% 92% 89% 90% 93%
69% 54% 71% 97%
Results
March 2, 2010 COMPSCI 773 S1T 28
Depth map without structured lighting Depth map with structured lighting
Conclusion: SDPS + gray code is very close to ground truth• 3D face database is currently built. Pb: Processing data• Specific study on face feature accuracy
15
3D face animation using markers and stereovision (2006-2007)
• Stereo web-cameras perform marker based motion capture to obtain rigid head motion and non rigid capture to obtain rigid head motion and non-rigid facial expression motion.
• Acquired 3D points are then mapped onto a 3D face model with a virtual muscle animation system that can create face expressions.M l i ki ti d t l • Muscle inverse kinematics updates muscle contraction parameters based on marker motion to create a virtual character's performance.
March 2, 2010 COMPSCI 773 S1T 29
Marker based expression modelling Marker based expression modelling systemsystem
• Data driven:– Stereo web-cameras, face
kmarkers.– Head motion -rigid – Expressions - non-rigid.
• Tracked 3D points – 3D face model mapping.
Vi t l l i ti – Virtual muscle animation system
• Muscle inverse kinematics (IK) – Jacobian Transpose
March 2, 2010 COMPSCI 773 S1T 30
16
Video ResultsVideo Results
31
March 2, 2010 COMPSCI 773 S1T 31
More results
March 2, 2010 COMPSCI 773 S1T 32
17
Results
March 2, 2010 COMPSCI 773 S1T 33
Results-video
March 2, 2010 COMPSCI 773 S1T 34
18
Dynamic face reconstructionDynamic face reconstruction• Reconstruction at video rates.
– Design using static reconstruction conclusions:• ‘One shot’ active illumination + Symmetric Dynamic Programming One shot active illumination Symmetric Dynamic Programming
(SDPS) • Project pattern every other frame to get a clean texture.
Monochromatic Basler video cameras + colour web camera to obtain colour information.
March 2, 2010 COMPSCI 773 S1T 35
Active binocular stereoActive binocular stereo• Strip colour pattern: much higher accuracy
– SAD stereo algorithm: Depth map
• Pattern colour should avoid skin tones.
Strip pattern SAD - without pattern
SAD - with strip pattern
93%80%
March 2, 2010 COMPSCI 773 S1T 36
19
Video resultsVideo results
March 2, 2010 COMPSCI 773 S1T 37
Marker-based Emotion Modelling
March 2, 2010 COMPSCI 773 S1T 38
Muscle positions Face texture Underlying 3D mesh 17 muscles: Animation – tracking 3D points-markers
20
Happiness – easiest to reproduce
Surprise
March 2, 2010 COMPSCI 773 S1T 39
3d face features extraction2007-2010
• Segmentation of face features from 2D images has Segmentation of face features from 2D images has been researched extensively in the past
• 2D approaches are error-prone in presence of challenging illumination conditions
• 3D scanners emphasize depth accuracy over texture resolutiontexture resolution
• Recent research: Multimodal approaches outperform 2D and 3D techniques alone in face biometrics and recognition
March 2, 2010 COMPSCI 773 S1T 40
21
• Texture and curvature information can be used to segment lips from 3D scans
3D face data: texture and geometry
to segment lips from 3D scans3D Face
Texture Curvature
March 2, 2010 COMPSCI 773 S1T 41
Results (II): the lips (examples)
• The computed “anatomical” lip contour approximates the texture defined contour well.
March 2, 2010 COMPSCI 773 S1T 42
22
Results (III): the lips
• Method works on self acquired data (! more noise, less prominent ridges)
March 2, 2010 COMPSCI 773 S1T 43
Results (IV): the eyes
March 2, 2010 COMPSCI 773 S1T 44
23
Image Analysis in Soil Science(in collaboration with NZ, France, Mexico, Germany)
• Aim: (low cost) quantitative study of preferential flow Aim: (low cost) quantitative study of preferential flow using dye solution infiltration– Field and lab work: calibration and column images– Image processing for quantitative reconstruction of the
infiltration pattern – Relate concentration of dye to gray-level of images
Produce 3D maps of preferential flows
March 2, 2010 COMPSCI 773 S1T 45
– Produce 3D maps of preferential flows
C. Duwig, P. Delmas, K. Müller, B. Prado, K. Ren, H. Morin, A. Woodward: Quantifying fluorescent tracer distribution in allophanic soils to image solute transport. European Journal of Soil Science, 2007
Estimating Soil Properties Using Image Processing Techniques
Optical scan of soil core thin section
March 2, 2010 COMPSCI 773 S1T 46
Electronic microscopy of soil core thin section
Dyed Soil Core Section
24
Results
Image of a soil core cut vertically under black light illumination for 30 seconds Segmented image:
White: area with dyeBlack: area without dye
March 2, 2010 COMPSCI 773 S1T 47
Black: area without dyeNext steps:• Reconstruct volume of flow pathways for a soil core• Analyse variations over several soil cores• Model the soil flow pathways
Dye concentration and flow pathways for a given soil core cut
Image Analysis in Soil Science• 2D slice and 3D reconstruction2D slice and 3D reconstruction
March 2, 2010 COMPSCI 773 S1T 48
25
X-ray tomography
Brand : X-ray CT scanner RX solutionsSpatial resolution : 5 µm3 (specimen diameter 4mm)
to 220 µ m3 (specimen diameter 210 mm)
Experimental set up
to 220 µ m (specimen diameter 210 mm)X-ray generator: multi-energy (40 to 230kV) and
different spot size (from 7 to 50 µm)Detector: large field of view (195.07 x 243.84 mm2)
Type: X-ray flat panel detectorMatrix size: 1920-1536 pixels2Data size: 14bit-16384 grey levelsRotation stage: position and velocity controlled,
annular shape and high capacity (70 kg) Reconstruction by proprietary software RX-actExperiment conducted on an Andosol, depth 30cmSoil core dimension: Height 60 mm, diameter 55 mmDistance Source Detecteur : SDD = 752 mmDistance Source Objet : SOD = 347 mmPixel resolution: 58.6 µmAcquisition time: 45 minsMeaningful reconstructed slices: 1300
X-ray shot #240 out of 1000
COMPSCI 773 S1T 49
March 2, 2010 COMPSCI 773 S1T 50
26
We use the CT data to build a particle system
ResultsSPH modelling results
We defined 1500 particles to trace the fluid. In the image we show part of the solution, the blue particles are the trace particles, red and yellow particles are the solid particles. The yellow particles shows a selected set of cuts where we are obtaining the flux behaviour, we do the same procedure at different deeps.
for the soil, the soil particle system consist in 496789 particles.
March 2, 2010 COMPSCI 773 S1T 51
Medical Image Analysis(in collaboration with University of Louisville, USA)
• Global image registration using the learned prior appearance Global image registration using the learned prior appearance model (a Markov-Gibbs random field with multiple pair-wise cliques)
A. El-Baz, G. Gimel’farb: CVPR 2008, Anchorage, Alaska, USA
March 2, 2010 COMPSCI 773 S1T 52
Kidney image Gibbs energies for 5,000 neighbours76 out of 5,000 cliques in the learned prior
27
90.2% 62.6%Overlap between kidneys aligned with our and the mutual information (MI)based approaches
March 2, 2010 COMPSCI 773 S1T 53
Kidney: our registration vs. MI-based registration
based app oac es
96.8% 54.2%Overlap between lungs aligned with our and the mutual information (MI)based approaches
March 2, 2010 COMPSCI 773 S1T 54
Lungs: our registration vs. MI-based registration
based app oac es
28
Recent Publications on Medical Image Analysis
A. El-Baz, G. Gimel’farb, R. Falk, M. Abo El-Ghar, Automatic A. El Baz, G. Gimel farb, R. Falk, M. Abo El Ghar, Automatic Analysis of 3D Low Dose CT Images for Early Diagnosis of Lung Cancer, Pattern Recognition, vol. 42, no. 6, pp. 1041-1051, 2009
A. El-Baz, M. Casanova, G. Gimel’farb, M. Mott, A. Switala: An MRI-Based Diagnostic Framework for Early Diagnosis of Dyslexia, International Journal of Computer Assisted Radiology and Surgery vol 3 no 3 4 pp 181 189 September 2008
March 2, 2010 COMPSCI 773 S1T 55
Surgery, vol. 3, no. 3-4, pp. 181-189, September 2008A. El-Baz, G. Gimel’farb, Image Segmentation with a Parametric
Deformable Model using Shape and Appearance Priors, CVPR 2008, Anchorage, Alaska, June 23-28, 2008
Image Description by Primitive Elements(in collaboration with INRIA Sophia Antipolis, France)
Generalized multi-marked Generalized multi marked random point process– Library of marks
(primitive objects) for image description
– Limited inter-object interactions
March 2, 2010 COMPSCI 773 S1T 56
– Efficient jump-diffusion random processes for fast convergence
F.Lafarge, G.Gimel’farb, and X.Descombes: Geometric Feature Extraction by a Multi-Marked Point Process. IEEE Trans. Pattern Analysis and Machine Intelligence, 2009
29
March 2, 2010 COMPSCI 773 S1T 57
March 2, 2010 COMPSCI 773 S1T 58
30
Image Matching under Large Occlusions and Contrast Variations
• Conventional matching: errors for objects under large occlusions Conventional matching: errors for objects under large occlusions and non-uniform variations of contrast
• Symmetric correlation with soft masking of outliers
Actual template Best match
March 2, 2010 COMPSCI 773 S1T 59
Occluded template Soft masks of occlusions
Occluded template Soft mask Changed background Soft mask
• Non-uniform contrast transformations to account for changing illumination directions (e.g. from frontal 0o to oblique 75o)
March 2, 2010 COMPSCI 773 S1T 60
Template 0o Image 75o Template transformation by polynomial model by QP adaptation
31
Unsupervised Texture Segmentation (in collaboration with Dresden University of Technology, Germany) • Multi-component Markov-Gibbs model with analytic potential estimates
March 2, 2010 COMPSCI 773 S1T 6161
Initial collage Region map found Restored 1st texture Restored 2nd texture
Shape Prior Modelling(in collaboration with Dresden University of Technology, Germany)
• Markov-Gibbs “texture-like” model of geometric shapes with analytic potential estimatesp
March 2, 2010 COMPSCI 773 S1T 6262
Training set of 7-region shapes Simulated set of the shapes learned
32
MGRFs with Higher-order Cliques(in collaboration with University of Louisville, Louisville, Kentucky, USA)
March 2, 2010 COMPSCI 773 S1T 6363
Greyscale image “Zebra” Energy image: 2nd-order cliques
MGRFs with Higher-order Cliques(in collaboration with University of Louisville, Louisville, Kentucky, USA)
March 2, 2010 COMPSCI 773 S1T 64
Energy image: 2nd/3rd-order cliques 2nd/3rd/4th-order cliques
33
MGRFs with Higher-order Cliques(in collaboration with University of Louisville, Louisville, Kentucky, USA)
March 2, 2010 COMPSCI 773 S1T 65
Energy image: 2nd/3rd/4th/5th-order cliques
Texture Synthesis by Bunch Sampling
March 2, 2010 COMPSCI 773 S1T 66
34
Texture Synthesis by Bunch Sampling
March 2, 2010 COMPSCI 773 S1T 67
Brain MRI Segmentation(in collaboration with University of Louisville, Louisville, Kentucky, USA)
March 2, 2010 COMPSCI 773 S1T 68
MRI brain slice Multi-modal segmentation Ground truth provided by a
radiologist
35
Questions?