CSCI 473/573
Human-Centered Robotics
September 9, 2015
Reading Assignment 1 Due on Friday
2
Stereo Depth Sensing
• Correspondence
• Reconstruction
3
Stereo Depth Sensing
Lambertian surface reflects light with equal
intensity in all directions.
4
How Kinect works?
•
•
•
•
Work at 30 Hz with resolution of 640 x 480
Viewing angle: 43°vertical by 57°horizontal field of view
Range limit for tracking: 0.7 – 6 m (2.3’ to 20’)
Practical range limit: 1.2 – 3.5 m
5
Depth Sensing using Kinect
Projected Pattern!
• Must be able to encode depth
• Must be different between each other
6
Depth Sensing using Kinect
Projected Pattern!
• Must be different between each other
• Must be able to encode depth
7
Depth Sensing using Kinect
Projected Pattern!
• Must be different between each other
-- Structured Light
• Must be able to encode depth
8
Correspondence using Structured Light
Structured light general principle:
Project a known pattern onto the scene
and infer depth from the deformation of
that pattern
9
http://www.sci.utah.edu/~gerig/CS6320-S2012/Materials/CS6320-CV-S2012-StructuredLight.pdf
Correspondence using Structured Light
Recall of stereo vision:
• Hard to solve the correspondence problem
• Slow
10
Correspondence using Structured Light
Early color-encoded structured light:
• Need very few images (one or two)
• But need a more complex correspondence algorithm
11
http://www.sci.utah.edu/~gerig/CS6320-S2012/Materials/CS6320-CV-S2012-StructuredLight.pdf
Correspondence using Structured Light
Structured light general principle:
12
Correspondence using Structured Light
Structured light general principle:
• Kinect uses infrared laser light with a speckle pattern.
• Color is not used for computing depth.
Shpunt et al, PrimeSense patent application, US 2008/0106746
13
Correspondence using Structured Light
Structured light general principle:
• Kinect uses infrared laser light with a speckle pattern.
• Color is not used for computing depth.
Unfortunately,
we do not
know the
design detail.
Shpunt et al, PrimeSense patent application, US 2008/0106746
14
Correspondence using Structured Light
Structured light general principle:
• The distinctive points make matching easy.
Shpunt et al, PrimeSense patent application, US 2008/0106746
15
Correspondence using Structured Light
Region-growing Random Dot Matching
1. Detect dots (“speckles”) and label them unknown
2. Randomly select a region anchor (a dot with unknown
depth)
a.
Windowed search via normalized cross correlation along scanline
Check that best match score is greater than threshold; if not, mark as “invalid” and
go to 2
b.
Region growing
1.
2.
3.
Neighboring pixels are added to a queue
For each pixel in queue, initialize by anchor’s shift; then search small local
neighborhood; if matched, add neighbors to queue
Stop when no pixels are left in the queue
3. Repeat until all dots are matched or are marked “invalid”
http://www.wipo.int/patentscope/search/en/WO2007043036
Depth Sensing using Kinect
Projected Pattern!
• Must be different between each other
-- Structured Light
• Must be able to encode depth
-- Depth from focus
17
Depth from focus
Principle:
Stuff that is more blurry is further away
18
Depth from focus
Principle:
Stuff that is more blurry is further away
19
Depth from focus
Principle:
Stuff that is more blurry is further away
• The Kinect dramatically improves the accuracy
of traditional depth from focus
• The Kinect uses a special (“astigmatic”) lens
with different focal length in x- and y- directions
• A projected circle then becomes an ellipse
whose orientation depends on depth
20
Depth from focus
Principle:
Stuff that is more blurry is further away
Freedman et al, PrimeSense patent application,
US 2010/0290698
21
Example: Book vs. No Book
Source: http://www.futurepicture.org/?p=97
Example: Book vs. No Book
Source: http://www.futurepicture.org/?p=97
Depth from focus
Principle:
Stuff that is more blurry is further away
24
Depth from focus
25
Depth Sensing using Kinect
Projected Pattern!
• Must be different between each other
-- Structured Light
• Must be able to encode depth
-- Depth from focus
-- Depth from stereo
26
Depth from Stereo
• Same stereo depth sensing algorithms
directly apply
27
Depth from Stereo
• But it solves the correspondence issues
and image distortion (to some extent).
Projector
Sensor
28
Depth Sensing using Structured Light
The Kinect combines
-- structured light
with two classic computer vision
techniques:
-- depth from focus
-- depth from stereo
The KEY is the pattern design!
29
30