Lab 2

04.02.2016

Ressurser

• OpenCV documentation: – http://opencv.org/documentation.html

• Eigen documentation : – http://eigen.tuxfamily.org/dox/ – Quick reference quide: http://eigen.tuxfamily.org/dox/group__QuickRefPage.html

• C++:

– http://en.cppreference.com/w/

• Image Watch: An image debugger plug-in for Visual Studio – Download directly from Visual Studio:

Tools Extensions and Updates… Online Search for “Image Watch”

2

Litt om cv::Mat

3

Huskeliste

• Lage prosjekt

• Konstruere en cv::Mat – Datatyper, lage «vinduer» – MatCommaInitializer

• Hente og endre piksler

– at<…>() – forEach

• Regne på matriser

– Operasjoner, MatExpr, …

4

Perspektivkameramodellen

5

The perspective camera model

6

𝐶

Vehicle

𝑄

𝑊

𝑉

Perspective Camera

World

𝜉𝑉𝑊

𝒒𝐶

𝜉𝐶𝑉

Point observed by the camera

𝑧𝑊 𝑦𝑊

𝑥𝑊

𝑥𝑉

𝑦𝑉 𝑧𝑉

𝑧𝐶

𝑥𝐶 𝑦𝐶

𝑊 is a local NED coordinate system (North East Down)

𝐶 is a standard coordinate system for a camera 𝑥𝐶 - Right 𝑦𝐶 - Down 𝑧𝐶 - Forward

𝑉 is a standard coordinate system for a vehicle 𝑥𝑉 - Forward 𝑦𝑉 - Right 𝑧𝑉 - Down

The perspective camera model

• The point 𝑄 – Position 𝒒𝑊 = 27.3,−20.9,−11.5 𝑇

• The vehicle

– 3m wide, 6m long – The origin of 𝑉 is chosen to be at

the center of the vehicle, 1m above the ground

– Pose relative to the world: 𝑥 = 6.7𝑚 𝑦 = −2.4𝑚 𝑧 = −14.0𝑚 𝑟𝑟𝑟𝑟 = 3.7° 𝑝𝑝𝑝𝑝𝑝 = −9.3° 𝑝𝑒𝑒𝑒𝑝𝑒𝑒 = 307.6°

• The camera – The optical center is 2m in front of

and 1m to the left of the vehicles center

– The optical center is 4m above ground

– The y-axis of 𝐶 is perpendicular to the xy-plane of 𝑉

– The optical axis, i.e. the z-axis of 𝐶 , is rotated 4.7° to the right of the x-axis of 𝑉

– The camera calibration matrix is

𝐾 =1028 0 400

0 1028 3000 0 1

7

The perspective camera model

Problem In which pixel of the image will we observe the point 𝑄? Sub-problems 1. Represent 𝜉𝑉𝑊 as a SE(3) object 𝑇𝑉𝑊 2. Represent 𝜉𝐶𝑉 as a SE(3) object 𝑇𝐶𝑉 3. Represent 𝜉𝐶𝑊 as a SE(3) object 𝑇𝐶𝑊 4. Determine the camera matrix 𝑃 = 𝐾 𝑅 𝒕 5. Determine the pixel 𝒖 = 𝑢,𝑣 that the

point 𝑄 projects to according to the perspective camera model

8

Sub-problem 1

Represent 𝜉𝑉𝑊 as a SE(3) object 𝑇𝑉𝑊 • Sophus::SE3d objects can be initialized with a Eigen::Matrix3d rotation matrix 𝑅 and a

Eigen::Vector3d translation vector 𝒕 • Roll, pitch, heading relates to the zyx-rotation sequence, so 𝑅 = 𝑅𝑧𝑅𝑦𝑅𝑥 • A basic rotation matrix like 𝑅𝑥 𝜃 can be created by

Eigen::AngleAxisd(theta * M_PI / 180, Eigen::Vector3d::UnitX())

9

// Visualization of world and vehicle cv::Mat cv_t_W_V, cv_R_W_V; cv::eigen2cv(t_W_V, cv_t_W_V); cv::eigen2cv(R_W_V, cv_R_W_V); cv::Affine3d cv_T_W_V(cv_R_W_V, cv_t_W_V); cv::viz::Viz3d my_window("window 1"); my_window.showWidget("World-axes", cv::viz::WCoordinateSystem(3.0)); my_window.showWidget("vehicle-axes", cv::viz::WCoordinateSystem(3.0), cv_T_W_V); my_window.showWidget("vehicle", cv::viz::WCube(cv::Vec3d(-3.0, -1.5, -1), cv::Vec3d(3.0, 1.5, 1.0)), cv_T_W_V); my_window.spin();

Sub-problem 2 and 3

Represent 𝜉𝐶𝑉 as a SE(3) object 𝑇𝐶𝑉 • Which basic rotations must 𝑉 undergo in order to coincide with 𝐶 ? • Two basic rotations is enough Represent 𝜉𝐶𝑊 as a SE(3) object 𝑇𝐶𝑊 • Recall that 𝑇𝐶𝑊 = 𝑇𝑉𝑊 𝑇𝐶𝑉

10

// Visualization of camera frustum cv::Mat cv_t_W_C, cv_R_W_C, cv_K; cv::eigen2cv(K, cv_K); cv::eigen2cv(T_W_C.translation(), cv_t_W_C); cv::eigen2cv(T_W_C.rotationMatrix(), cv_R_W_C); cv::Affine3d cv_T_W_C(cv_R_W_C, cv_t_W_C); my_window.showWidget("camera_frustum", cv::viz::WCameraPosition(cv_K, 1.0, cv::viz::Color::red()), cv_T_W_C);

Sub-problem 4 and 5

Determine the camera matrix 𝑃 = 𝐾 𝑅 𝒕 • Recall that in the perspective camera model 𝑅 = 𝑅𝑊𝐶 and 𝑝 = 𝑝𝑊𝐶 , so we can not read 𝑅 and

𝒕 directly from 𝑇𝐶𝑊 Determine the pixel 𝒖 = 𝑢,𝑣 that the point 𝑄 projects to according to the perspective camera model • Recall that 𝒖� = 𝑃𝑿�

11

// Visualization of point (as a small sphere) my_window.showWidget("Q", cv::viz::WSphere({ Q_W(0), Q_W(1), Q_W(2) }, 0.1));

Laplace blending

12

Steg 1: Lag nytt prosjekt og vis frem bilder

• Kopier «opencv_project_template» og gi nytt navn – Husk å endre «PROJECT_NAME» i CMakeLists.txt

• Lag nytt prosjekt med Cmake

• Skriv et program som leser to bilder

– img_1: free_cat.jpg – img_2: free_tiger.jpg – Bildene bør konverteres til flyttallsbilder:

cv::imread(…).convertTo(img_1, CV_32F, 1.0/255.0);

• Vis bildene frem – cv::namedWindow() – cv::imshow()

13

Steg 2: Enkel blanding av bilder

• Samregistrer bildene ved å angi tre punktkorrespondanser

• Lag maske med rampe

14

cv::Point2f pts_1[] = {{321, 200}, {647, 200}, {476, 509}}; cv::Point2f pts_2[] = {{441, 726}, {780, 711}, {615, 1142}}; cv::Mat trans_mat = cv::getAffineTransform(pts_2, pts_1); cv::warpAffine(img_2, img_2, trans_mat, img_1.size());

cv::Mat mask = cv::Mat::zeros(img_1.size(), CV_32FC1); cv::rectangle(mask, cv::Rect{img_1.cols/2, 0, img_1.cols/2 + 1, img_1.rows}, 1, CV_FILLED); cv::blur(mask, mask, cv::Size{3, 3});

Steg 2: Enkel blanding av bilder

• Lag funksjon som gjør enkel blanding av to bilder vektet med masken

– Tips: cv::blendLinear()

• Bruk funksjonen og vis frem resultatet – Prøv med forskjellige masker

• Andre sømmer, sirkler, større glattefilter

15

cv::Mat linear_blend(cv::Mat& img_1, cv::Mat& img_2, cv::Mat& mask)

Steg 3: Laplaceblanding

• Lag funksjonen – cv::pyrDown()

16

std::vector<cv::Mat> construct_gaussian_pyramid(cv::Mat& img)

Steg 3: Laplaceblanding

• Lag funksjonen – Bruk construct_gaussian_pyramid – cv::pyrUp()

17

std::vector<cv::Mat> construct_laplacian_pyramid(cv::Mat& img)

Steg 3: Laplaceblanding

• Lag funksjonen – cv::pyrUp()

18

cv::Mat collapse_pyramid(std::vector<cv::Mat>& pyr)

Steg 3: Laplaceblanding

• Lag funksjonen – For eksempel slik:

19

cv::Mat laplace_blending(cv::Mat& img_1, cv::Mat& img_2, cv::Mat& mask)

std::vector<cv::Mat> mask_pyr = construct_gaussian_pyramid(mask); std::vector<cv::Mat> img_1_pyr = construct_laplacian_pyramid(img_1); std::vector<cv::Mat> img_2_pyr = construct_laplacian_pyramid(img_2); std::vector<cv::Mat> blend_pyr(img_1_pyr.size()); for (int i = 0; i < img_1_pyr.size(); i++) { // TODO: Perform linear blend on each level. } return collapse_pyramid(blend_pyr);

Steg 3: Laplaceblanding

• Bruk laplace_blending(img_1, img_2, mask) – Vis frem resultat – Sammenlign med enkel blending

20

Steg 4: Moroplukk

• Prøv andre bilder – Ta bilder med kameraet – Finn bilder på nett – Angi nye punkter for samregistrering

• Andre masker

– Last ned GIMP for å tegne finere masker

21

Steg 5: Dypdykk

• Implementer pyramiden selv – Ikke bruk cv::pyrDown() eller cv::pyrUp()

• Ta en titt på cv::seamlessClone()

• Prøv å implementere warpingen selv

22