Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
research:hflow:start [2015/08/12 13:15]
aneufeld
research:hflow:start [2021/03/02 13:28] (current)
Line 1: Line 1:
-====== ​Monocular 3D Reconstruction of Traffic Scenes ​from Optical Flow ======+====== ​Estimating Vehicle Ego-Motion and Piecewise Planar Scene Structure ​from Optical Flow in a Continuous Framework ​====== 
 + 
 +{{:​research:​hflow:​neufeld_gcpr2015.pdf|GCPR 2015 slides}}
  
 We estimate a 3D scene structure and the vehicle egomotion given the forward and backward optical flow fields between two consecutive frames. ​ We estimate a 3D scene structure and the vehicle egomotion given the forward and backward optical flow fields between two consecutive frames. ​
Line 17: Line 19:
 Given rotation $R \in \text{SO}(3)$,​ translation $t \in S(2)$ and 3D point $X$, the point $X'$ relative to the second camera is given by $X' = R^\top (X - t)$. The translation is restricted to the unit sphere, since the translation norm cannot be estimated without additional information. Given rotation $R \in \text{SO}(3)$,​ translation $t \in S(2)$ and 3D point $X$, the point $X'$ relative to the second camera is given by $X' = R^\top (X - t)$. The translation is restricted to the unit sphere, since the translation norm cannot be estimated without additional information.
 Let $\pi$ denote the projection onto the image plane, Let $\pi$ denote the projection onto the image plane,
-$$ \pi\begin{pmatrix} x_1 \\ x_2 \\ x_3 \end{pmatrix} = \frac{1}{x_3} \begin{pmatrix} x_1 \\ x_2 \end{pmatrix}. $$+$$ \pi\begin{pmatrix} x_1 \\ x_2 \\ x_3 \end{pmatrix} = \frac{1}{x_3} \begin{pmatrix} x_1 \\ x_2 \\ x_3 \end{pmatrix}. $$
 Let $z(x)$ denote the depth of pixel $x$ in the image plane, $X = z(x) x$. Depth can be calculated from the plane parameters, $z(x,v) = (v^\top x)^{-1}$. Let $z(x)$ denote the depth of pixel $x$ in the image plane, $X = z(x) x$. Depth can be calculated from the plane parameters, $z(x,v) = (v^\top x)^{-1}$.
 Optical flow is given by Optical flow is given by
Line 54: Line 56:
  
 The proposed energy function is non-convex in rotation $R \in \text{SO}(3)$,​ $t \in S(2)$ and planes $v_i \in \mathbb{R}^3$. We choose the Levenberg-Marquardt algorithm for optimization. Note that the energy can be written in form The proposed energy function is non-convex in rotation $R \in \text{SO}(3)$,​ $t \in S(2)$ and planes $v_i \in \mathbb{R}^3$. We choose the Levenberg-Marquardt algorithm for optimization. Note that the energy can be written in form
-$$E(R,t,v) = \sum_j \left( f_j(R,t,v) \right)^2 = \lVert f(R,t,v) \rVert_2^2.$$+$$E(R,t,v) = \sum_j \left( f_j(R,t,v) \right)^2 = \lVert f(R,t,v) \rVert_2^2,$$ 
 +which can be optimized using Levenberg-Marquardt. We apply a damping factor, which is updated after each iteration.
  
 ===== Results ===== ===== Results =====
  
-We demonstrate the output of our algorithms on a few examples here, the paper contains a quantitative evaluation. From top to bottom, each example shows the reference frame, estimated depth and estimated ​normals. Depth is encoded as follows, the scale is derived from stereo ​data.+We demonstrate the output of our algorithms on a few examples here, the paper contains a quantitative evaluation. From top to bottom, each example shows the reference frame, estimated depth and normals ​and the flow error as measured in the paper (click for a full size view). Depth and depth error (in pixels) are encoded as follows, the scale is derived from ground truth data.
  
-{{ :​research:​hflow:​depth.png?​nolink&​200 |}}+{{ :​research:​hflow:​depth2.png?​nolink&​200 |}}
  
 The normal is color-encoded based on the approximately equidistant LAB colorspace. The next image shows the encoding after applying the Hammer-Aitov projection to a sphere and a simple ray-traced scene. The normal is color-encoded based on the approximately equidistant LAB colorspace. The next image shows the encoding after applying the Hammer-Aitov projection to a sphere and a simple ray-traced scene.
Line 66: Line 69:
 {{ :​research:​hflow:​planes.png?​nolink&​500 |}} {{ :​research:​hflow:​planes.png?​nolink&​500 |}}
  
 +The following examples show the first 30 KITTI stereo/flow training sequences.
  
 | {{ :​research:​hflow:​000000_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000000.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000000.png?​direct&​200 |}} {{ :​research:​hflow:​err_000000.png?​direct&​200 |}} | {{ :​research:​hflow:​000001_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000001.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000001.png?​direct&​200 |}} {{ :​research:​hflow:​err_000001.png?​direct&​200 |}} | {{ :​research:​hflow:​000002_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000002.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000002.png?​direct&​200 |}} {{ :​research:​hflow:​err_000002.png?​direct&​200 |}} | | {{ :​research:​hflow:​000000_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000000.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000000.png?​direct&​200 |}} {{ :​research:​hflow:​err_000000.png?​direct&​200 |}} | {{ :​research:​hflow:​000001_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000001.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000001.png?​direct&​200 |}} {{ :​research:​hflow:​err_000001.png?​direct&​200 |}} | {{ :​research:​hflow:​000002_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000002.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000002.png?​direct&​200 |}} {{ :​research:​hflow:​err_000002.png?​direct&​200 |}} |
-| {{ :​research:​hflow:​000009_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000009.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000009.png?​direct&​200 |}} {{ :​research:​hflow:​err_000009.png?​direct&​200 |}} | {{ :​research:​hflow:​000019_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000019.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000019.png?​direct&​200 |}} {{ :​research:​hflow:​err_000019.png?​direct&​200 |}} | {{ :​research:​hflow:​000023_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000023.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000023.png?​direct&​200 |}} {{ :​research:​hflow:​err_000023.png?​direct&​200 |}} | +| ||| 
-| {{ :​research:​hflow:​000024_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000024.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000024.png?​direct&​200 |}} {{ :​research:​hflow:​err_000024.png?​direct&​200 |}} |   |+| {{ :​research:​hflow:​000003_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000003.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000003.png?​direct&​200 |}} {{ :​research:​hflow:​err_000003.png?​direct&​200 |}} | {{ :​research:​hflow:​000004_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000004.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000004.png?​direct&​200 |}} {{ :​research:​hflow:​err_000004.png?​direct&​200 |}} | {{ :​research:​hflow:​000005_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000005.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000005.png?​direct&​200 |}} {{ :​research:​hflow:​err_000005.png?​direct&​200 |}} | 
 +| ||| 
 +| {{ :​research:​hflow:​000006_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000006.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000006.png?​direct&​200 |}} {{ :​research:​hflow:​err_000006.png?​direct&​200 |}} | {{ :​research:​hflow:​000007_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000007.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000007.png?​direct&​200 |}} {{ :​research:​hflow:​err_000007.png?​direct&​200 |}} | {{ :​research:​hflow:​000008_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000008.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000008.png?​direct&​200 |}} {{ :​research:​hflow:​err_000008.png?​direct&​200 |}} | 
 +| ||| 
 +| {{ :​research:​hflow:​000009_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000009.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000009.png?​direct&​200 |}} {{ :​research:​hflow:​err_000009.png?​direct&​200 |}} | {{ :​research:​hflow:​000010_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000010.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000010.png?​direct&​200 |}} {{ :​research:​hflow:​err_000010.png?​direct&​200 |}} | {{ :​research:​hflow:​000011_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000011.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000011.png?​direct&​200 |}} {{ :​research:​hflow:​err_000011.png?​direct&​200 |}} | 
 +| ||| 
 +| {{ :​research:​hflow:​000012_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000012.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000012.png?​direct&​200 |}} {{ :​research:​hflow:​err_000012.png?​direct&​200 |}} | {{ :​research:​hflow:​000013_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000013.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000013.png?​direct&​200 |}} {{ :​research:​hflow:​err_000013.png?​direct&​200 |}} | {{ :​research:​hflow:​000014_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000014.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000014.png?​direct&​200 |}} {{ :​research:​hflow:​err_000014.png?​direct&​200 |}} | 
 +| ||| 
 +| {{ :​research:​hflow:​000015_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000015.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000015.png?​direct&​200 |}} {{ :​research:​hflow:​err_000015.png?​direct&​200 |}} | {{ :​research:​hflow:​000016_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000016.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000016.png?​direct&​200 |}} {{ :​research:​hflow:​err_000016.png?​direct&​200 |}} | {{ :​research:​hflow:​000017_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000017.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000017.png?​direct&​200 |}} {{ :​research:​hflow:​err_000017.png?​direct&​200 |}} | 
 +| ||| 
 +| {{ :​research:​hflow:​000018_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000018.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000018.png?​direct&​200 |}} {{ :​research:​hflow:​err_000018.png?​direct&​200 |}} | {{ :​research:​hflow:​000019_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000019.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000019.png?​direct&​200 |}} {{ :​research:​hflow:​err_000019.png?​direct&​200 |}} | {{ :​research:​hflow:​000020_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000020.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000020.png?​direct&​200 |}} {{ :​research:​hflow:​err_000020.png?​direct&​200 |}} | 
 +| ||| 
 +| {{ :​research:​hflow:​000021_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000021.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000021.png?​direct&​200 |}} {{ :​research:​hflow:​err_000021.png?​direct&​200 |}} | {{ :​research:​hflow:​000022_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000022.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000022.png?​direct&​200 |}} {{ :​research:​hflow:​err_000022.png?​direct&​200 |}} | {{ :​research:​hflow:​000023_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000023.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000023.png?​direct&​200 |}} {{ :​research:​hflow:​err_000023.png?​direct&​200 |}} 
 +| ||
 +| {{ :​research:​hflow:​000024_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000024.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000024.png?​direct&​200 |}} {{ :​research:​hflow:​err_000024.png?​direct&​200 |}} | {{ :​research:​hflow:​000025_10.png?​direct&​200 ​|}} {{ :​research:​hflow:​depth_000025.png?​direct&​200 ​|}} {{ :​research:​hflow:​normals_lab_000025.png?​direct&​200 |}} {{ :​research:​hflow:​err_000025.png?​direct&​200 |}} | {{ :​research:​hflow:​000026_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000026.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000026.png?​direct&​200 |}} {{ :​research:​hflow:​err_000026.png?​direct&​200 |}} | 
 +| ||| 
 +| {{ :​research:​hflow:​000027_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000027.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000027.png?​direct&​200 |}} {{ :​research:​hflow:​err_000027.png?​direct&​200 |}} | {{ :​research:​hflow:​000028_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000028.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000028.png?​direct&​200 |}} {{ :​research:​hflow:​err_000028.png?​direct&​200 |}} | {{ :​research:​hflow:​000029_10.png?​direct&​200 |}} {{ :​research:​hflow:​depth_000029.png?​direct&​200 |}} {{ :​research:​hflow:​normals_lab_000029.png?​direct&​200 |}} {{ :​research:​hflow:​err_000029.png?​direct&​200 |}} | 
  
 ===== References ===== ===== References =====