Forums - 3D-Reconstruction

13 posts / 0 new
Last post
3D-Reconstruction
Christian_Geißler
Join Date: 23 Aug 11
Posts: 2
Posted: Fri, 2011-11-04 04:13

Hello,

ive got a question regarding the 3D-Reconstruction Module of FastCV:

Description of my problem: I have a couple of corresponding 3D->2D points and I want to calculate the OpenGL-Pose (4x4 Modelviewmatrix). I also have the ProjectionMatrix. How exactly can I do that? Im a little confused by thoose functions, cause they only evaluate and optimize a 3x4 Matrix, but there doesn't seem to be a function to actually calculate such a Matrix / Pose.

 

  • Up0
  • Down0
Prasun_Choudhury
Join Date: 3 Dec 10
Posts: 18
Posted: Mon, 2011-11-07 10:41

One can create the OpenGL modelview matrix by using the fastCV function fcvGeomPoseRefineGNf32(.). The 2D->3D point information is given by the input parameter 'corrs', initpose describes the initial pose and refinedpose will be the updated pose. Both initpose and refinedpose are 'float' vectors of size 12 and contains the rotation (R) and translation (T) information. R is a 3 x 3 matrix ([r1, r2, r3; r4, r5, r6; r7, r8, r9]) and T is a 3 x 1 vector ([t1; t2; t3]). Now the rotation and translation information in initpose and refined pose is ordered in the following manner: [r1; r2; r3; t1; r4; r5; r6; t2; r7; r8; r9; t3], where both initpose and refined pose are 12 x 1 vectors.

Now, to create the modelview matrix (dimension: 4 x 4) in OpenGL, the 12 x1 pose vector has to be assigned to the first 3 rows of modelview matrix and the last row of the modelview matrix will be [0, 0, 0, 1]. So, overall the modelview matrix will look like: M = [r1, r2, r3, t1; r4, r5, r6, t2; r7, r8, r9, t3; 0, 0, 0, 1]. The modelview matrix in OpenGL is of dimension 4 x 4.

The OpenGL projection matrix is independent from the pose we calculate in fastCV. The projection matrix is used for rendering only and it describes the camera frustum, viewing direction, etc. The scene contents within the viewing frustum will only be rendered on the screen and everything outside the viewing frustum will be clipped away. So, one needs to choose the viewing frustum paramters accordingly; else, contents of the scene that one would like to render will be alipped away.

  • Up0
  • Down0
Lennart_Brueggemann
Join Date: 20 Sep 11
Posts: 17
Posted: Tue, 2011-11-08 06:14

Thanks, that was very helpful. What should go into the 'indices' field in the 'corrs' struct? Since we haven't done any refinement to that point, and we want to incorporate all correspondences we got so far, I supposed it should have the same length as the corrs.from array and just increment by one in every element.

So far it works, but the returned refinedpose is exactly the same as initpose.

  • Up0
  • Down0
Prasun_Choudhury
Join Date: 3 Dec 10
Posts: 18
Posted: Tue, 2011-11-08 11:01

Yes, you are correct. If you want all the points to be used for pose estimation, then the size of indices should be the # of points and the index of each point in your array should be just incremented by 1. In other words, for N points, the indices array will be from 0 to N - 1.

One way to do a sanity check for correct usage of the function is to have an identical set of to and from points and then call the pose estimation function. In this case, the refined pose should be same as 'initpose'.

  • Up0
  • Down0
Lennart_Brueggemann
Join Date: 20 Sep 11
Posts: 17
Posted: Mon, 2011-11-14 02:26

Are the initial and refined poses the poses of the camera or the object one wants to track? We probably mixed this together here.

 

Doing a sanity check by using the same from and to values results in the same, both poses are identical. What information gives us the return value of fcvGeomPoseRefineGNf32 (reprojection error)? It varies between 270 and 310 in our case, when we find a lot of correspondences.

  • Up0
  • Down0
Prasun_Choudhury
Join Date: 3 Dec 10
Posts: 18
Posted: Mon, 2011-11-14 20:43

The initial and refined pose are the poses of the camera that we are tracking. Lets say the current pose of camera is 'pose1' at frame1. We want to find 'pose2', i.e. the pose of the camera at frame2. For a generic case, we will have a set of corresponding points in frame1 and frame2 and the refined pose will update the camera pose such that the points in frame1 will be mapped to points in frame2. Now, if frame1 and frame2 are identical (i.e. the camera didn't move at all), we will have identical set of points in frame1 and frame2; hence the refined pose will be identical to the initial pose.

I would have expected the reprojection error to be near zero; the reprojection error is result of fcvGeomPoseEvaluateErrorf32 being called based on the calculated refined pose. So, if refined pose is same as initial pose and the set of points are the same in both the frames, theoretically reprojection error should be 0.

  • Up0
  • Down0
Lennart_Brueggemann
Join Date: 20 Sep 11
Posts: 17
Posted: Tue, 2011-11-15 07:23

Thanks, it *seems* to work now. The error with initpose being the same as refinedpose was most probably due to a faulty pointer reference in the arguments for fcvGeomPoseRefineGNf32.

  • Up0
  • Down0
Lennart_Brueggemann
Join Date: 20 Sep 11
Posts: 17
Posted: Thu, 2011-12-01 07:44

Does the refined camera pose need any calculations to use them further (such as sine/cosine)? E.g, we're giving an identity matrix to the pose calculator, and get back a matrix with relatively big values (6.xx to 9.xx). Is it in the same "format" as the initial pose matrix?

  • Up0
  • Down0
Prasun_Choudhury
Join Date: 3 Dec 10
Posts: 18
Posted: Thu, 2011-12-01 18:46

The refinedPose is in the same format as the initialPose and shouldn't need any further calculations like sine/cosine/etc. How are your set of corresponding points? Have the points moved significantly between the two frames? In that case, the solution won't be very reliable. The function is more meant for incremental motion & pose updates.

  • Up0
  • Down0
Lennart_Brueggemann
Join Date: 20 Sep 11
Posts: 17
Posted: Fri, 2011-12-02 03:51

The points shouldn't be that far off. Since we're currently testing our implementation we use very similar (randomly generated) correspondences (maybe 10 to 20 pixels in screenspace) and get a reprojection error of 1-10 which seems to be okay. The "from" values for the corrs struct are worldspace coordinates, the "to" values screenspace coordinates, is that correct?

Basically, this is the data we use:

from: 3D worldspace coordinates (in tuples of three)

to: 2D Screenspace coordinate (in tuples of two)

indices: Array from 0 to the number of our correspondences

fromStride/toStride: both 0 as suggested in the API docs

numIndices: Length of the indices array

numCorrespondences: Length of the from array

initPose: The rotation matrix of our camera plus its translation vector combined in one float array in the format [r1; r2; r3; t1; r4; r5; r6; t2; r7; r8; r9; t3]

 refinedPose: We extract the rotation values and set them as the rotation matrix of our camera, and the translation values as the position of our camera.

 

Our native method call looks like this:

fcvGeomPoseRefineGNf32(&correspondenceData, 2, 5, 50, (float32_t*) &initPose, pRefinedPose);

The values for min/maxIterations and stopCriteria are more or less arbitrary choices.

  • Up0
  • Down0
Prasun_Choudhury
Join Date: 3 Dec 10
Posts: 18
Posted: Mon, 2011-12-05 10:25

I think your usage of the function is correct but I see that there is a mistake in FastCV documentation. The 'fromStride' and 'toStride' shouldn't be set to 0 as mentioned in the API docs. If the 'from' array is the set of 3D points, 'fromStride' should be set to 3 and for 2D points in the 'to' array, 'toStride' should be set to 2. Can you try with this setting: fromStride = 3; toStride = 2 and everything else in your implementation remains the same.

  • Up0
  • Down0
Lennart_Brueggemann
Join Date: 20 Sep 11
Posts: 17
Posted: Wed, 2011-12-07 04:18

Hm, then thats fine either. Another thing that came to my mind is that our 3D-Engine uses a different coordinate system (rotated by 180°, so positive y goes down and positive z goes into the screen). Does FastCV normally use a "normal" (positive y is up...) coordinate system and we need to convert the matrices, or does that not matter?

  • Up0
  • Down0
Prasun_Choudhury
Join Date: 3 Dec 10
Posts: 18
Posted: Wed, 2011-12-07 17:58

Yes, the coordinate system will matter if you want to fill up some transformation matrices in graphics APIs or rendering libraries. For example, in OpenGL one needs to set up the modelview matrix prior to rendering and the convention of FastCV is same as OpenGL's axes convention. It follows the typical right handed coordinate system with the origin at the center of the frame: X points to right, Y points up and Z is out of the board.

Now, if you use some other graphics API like DirectX or some of your own rendering library, you will need to adjust the transofrmation matrices to render correctly. As an example, in DirectX, Y points down and the origin is at the upper left corner. So, to be consistent with DirectX, the rotation matrix as output from fastCV has to be rotated around X axis by 180 degrees prior to rendering. Similarly, the translation is also required to align the origin to the upper left corner. From your description, I feel that you are using the coordinate system similar to DirectX (though I am not sure where your origin is) and you need to do the above transformation.

 

  • Up0
  • Down0
or Register

Opinions expressed in the content posted here are the personal opinions of the original authors, and do not necessarily reflect those of Qualcomm Incorporated or its subsidiaries (“Qualcomm”). The content is provided for informational purposes only and is not meant to be an endorsement or representation by Qualcomm or any other party. This site may also provide links or references to non-Qualcomm sites and resources. Qualcomm makes no representations, warranties, or other commitments whatsoever about any non-Qualcomm sites or third-party resources that may be referenced, accessible from, or linked to this site.