Confusing coordinate system.

[mcourteaux]

I'm trying to wrap my head around the coordinate system, and it looks unlike anything I've seen before. I setup a simple test transforms.json:

{
    "camera_angle_x": 2.1273956954128375,
    "scale": 1.0,
    "offset": [
        0.0,
        0.0,
        0.0
    ],
    "frames": [
        {
            "file_path": "png/view_f0000f",
            "transform_matrix": [
                [
                    1.0,
                    0.0,
                    0.0,
                    0.8
                ],
                [
                    0.0,
                    1.0,
                    0.0,
                    1.0
                ],
                [
                    0.0,
                    0.0,
                    1.0,
                    1.2
                ],
                [
                    0.0,
                    0.0,
                    0.0,
                    1.0
                ]
            ]
        }
    ]
}

Notice how the camera I set up is having identity rotational part, and translation vector [0.8, 1.0, 1.2]. The result looks like this:

Confusing things:

• The camera and the unit cube have different orientations for X, Y, and Z (i.e.: red, green, blue axis). (see first image)
• The position of the camera relative to the unit cube seems wrong. The the camera seems to be at x=1, and z=0.8, whereas I specified x to be 0.8, and z to be 1.2. (see second image).
• The camera is at y=1.2, whereas it should be 1.0. (see third image).

So, position in the testbed is (1, 1.2, 0.8), but it should be (0.8, 1.0, 1.2). This shuffles around ALL axis. After long searching, I found this snippet:

instant-ngp/include/neural-graphics-primitives/nerf_loader.h

Lines 50 to 70 in 409613a

	auto nerf_matrix_to_ngp(const Eigen::Matrix<float, 3, 4>& nerf_matrix) {
	Eigen::Matrix<float, 3, 4> result;
	int X=0,Y=1,Z=2;
	result.col(0) = Eigen::Vector3f{ nerf_matrix(X,0), nerf_matrix(Y,0), nerf_matrix(Z,0)};
	result.col(1) = Eigen::Vector3f{-nerf_matrix(X,1), -nerf_matrix(Y,1), -nerf_matrix(Z,1)};
	result.col(2) = Eigen::Vector3f{-nerf_matrix(X,2), -nerf_matrix(Y,2), -nerf_matrix(Z,2)};
	result.col(3) = Eigen::Vector3f{ nerf_matrix(X,3), nerf_matrix(Y,3), nerf_matrix(Z,3)} * scale + offset;
	if (from_mitsuba) {
	result.col(0) *= -1;
	result.col(2) *= -1;
	} else {
	// Cycle axes xyz->yzx
	Eigen::Vector4f tmp = result.row(0);
	result.row(0) = (Eigen::Vector4f)result.row(1);
	result.row(1) = (Eigen::Vector4f)result.row(2);
	result.row(2) = tmp;
	}
	return result;
	}

So, my question kinda reduces to: how to think about this coordinate system, as I still haven't figured out how to convert my dataset transformations to yours.

[mmalex]

sorry for the shift in coordinate system! internally, ngp uses entirely 0-1 bounding box, as displayed in the gui, with cameras looking down positive z. that's just the convention we chose early on. however, we wanted to be compatible specifically with the original nerf datasets, which place the origin at 0, scale the cameras to be around 3 units from the origin, and have a different convention for 'up', camera up, etc. so the snippet you found could be described as 'converting from the original nerf paper conventions to ngp's conventions'. over time, the 'original nerf' format of transforms.json became the dominant (only practical) way to get data into ngp; so, it comes across as confusing as we don't explicitly talk about the mapping. In the end, I am afraid it is what it is. you can adjust the scale and offset components of the transformation by adding extra parameters to the json, see nerf_loader.cu that looks for keys 'offset', 'scale' and 'aabb'. however the coordinate flipping is hard coded for the time being.

…

On Fri, Jan 21, 2022 at 5:07 PM Martijn Courteaux ***@***.***> wrote: I'm trying to wrap my head around the coordinate system, and it looks unlike anything I've seen before. I setup a simple test transforms.json: { "camera_angle_x": 2.1273956954128375, "scale": 1.0, "offset": [ 0.0, 0.0, 0.0 ], "frames": [ { "file_path": "png/view_f0000f", "transform_matrix": [ [ 1.0, 0.0, 0.0, 0.8 ], [ 0.0, 1.0, 0.0, 1.0 ], [ 0.0, 0.0, 1.0, 1.2 ], [ 0.0, 0.0, 0.0, 1.0 ] ] } ] } Notice how the camera I set up is having identity rotational part, and translation vector [0.8, 1.0, 1.2]. The result looks like this: [image: image] <https://user-images.githubusercontent.com/845012/150566804-e932cdc4-b977-4a0c-9d4f-3ecaae490d93.png> [image: image] <https://user-images.githubusercontent.com/845012/150567363-a5ecf787-111a-46ca-9db7-1b499c115acd.png> [image: image] <https://user-images.githubusercontent.com/845012/150567796-e4dec845-4aea-463e-b456-751db1adf200.png> Confusing things: - The camera and the unit cube have different orientations for X, Y, and Z (i.e.: red, green, blue axis). (see first image) - The position of the camera relative to the unit cube seems wrong. The the camera seems to be at x=1, and z=0.8, whereas I specified x to be 0.8, and z to be 1.2. (see second image). - The camera is at y=1.2, whereas it should be 1.0. (see third image). So, position in the testbed is (1, 1.2, 0.8), but it should be (0.8, 1.0, 1.2). This shuffles around ALL axis. After long searching, I found this snippet: https://github.com/NVlabs/instant-ngp/blob/409613afdc08f69342a9269b9e674604229d183f/include/neural-graphics-primitives/nerf_loader.h#L50-L70 So, my question kinda reduces to: how to think about this coordinate system, as I still haven't figured out how to convert my dataset transformations to yours. — Reply to this email directly, view it on GitHub <#72>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFOYTZYYMZ3QX7H22NUVDTUXGHEHANCNFSM5MQA7I5Q> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[mcourteaux]

Looking at the fox dataset, it seems that the -y is the camera up axis. I'm still puzzling, haha...

(a few minutes later...)

This was key!! The camera orientation was flipped (y-down). Adding that 180 degree rotation upon converting makes it work.

Now, still rotate everything into the normal upward rotation in your coordinate system.

I looked through the code, and my intuition is that the up setting doesn't do anything for nerf.

[mcourteaux]

For reference for others. This is what I have right now, based on a position and euler rotations for a Blender-like camera:

def generate_transform_matrix(pos, rot):
    def Rx(theta):
      return np.matrix([[ 1, 0            , 0            ],
                        [ 0, np.cos(theta),-np.sin(theta)],
                        [ 0, np.sin(theta), np.cos(theta)]])
    def Ry(theta):
      return np.matrix([[ np.cos(theta), 0, np.sin(theta)],
                        [ 0            , 1, 0            ],
                        [-np.sin(theta), 0, np.cos(theta)]])
    def Rz(theta):
      return np.matrix([[ np.cos(theta), -np.sin(theta), 0 ],
                        [ np.sin(theta), np.cos(theta) , 0 ],
                        [ 0            , 0             , 1 ]])

    R = Rz(rot[2]) * Ry(rot[1]) * Rx(rot[0])
    xf_rot = np.eye(4)
    xf_rot[:3,:3] = R

    xf_pos = np.eye(4)
    xf_pos[:3,3] = pos - average_position

    # barbershop_mirros_hd_dense:
    # - camera plane is y+z plane, meaning: constant x-values
    # - cameras look to +x

    # Don't ask me...
    extra_xf = np.matrix([
        [-1, 0, 0, 0],
        [ 0, 0, 1, 0],
        [ 0, 1, 0, 0],
        [ 0, 0, 0, 1]])
    # NerF will cycle forward, so lets cycle backward.
    shift_coords = np.matrix([
        [0, 0, 1, 0],
        [1, 0, 0, 0],
        [0, 1, 0, 0],
        [0, 0, 0, 1]])
    xf = shift_coords @ extra_xf @ xf_pos
    assert np.abs(np.linalg.det(xf) - 1.0) < 1e-4
    xf = xf @ xf_rot
    return xf

[mcourteaux]

Absolutely beautiful!!

nerf_barbershop_spherical_2.small.mp4

[mmalex]

thankyou for this! I am tempted to make a pullrequest for you that disables as much of the coordinate transformation as possible - dont worry the default wont change so your dataset will continue to work - I feel guilty about the trouble we caused :)

[mcourteaux]

I think I wouldn't be the only one that would appreciate a non-weirdly-behaving coordinate system 😋 I must admit, I got to this point by brute forcing transformation matrices and a little bit of iterative educated guessing. I'm confused as the camera upside-down thing we discussed earlier is not clearly in my transformation code anymore.

https://twitter.com/ID_AA_Carmack/status/432298265254449152

[jc211]

To remove transform nastiness from the nerf code, the only two areas I found that need to be changed are in nerf_loader.h

	auto nerf_matrix_to_ngp(const Eigen::Matrix<float, 3, 4>& nerf_matrix) {
		Eigen::Matrix<float, 3, 4> result;
		int X=0,Y=1,Z=2;
		result.col(0) = Eigen::Vector3f{ nerf_matrix(X,0),  nerf_matrix(Y,0),  nerf_matrix(Z,0)};
		result.col(1) = Eigen::Vector3f{ nerf_matrix(X,1),  nerf_matrix(Y,1),  nerf_matrix(Z,1)};
		result.col(2) = Eigen::Vector3f{ nerf_matrix(X,2),  nerf_matrix(Y,2),  nerf_matrix(Z,2)};
		result.col(3) = Eigen::Vector3f{ nerf_matrix(X,3),  nerf_matrix(Y,3),  nerf_matrix(Z,3)} * scale + offset;
		return result;
	}

	void nerf_ray_to_ngp(Ray& ray) {
		ray.o = ray.o * scale + offset;
	}

If you change this and compile, the transforms are expected to be in the opencv convention (and in this case the ngp convention) which is

Otherwise, the cameras in transforms.json are actually already in the blender convention (camera looking down -z and up is +y). You just have to make sure the intrinsic matrix is all positive.