Transformation Matrix From Custom Data

[SergioMOrozco]

I am using a Pybullet simulation environment to generate a sequence of images along with camera poses as input to Instant-Ngp in the form of the transformations.json file; however, I cannot figure out how convert between the Pybullet coordinate system and the Instant-Ngp coordinate system.

The pybullet camera that I am using uses the following coordinate system:

• X = LEFT
• Y = UP
• Z = FORWARD

In my mind, this should be a conversion between the two camera coordinate systems, where the Instant Ngp coordinate system uses:

• X = RIGHT
• Y = DOWN
• Z = FORWARD

Therefore, something like this should suffice:

tmp_x = transformation_matrix[0,0:4].copy()
tmp_y = transformation_matrix[1,0:4].copy()
tmp_z = transformation_matrix[2,0:4].copy()

transformation_matrix[0,0:4] = -tmp_x
transformation_matrix[1,0:4] = -tmp_y
transformation_matrix[2,0:4] = tmp_z

Admittedly, my understanding of linear algebra is comically terrible. If somebody could at least guide me in the right direction, I would really appreciate it. I've read through every open/closed issue that discusses the weird coordinate system that Instant Ngp uses, but I haven't found any answers to be satisfactory.

[SergioMOrozco]

UPDATE:

So I managed to make some progress. In my pybullet simulation environment, I have the following coordinate system:

Which I interpret as:

• X = Forward
• Y = Left
• Z = Up

And I have the following coordinate system in Instant Ngp in the camera frame:

Which I interpret as (kind of hard to tell but the Z is pointing straight out the camera):
X = Right
Y = Down
Z = Forward

Therefore the conversion should be (X,Y,Z)->(Z,-X,-Y).

During the simulation, I capture multiple images and their corresponding transformation matrices. Here is a quick video that demonstrates the motion:

Screencast.from.05-26-2023.12.15.02.PM.webm

So, it goes from left to right in a semi-circular motion. I am able to replicate this motion like so:

But the rotation is completely off. It seems odd to me that the translation would be correct, but the rotation wouldn't be. Here is the matrix conversion code I used to get the above image:

tmp_x = transformation_matrix[0,0:4].copy()
tmp_y = transformation_matrix[1,0:4].copy()
tmp_z = transformation_matrix[2,0:4].copy()

transformation_matrix[0,0:4] = tmp_z
transformation_matrix[1,0:4] = -tmp_x
transformation_matrix[2,0:4] = -tmp_y

I don't know why this is so hard to do, but I'm struggling pretty hard with this.

[Student204161]

We might be struggling with what boils down to the same problem.
#1364 (comment)

My theory is that the problem is the camera poses found using colmap2nerf.py are flipped/rotated but that they get further transformed when inputted into instant-ngp... I haven't been able to find out what the conversion is & can't seem to find the answer anywhere. Am not well versed in c++ code, so I am not confident about finding an answr by reading through the .cu files...

edit: there are functions in nerf_loader.h for transforming back & forth from ngp and nerf coords

[SergioMOrozco]

@Student204161 I actually found the issue in my code, which was a simple matrix inversion

np.linalg.inv(transformation_matrix)

along with some scaling. You can take a look at my whole github repo. Hopefully it helps https://github.com/FezTheImmigrant/pybullet_playground. The file worth looking into is kuka.py

[Student204161]

Thanks for the reply - I'm glad you worked out your problem, I fortunately solved my problem some days ago... ended up spending almost a month on it but I'm just glad it works now.

I had some code errors also, but in the end I was able convert the raw volumes into the ngp coordinate system, from the nerf coordinate system (* and so I was able to do the reprojection)

Have a good day

[tiago1405]

Hi @SergioMOrozco and @Student204161 I hate to reopen a long closed thread but could you share what your solutions were? I am currently banging my head against the wall trying to transform from pybullet from [X = forward, Y = Left, Z = Up] to COLMAP/OpenCV [X = right, Y = down, Z = forward] but my while my placements are correct, my angles are all wrong. The cameras seem like they are improperly rotated about the X-Axis by around 180 degrees.
I'll leave a relevant snippet below.

def write_colmap_format(cam_intrinsics, camera_pos, target_pos, image_file, output_dir, image_id, client_id=0, verbose:bool = False):
    """Write camera parameters in COLMAP format using PyBullet camera functions"""
    os.makedirs(output_dir, exist_ok=True)
    
    # Get view matrix using PyBullet function
    view_matrix = pyb.computeViewMatrix(
        cameraEyePosition=camera_pos,
        cameraTargetPosition=target_pos,
        cameraUpVector=[0, 0, 1],
        physicsClientId=client_id
    )

    # Convert from PyBullet 4x4 matrix (16-element list) to numpy 3x3 rotation and translation
    pyb_rotation = np.array(view_matrix).reshape(4, 4)[:3, :3]  # Extract rotation component
    # view_matrix = correction @ view_matrix
    conversion = np.array([
        # PyBullet => COLMAP
        [0, -1, 0], # -Y => X
        [0, 0, -1], # -Z => Y
        [1, 0, 0]   # X => Z
    ])
    # Transform position and rotation to COLMAP convention
    camera_pos_colmap = conversion @ camera_pos
    
    # Convert to camera-to-world transform
    R_colmap = pyb_rotation @ conversion
    # Transpose for camera-to-world
    R_c2w = R_colmap.T
    t = -R_c2w @ camera_pos_colmap

    q = rotation_matrix_to_quaternion(R_c2w)

def rotation_matrix_to_quaternion(R):
    """ Convert rotation matrix to quaternion in Hamilton convention"""
    trace = np.trace(R)
    if trace > 0:
        S = np.sqrt(trace + 1.0) * 2
        w = 0.25 * S
        x = (R[2,1] - R[1,2]) / S
        y = (R[0,2] - R[2,0]) / S
        z = (R[1,0] - R[0,1]) / S
    elif R[0,0] > R[1,1] and R[0,0] > R[2,2]:
        S = np.sqrt(1.0 + R[0,0] - R[1,1] - R[2,2]) * 2
        w = (R[2,1] - R[1,2]) / S
        x = 0.25 * S
        y = (R[0,1] + R[1,0]) / S
        z = (R[0,2] + R[2,0]) / S
    elif R[1,1] > R[2,2]:
        S = np.sqrt(1.0 + R[1,1] - R[0,0] - R[2,2]) * 2
        w = (R[0,2] - R[2,0]) / S
        x = (R[0,1] + R[1,0]) / S
        y = 0.25 * S
        z = (R[1,2] + R[2,1]) / S
    else:
        S = np.sqrt(1.0 + R[2,2] - R[0,0] - R[1,1]) * 2
        w = (R[1,0] - R[0,1]) / S
        x = (R[0,2] + R[2,0]) / S
        y = (R[1,2] + R[2,1]) / S
        z = 0.25 * S
    return np.array([w, x, y, z])

Appreciate any help you two can offer!