Integrating Custom SurfaceTexture in Camera2 API with CameraKit SDK

[Reza-saeedi]

Hello Snapchat Team,
I am currently developing an Android camera application using the Camera2 API. My implementation involves creating a SurfaceTexture and a CaptureSession as shown below:

@Override
public void onOpened(CameraDevice camera) {
    SurfaceTexture surfaceTexture = new SurfaceTexture(10);
    surfaceTexture.setDefaultBufferSize(width, height);
    Surface surface = new Surface(surfaceTexture);
    try {
        camera.createCaptureSession(Arrays.asList(surface), new CaptureSessionCallback(), cameraThreadHandler);
    } catch (CameraAccessException e) {
        // Handle exception
    }
}

I am looking to integrate this setup with the CameraKit SDK. According to the SDK documentation and sample code, I understand that I should first detach the SurfaceTexture from any GL context before using it as an input for CameraKit. The relevant SDK sample for creating a session input appears as follows:

val input = inputFrom(
    surfaceTexture = inputSurfaceTexture,
    width = inputTextureSize.width,
    height = inputTextureSize.height,
    facingFront = true,
    rotationDegrees = 0,
    horizontalFieldOfView = DEFAULT_IMAGE_INPUT_FIELD_OF_VIEW,
    verticalFieldOfView = DEFAULT_IMAGE_INPUT_FIELD_OF_VIEW
)

The documentation notes the necessity of detaching the texture from any GL context to prevent conflicts:

// It is a must to always detach the texture from a GL context before connecting the texture as an input to CameraKit where the texture gets attached to an internal GL context.

For some reasons related to my application's design, I have my own GL context and need to ensure that CameraKit's outputs are compatible with it. Here are my questions:

How can I properly detach the GL context from the camera SurfaceTexture?
How can I either pass my own GL context as a sharedContext to CameraKit, or alternatively, obtain the CameraKit GL context to use as a sharedContext for managing this integration?
I appreciate any guidance or examples you could provide to help resolve this issue.

Thank you for your assistance.

[DipoSnap]

Hey @Reza-saeedi, Happy new year!
Detaching with surfaceTexture.detachFromGLContext() before passing the texture to camera kit should be all you need.
We currently do not support shared contexts (providing your own or obtaining Camera Kit's).

Could you share more on what your use case is?

[Reza-saeedi]

Hey @DipoSnap ,

Thank you for your response!

In my application, I am using my own GL context, which is shared with other SDKs to modify camera frames in a SurfaceTexture. These SDKs support shared contexts, allowing me to leverage their features without additional conversions.

I understand that calling surfaceTexture.detachFromGLContext() before passing the texture to Camera Kit is required. However, the issue with this approach is that detaching the texture stops camera frame updates in my SurfaceTexture. Since my application relies on real-time processing and rendering using my own GL context (which is shared with other SDKs), this disrupts the workflow.

Since Camera Kit does not support shared contexts, my concern is integrating it efficiently with my existing GL pipeline. A possible but inefficient approach would be to convert the frame to a Bitmap, process it in Camera Kit, and then convert the output back to my GL context. However, this adds unnecessary overhead and latency.

Is there any alternative way to achieve this integration more efficiently?
Would there be any workaround to pass or retrieve the GL context from Camera Kit to avoid performance bottlenecks?

I appreciate any suggestions you might have!

[fhdshsrhfg]

https://snp.is/ALDURAA

[Reza-saeedi]

Hi @DipoSnap , Just following up on this issue. I understand everyone is busy, but I wanted to kindly ask if there's any update or insight you could share regarding this integration challenge with Camera Kit and shared GL contexts.

This is currently blocking some parts of my real-time rendering pipeline, and I would really appreciate any guidance or suggestions on how to proceed more efficiently.

Thanks again for your time and support!