Is there a way to configure an observability tool in Theia AI?

[jwzaden]

I am building a custom IDE that makes use of Theia AI. It would be helpful if we could easily integrate a tool like LangFuse to gain some insight into how our prompts are performing, but I haven't found simple way to do this. The docs for LangFuse indicate that we need to modify the code close to the prompt execution (LangFuse tracing). I would prefer to not modify the Theia AI code if possible.

Does Theia AI provide a means to integrate an observability tool?

[sdirix]

Hi @jwzaden,

Yes, you can integrate LangFuse. You don't necessarily need to modify Theia AI code, but of course some programming is necessary.

I had a quick look at the docs, it seems LangFuse offers two different ways of integration:

• Replacing the actual used SDK, e.g. wrapping the OpenAI SDK, or
• Manually managing tracing

If you want to go the first route, and for example want to wrap the Open AI SDK, then you need to rebind the OpenAiLanguageModelManager to your own version. In it you would create your own instances of the OpenAiModel which uses the LangFuse wrapped SDK instead of the original one.

For manual tracing, you need to decide on which level you want to trace. If you want to trace on a very low level, you could use a similar approach to the first one, just instead of using the wrapped SDK, you add the additional tracing calls. Alternatively you could go a level higher and integrate tracing in a custom version of the LanguageModelService. All requests and answers of LLMs pass through sendRequest which therefore is in a good position for such tracing. In fact, our own AI history view is collecting its data there. However this is one level up the raw SDK access, so this might be not specific enough for your requirements.

Edit: I want to mention it for completeness sake: There might be some webpack black magic you could do to replace the OpenAI SDK calls automatically with LangFuse wrapped ones, so you would not need to do any Theia customization at all. However this might be a bit brittle. It's certainly less explicit and a surprise in the code base.

[jwzaden]

Thank you for the detailed response! This is quite helpful.