docs: Add Humanloop as an observability provider #1

jamesbaskerville · 2025-03-11T13:38:00Z

Humanloop (https://humanloop.com) now supports ingesting OTEL traces from the Vercel AI SDK. This doc details the steps to setup this integration as well as configuration options.

andreibratu · 2025-03-12T17:34:04Z

content/providers/05-observability/humanloop.mdx

+
+# Humanloop Observability
+
+[Humanloop](https://humanloop.com/) is the LLM evals platform for enterprises, giving you the tools that top teams use to ship and scale AI with confidence. Humanloop integrates with the AI SDK to provide:


critique this final version

Humanloop is an enterprise LLMOps platform that helps you confidently evaluate, deploy, and scale AI features.

Our AI SDK integration allows you to seamlessly import telemetry data into Humanloop via the OpenTelemetry protocol.

You can visualize app traces and metrics for latency, cost, and errors. You can then set up automatic monitoring using code, human, and LLM evaluators.

andreibratu · 2025-03-12T17:34:28Z

content/providers/05-observability/humanloop.mdx

+
+[Humanloop](https://humanloop.com/) is the LLM evals platform for enterprises, giving you the tools that top teams use to ship and scale AI with confidence. Humanloop integrates with the AI SDK to provide:
+
+The AI SDK can log to [Humanloop](https://humanloop.com/) via OpenTelemetry. This integration enables trace visualization, cost/latency/error monitoring, and evaluation by code, LLM, or human judges.


remove this paragraph, rephrased version in previous comment

andreibratu · 2025-03-12T17:39:13Z

content/providers/05-observability/humanloop.mdx

+
+| Parameter             | Required | Description                                                                    |
+| --------------------- | -------- | ------------------------------------------------------------------------------ |
+| `humanloopPromptPath` | Yes      | Path to the prompt on Humanloop. Generation spans create Logs for this Prompt. |


Reminder that this is a first point of contact for new users: they're not yet in on the HL lingo

Alternative descriptions:

Prompt path in the Humanloop workspace. A Prompt file logs requests made to the LLM provider

Flow path in the Humanloop workspace. A Flow file holds traces of the full user-LLM interaction

Unique trace ID for the current user session

andreibratu · 2025-03-12T17:46:51Z

content/providers/05-observability/humanloop.mdx

+```ts
+experimental_telemetry: {
+  isEnabled: true,
+  functionId: 'unique-function-id', // Optional identifier for the function


Don't understand this comment; can you try describing why I would add a functionId/ what id does?

andreibratu · 2025-03-12T17:47:45Z

content/providers/05-observability/humanloop.mdx

+```bash
+OTEL_EXPORTER_OTLP_ENDPOINT=https://api.humanloop.com/v5/import/otel
+OTEL_EXPORTER_OTLP_PROTOCOL=http/json
+OTEL_EXPORTER_OTLP_HEADERS="X-API-KEY=xxxxxx" # Humanloop API key


Actually like the xxxxx pattern here; it's a good way to signal you need to add X-API-KEY= prefix

andreibratu · 2025-03-12T17:48:12Z

content/providers/05-observability/humanloop.mdx

+
+<Tabs items={['Next.js', 'Node.js']}>
+  <Tab>
+    Next.js has support for OpenTelemetry instrumentation on the framework level. Learn more about it in the [Next.js OpenTelemetry guide](https://nextjs.org/docs/app/building-your-application/optimizing/open-telemetry).


"Next.js has framework level support for OpenTelemetry instrumentation"

andreibratu · 2025-03-12T17:49:13Z

content/providers/05-observability/humanloop.mdx

+    }
+    ```
+
+    Your calls to the AI SDK should now be logged to Humanloop.


"Your AI SDK project will now log to Humanloop"

// More confidence, more salemanship - you are writing from a position of knowledge

andreibratu · 2025-03-12T17:50:02Z

content/providers/05-observability/humanloop.mdx

+
+    ### Node.js Implementation
+
+    OpenTelemetry has a package to auto-instrument Node.js applications. Learn more about it in the [OpenTelemetry Node.js guide](https://opentelemetry.io/docs/languages/js/getting-started/nodejs/).


"Add OpenTelemetry to your Node.js project. To learn more it, check out the [....."

andreibratu · 2025-03-12T17:50:45Z

content/providers/05-observability/humanloop.mdx

+OTEL_EXPORTER_OTLP_HEADERS="X-API-KEY=xxxxxx" # Humanloop API key
+```
+
+## Framework Implementation


OpenTelemetry Setup?

andreibratu · 2025-03-12T17:54:17Z

content/providers/05-observability/humanloop.mdx

+    OTEL_EXPORTER_OTLP_HEADERS="X-API-KEY=xxxxxx" # Humanloop API key
+    ```
+
+    Register the OpenTelemetry SDK and add Humanloop metadata to the spans. The `humanloopPromptPath` specifies the (Prompt File)[http://localhost:3001/docs/v5/explanation/prompts] in Humanloop to which the spans will be logged.


Oops, localhost link

andreibratu · 2025-03-12T17:55:15Z

content/providers/05-observability/humanloop.mdx

+const result = await generateText({
+  model: openai('gpt-4o'),
+  prompt: 'Write a short story about a cat.',
+  experimental_telemetry: { isEnabled: true },


want highlight on this line

andreibratu · 2025-03-12T17:57:43Z

content/providers/05-observability/humanloop.mdx

+
+## Debugging
+
+If you aren't using Next.js 15+, you will also need to enable the experimental instrumentation hook (available in 13.4+).


Need to elaborate this further: it's next.js only, should mention what the hook's benefits, THEN mentions you don't need extra configuration for >= 15

andreibratu · 2025-03-12T17:59:09Z

content/providers/05-observability/humanloop.mdx

+
+After instrumenting your AI SDK application with Humanloop, you can then:
+
+- Experiment with different [versions of Prompts](https://humanloop.com/docs/v5/guides/evals/comparing-prompts) and try them out in the Editor


"Try tweaking your Prompt in the workspace editor to improve its performance"

I see your point about provider: they can tweak the Prompt but not call it after optimisation. Let's mention that briefly in the same paragraph: "Our AI SDK Provider implementation is coming soon, allowing you to switch between Prompt versions as you make tweaks"

andreibratu · 2025-03-12T18:00:18Z

content/providers/05-observability/humanloop.mdx

+After instrumenting your AI SDK application with Humanloop, you can then:
+
+- Experiment with different [versions of Prompts](https://humanloop.com/docs/v5/guides/evals/comparing-prompts) and try them out in the Editor
+- Create [custom Evaluators](https://humanloop.com/docs/v5/explanation/evaluators) -- Human, Code, or LLM -- to monitor and benchmark your AI application


I would rather only point to monitoring - they're still dipping their toes and this is the next quantum of utility: BAM you have your AI Vercel project, now you also have monitoring in HL. That's a workable setup already and makes them come back for more

andreibratu · 2025-03-12T18:01:41Z

content/providers/05-observability/index.mdx

 Several LLM observability providers offer integrations with the AI SDK telemetry data:

 - [Braintrust](/providers/observability/braintrust)
+- [Humanloop](/providers/observability/humanloop)


Petition to put ourselves first /s

andreibratu · 2025-03-12T18:02:59Z

content/providers/05-observability/humanloop.mdx

+
+## Trace Grouping
+
+To group multiple AI SDK calls into a single Flow Log, create and pass a Flow Log ID to the telemetry metadata of each AI SDK call.


"To trace a user-LLM session end to end, we will use Humanloop's tracing feature: Flows (add linked here). Pass A Flow Log ID to the telemetra metadata on all AI SDK calls.

andreibratu · 2025-03-12T18:03:15Z

content/providers/05-observability/humanloop.mdx

+2. Pass the Flow Log ID to each AI SDK call
+3. Update the Flow Log when all executions are complete
+
+The Flow Log serves as a parent container for all related Prompt Logs in Humanloop.


drop this paragraph

andreibratu · 2025-03-12T18:03:55Z

content/providers/05-observability/humanloop.mdx

+
+1. Create a Flow Log in Humanloop
+2. Pass the Flow Log ID to each AI SDK call
+3. Update the Flow Log when all executions are complete


Confused, expected the flow trace to be completed automatically

andreibratu

did a first pass, ping me IRL

jamesbaskerville force-pushed the humanloop/add-observability-provider branch 2 times, most recently from 42abcba to ece140f Compare March 12, 2025 14:17

docs: Add Humanloop as an observability provider

acf9026

Humanloop (https://humanloop.com) now supports ingesting OTEL traces from the Vercel AI SDK. This doc details the steps to setup this integration as well as configuration options.

jamesbaskerville force-pushed the humanloop/add-observability-provider branch from ece140f to acf9026 Compare March 12, 2025 14:18

andreibratu reviewed Mar 12, 2025

View reviewed changes

andreibratu suggested changes Mar 12, 2025

View reviewed changes


		# Humanloop Observability

		[Humanloop](https://humanloop.com/) is the LLM evals platform for enterprises, giving you the tools that top teams use to ship and scale AI with confidence. Humanloop integrates with the AI SDK to provide:


		[Humanloop](https://humanloop.com/) is the LLM evals platform for enterprises, giving you the tools that top teams use to ship and scale AI with confidence. Humanloop integrates with the AI SDK to provide:

		The AI SDK can log to [Humanloop](https://humanloop.com/) via OpenTelemetry. This integration enables trace visualization, cost/latency/error monitoring, and evaluation by code, LLM, or human judges.


		### Node.js Implementation

		OpenTelemetry has a package to auto-instrument Node.js applications. Learn more about it in the [OpenTelemetry Node.js guide](https://opentelemetry.io/docs/languages/js/getting-started/nodejs/).


		## Debugging

		If you aren't using Next.js 15+, you will also need to enable the experimental instrumentation hook (available in 13.4+).


		After instrumenting your AI SDK application with Humanloop, you can then:

		- Experiment with different [versions of Prompts](https://humanloop.com/docs/v5/guides/evals/comparing-prompts) and try them out in the Editor


		## Trace Grouping

		To group multiple AI SDK calls into a single Flow Log, create and pass a Flow Log ID to the telemetry metadata of each AI SDK call.

docs: Add Humanloop as an observability provider #1

Are you sure you want to change the base?

docs: Add Humanloop as an observability provider #1

Uh oh!

Conversation

jamesbaskerville commented Mar 11, 2025

Uh oh!

andreibratu Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andreibratu Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andreibratu left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

andreibratu Mar 12, 2025 •

edited

Loading

andreibratu Mar 12, 2025 •

edited

Loading