[processor/transform] Create `With*Functions` factory options to provide custom OTTL funcs programatically. #39966

franciscovalentecastro · 2025-05-09T10:40:29Z

Description

Create With*Functions factory options to provide custom OTTL functions for logs, metrics or traces to the resulting processor. Also created the "exported" Default*Functions functions that can be used to provide a "subset" of the default functions used in the the transform processor.

Since in the current collector design the Config doesn't have access to the Factory, the Config.Validate method can't access the additional OTTL functions set in the NewFactory. I have found two ways to circumvent this :

[Option 1, Current] Add un-exported fields of the form *Functions to the Config struct to use in the Validate() method.
[Option 2] Create ValidateFunctions methods that can be used to create a custom Config.Validate().

Some details about the solution :

To create a custom transform processor the user needs to create a NewFactory providing the custom OTTL functions.
This is a result of the discussion in this thread : [processor/transform] Create Additional*Funcs config properties to provide additional OTTL funcs programatically. #39641 (comment)
This is intended for developers of Opentelemetry Collector Distributions.

Usage

The factory options can be used like the following example :

NewFactoryWithOptions(
	WithLogFunctions(transformprocessor.DefaultLogFunctions()),
	WithLogFunctions([]ottl.Factory[ottllog.TransformContext]{FunctionOne, FunctionTwo, ...}),
	WithLogFunctions([]ottl.Factory[ottllog.TransformContext]{FunctionThree, FunctionFour, ...}),
)

Link to tracking issue

[processor/transform] Register programatically custom OTTL funcs in transform processor. #39698

Note : There is a similar followup proposed update for the processor/filter.

Testing

Added unit tests for the modified methods and functions.

Documentation

Added comments to the new With*Functions, Default*Functions and FactoryWithOptions functions.

processor/transformprocessor/factory.go

processor/transformprocessor/README.md

processor/transformprocessor/config.go

processor/transformprocessor/factory.go

franciscovalentecastro · 2025-05-16T15:38:15Z

Following up on the review request CC @edmocosta @TylerHelmuth @evan-bradley. This PR implements the approach discussed in #39641 (comment).

evan-bradley

Thanks for taking this on, and sorry for the slow review.

Could we adjust the approach slightly?

Make LogFunctions, SpanFunctions, etc. exported at the top level of the transform processor module.
Instead of supplying additional functions, can we make it so the user specifies all functions they would like to include?

So the invocation would look like this:

NewFactory(
	WithSpanFunctions(
		transformprocessor.DefaultSpanFunctions(),
		[]ottl.Factory[ottlspan.TransformContext]{FunctionOne, FunctionTwo, ...},
		[]ottl.Factory[ottlspan.TransformContext]{FunctionThree, FunctionFour, ...},
	),
)

I think this has two advantages:

Treats all the functions the same. The OTTL stdlib functions aren't anything special, so we shouldn't segment them in the code.
Allows the component author to make it so only the functions they want are enabled in the transform processor. This could be useful if, for example, a distro author wants to remove MD5 for security reasons or standardize all statements to use the pattern replacement functions instead of the match functions.

edmocosta · 2025-05-19T11:24:13Z

Thanks for taking this on, and sorry for the slow review.

Could we adjust the approach slightly?

Make LogFunctions, SpanFunctions, etc. exported at the top level of the transform processor module.

Instead of supplying additional functions, can we make it so the user specifies all functions they would like to include?

So the invocation would look like this:
NewFactory(
	WithSpanFunctions(
		transformprocessor.DefaultSpanFunctions(),
		[]ottl.Factory[ottlspan.TransformContext]{FunctionOne, FunctionTwo, ...},
		[]ottl.Factory[ottlspan.TransformContext]{FunctionThree, FunctionFour, ...},
	),
)
I think this has two advantages:

Treats all the functions the same. The OTTL stdlib functions aren't anything special, so we shouldn't segment them in the code.

Allows the component author to make it so only the functions they want are enabled in the transform processor. This could be useful if, for example, a distro author wants to remove MD5 for security reasons or standardize all statements to use the pattern replacement functions instead of the match functions.

I agree with @evan-bradley approach, and in addition to that I'd:

Move the pkg/ottlprocessor back into the transform processor package, I don't think we should have such shared options at the OTTL package, which already exports all necessary utilities to delegate this kind of logic to components (which might differ from each other).
Add the following logs:
- When custom functions are being used, so we know we're not dealing with the standard's transform processor behavior and extra functions might exists.
- When an OTTL standard function is overridden. We can't support custom functions overrides upstream, and that let users and community know that it's a custom function override, and that its behavior might be different from the standard version.
- When custom functions with duplicate names are configured (should we allow it at all?)

quentinmit · 2025-05-20T20:22:30Z

Treats all the functions the same. The OTTL stdlib functions aren't anything special, so we shouldn't segment them in the code.

Allows the component author to make it so only the functions they want are enabled in the transform processor. This could be useful if, for example, a distro author wants to remove MD5 for security reasons or standardize all statements to use the pattern replacement functions instead of the match functions.

I thought about doing this, but I considered that we are likely to have other ways for users to add OTTL functions in the future (e.g. an "OTTL function" extension type, or functions loaded from config with wasm), and if we have this option pass a complete function list, it feels like that would have unclear semantics if combined with user-defined functions.

I agree with @evan-bradley approach, and in addition to that I'd:

Move the pkg/ottlprocessor back into the transform processor package, I don't think we should have such shared options at the OTTL package, which already exports all necessary utilities to delegate this kind of logic to components (which might differ from each other).

Maybe a compromise to avoid duplicate code would be to put the options in an internal package and reexpose them in transformprocessor and filterprocessor? I do think there's value in having a consistent API that can be used by any component with user-configured OTTL expressions.

Add the following logs:

When custom functions are being used, so we know we're not dealing with the standard's transform processor behavior and extra functions might exists.

When an OTTL standard function is overridden. We can't support custom functions overrides upstream, and that let users and community know that it's a custom function override, and that its behavior might be different from the standard version.

When custom functions with duplicate names are configured (should we allow it at all?)

How would you combine this with @evan-bradley's request to pass in the complete function list? At that point every function would be "custom".

edmocosta · 2025-05-21T11:32:54Z

Maybe a compromise to avoid duplicate code would be to put the options in an internal package and reexpose them in transformprocessor and filterprocessor? I do think there's value in having a consistent API that can be used by any component with user-configured OTTL expressions.

Yes, I agree we should aim for consistent APIs, but I still don't think it should be enforced/defined by the OTTL package. It already provides the parsers/utilities that components need to use the language, but how they do that, is up to them.

There are many different use-cases for OTTL at the moment (processor, connectors...), and it would be tricky to come up with some common options that perfectly fit all components. For example, providing the WithAdditionalSpanEventFunctions option for a processor that only works with logs wouldn't make much sense.

How would you combine this with @evan-bradley's request to pass in the complete function list? At that point every function would be "custom".

That's a good question, I wish we could find a good solution for that. If I had to choose between both, I would rather start limiting it to additional functions than not having those logs.

Another option would be keeping functions segmented (WithAdditional*Funcs), and later - if needed - add another factory option to customize the standard functions in a controlled way.

evan-bradley · 2025-05-22T19:28:08Z

I considered that we are likely to have other ways for users to add OTTL functions in the future (e.g. an "OTTL function" extension type, or functions loaded from config with wasm)

@quentinmit That's a good consideration. Personally, I don't want to plan too far in advance for that since those ideas may not end up happening or may end up taking a different form than we'd expect. Additionally, I would personally be okay with WithSpanFunctions for standard Go functions, and e.g. WithExtensionFunctions and WithWASMFunctions for things that handle those two situations, for example. I think it's likely possible that we won't necessarily segment the functions by signal at that layer since the interfaces involved with supplying functions to the transform processor would probably be in a better position to separate functions by signal. This is a digression since these are both still a long ways out, but I think this illustrates why it's difficult to plan for what these might look like. I think we'll be able to make sure the API feels clean once we get to that point.

There are many different use-cases for OTTL at the moment (processor, connectors...), and it would be tricky to come up with some common options that perfectly fit all components. For example, providing the WithAdditionalSpanEventFunctions option for a processor that only works with logs wouldn't make much sense.

@edmocosta I think we could likely define these in a common way eventually, but for now I agree it's okay for these options to live in the transform processor. We can separate them into a different package down the road once we have a better understanding of how they're used.

That's a good question, I wish we could find a good solution for that. If I had to choose between both, I would rather start limiting it to additional functions than not having those logs.

Another option would be keeping functions segmented (WithAdditional*Funcs), and later - if needed - add another factory option to customize the standard functions in a controlled way.

I think we could simply emit a log if any of the With*Funcs are passed to NewFactory: at the point the user is using these, they're at least intending to operate outside what's supported upstream. I would like to resist printing anything other than a debug log until we see that this is an issue though. I see the value in alerting the user that they should be mindful that there are customizations, but emitting a log every time the Collector starts up feels too noisy unless it's necessary.

Overall I don't think segmenting the core/user-defined functions in the With*Funcs options would result in significantly worse UX, but I feel like it would be a little cleaner if we treat all functions the same way.

edmocosta · 2025-05-22T21:57:09Z

I think we could simply emit a log if any of the With*Funcs are passed to NewFactory: at the point the user is using these, they're at least intending to operate outside what's supported upstream. I would like to resist printing anything other than a debug log until we see that this is an issue though. I see the value in alerting the user that they should be mindful that there are customizations, but emitting a log every time the Collector starts up feels too noisy unless it's necessary.

Agreed, those extra logs that I mentioned would be useful for troubleshooting, so I think they should be debug logs. Custom collector distributions using this new API probably wouldn't want them to be printed in any other level as well.

franciscovalentecastro · 2025-05-23T16:02:24Z

I implemented all the last details and comments on this PR. This PR is ready for another review. The With*Functions can be used as described before :

NewFactoryWithOptions(
	WithLogFunctions(
		transformprocessor.DefaultLogFunctions(),
		[]ottl.Factory[ottllog.TransformContext]{FunctionOne, FunctionTwo, ...},
		[]ottl.Factory[ottllog.TransformContext]{FunctionThree, FunctionFour, ...},
	),
)

A summary of the updates :

Created With*Functions instead of WithAdditional*Functions.
Implemented a "Debug" level logs in the "create processor" step when the "default" functions are overriden with
With*Functions. There is one log for each "logs, metrics or traces" processors.
Created functions.go and functions_test.go to define "exported" Default*Functions and helper functions.
Modified the signature of the NewProcessor functions to receive the defined set of functions and updated tests accordingly.
Edit : Now the FactoryOption are added to the NewFactoryWithOptions signature instead of NewFactory.

CC @edmocosta @evan-bradley @quentinmit

evan-bradley · 2025-05-23T18:55:15Z

processor/transformprocessor/factory.go

+// WithDataPointFunctions set ottl datapoint functions in resulting processor.
+func WithDataPointFunctions(dataPointFunctions ...[]ottl.Factory[ottldatapoint.TransformContext]) FactoryOption {
+	return func(factory *transformProcessorFactory) {
+		factory.dataPointFunctions = createFunctionsMap(dataPointFunctions)


Can we have it so the functions are added to the existing map instead of overriding it? I think the UX will be better if the user can call the With*Functions options multiple times so they can easily add functions from different sources without manually merging them.

Can we have it so the functions are added to the existing map instead of overriding it?

Yes, that can work too. I updated the FactoryOptions in my last commit cc848b6 .

The new behaviour overrides the "default" functions on its first use of any With*Functions and every subsequent use merges the new functions to the "current" functions map.

Before Last Update

For comparison, this is the previous way of setting functions.

NewFactoryWithOptions( WithLogFunctions( transformprocessor.DefaultLogFunctions(), []ottl.Factory[ottllog.TransformContext]{FunctionOne, FunctionTwo, ...}, []ottl.Factory[ottllog.TransformContext]{FunctionThree, FunctionFour, ...}, ), )

Now functions can be set like this :

After Last Update

NewFactoryWithOptions( WithLogFunctions(transformprocessor.DefaultLogFunctions()), WithLogFunctions([]ottl.Factory[ottllog.TransformContext]{FunctionOne, FunctionTwo, ...}), WithLogFunctions([]ottl.Factory[ottllog.TransformContext]{FunctionThree, FunctionFour, ...}), )

My bad, I missed the variadic argument in the With*Functions functions. I think both of these are roughly equivalent, though I still have a slight preference for the second pattern outlined in your comment.

@edmocosta @TylerHelmuth Please let us know if you feel differently about how this should look.

processor/transformprocessor/factory.go

evan-bradley

Thanks for your persistence through this review process @franciscovalentecastro. This looks good to me, just one request for a few additional test cases.

processor/transformprocessor/factory_test.go

evan-bradley · 2025-06-10T13:03:54Z

@edmocosta @TylerHelmuth Please take a look when you are able.

@quentinmit your review would be welcome here as well.

processor/transformprocessor/factory.go

… from `create*Processor` when setting custom functions.

… of `WithAdditional*Functions`. Create debug log.

… from `make checkapi`.

…ssert unexported properties of config.

…gar`.

… multiple times.

Co-authored-by: Edmo Vamerlatti Costa <[email protected]>

…ctions log messages and fix typo.

…tered non-default functions.

evan-bradley · 2025-06-16T17:33:51Z

@edmocosta could you give this another look when you are able?

franciscovalentecastro · 2025-06-20T15:20:23Z

@evan-bradley @edmocosta @quentinmit Thank you 🙏 for all the reviews, comments and design guidance throughout this FR / PR !

…ide custom OTTL funcs programatically. (open-telemetry#39966) Co-authored-by: Edmo Vamerlatti Costa <[email protected]> Co-authored-by: Evan Bradley <[email protected]>

github-actions bot added the processor/transform Transform processor label May 9, 2025

github-actions bot requested review from TylerHelmuth, edmocosta and evan-bradley May 9, 2025 10:40

This was referenced May 9, 2025

[processor/transform] Create Additional*Funcs config properties to provide additional OTTL funcs programatically. #39641

Closed

[processor/transform] Register programatically custom OTTL funcs in transform processor. #39698

Closed

quentinmit reviewed May 9, 2025

View reviewed changes

github-actions bot added the pkg/ottl label May 12, 2025

franciscovalentecastro marked this pull request as ready for review May 12, 2025 22:36

franciscovalentecastro requested a review from a team as a code owner May 12, 2025 22:36

github-actions bot assigned edmocosta May 12, 2025

evan-bradley reviewed May 18, 2025

View reviewed changes

franciscovalentecastro force-pushed the fcovalente-with-add-ottl-funcs branch from 6271568 to 5338205 Compare May 21, 2025 20:42

franciscovalentecastro changed the title ~~[processor/transform] Create WithAdditional*Funcs factory options to provide additional OTTL funcs programatically.~~ [processor/transform] Create With*Funcs factory options to provide custom OTTL funcs programatically. May 23, 2025

franciscovalentecastro changed the title ~~[processor/transform] Create With*Funcs factory options to provide custom OTTL funcs programatically.~~ [processor/transform] Create With*Functions factory options to provide custom OTTL funcs programatically. May 23, 2025

franciscovalentecastro force-pushed the fcovalente-with-add-ottl-funcs branch from 22c2d08 to 7779541 Compare May 26, 2025 17:51

evan-bradley reviewed Jun 4, 2025

View reviewed changes

franciscovalentecastro force-pushed the fcovalente-with-add-ottl-funcs branch from 151f746 to cc848b6 Compare June 4, 2025 16:56

evan-bradley reviewed Jun 10, 2025

View reviewed changes

processor/transformprocessor/factory_test.go Show resolved Hide resolved

edmocosta reviewed Jun 10, 2025

View reviewed changes

processor/transformprocessor/factory.go Outdated Show resolved Hide resolved

processor/transformprocessor/factory.go Outdated Show resolved Hide resolved

processor/transformprocessor/factory.go Outdated Show resolved Hide resolved

franciscovalentecastro and others added 17 commits June 12, 2025 14:40

[processor/transform] Refactor WithAdditional*Functions to get logs…

928adb3

… from `create*Processor` when setting custom functions.

[processor/transform] Revert changes on internal/functions.go.

10a92f2

[processor/transform] Create With*Functions factory options instead…

b66edc7

… of `WithAdditional*Functions`. Create debug log.

[processor/test] Use variadic arguments in With*Functions.

9bc00fb

[processor/transform] Add CHANGELOG entry.

cea8fb6

[processor/transform] Fix lint error.

706fb13

[processor/transform] Create NewFactoryWithOptions to avoid failure…

5b368de

… from `make checkapi`.

[processor/transform] Add assertConfigContainsDefaultFunctions to a…

29b456d

…ssert unexported properties of config.

[processor/transform] Update non-default functions log and remove `Su…

30fce1b

…gar`.

[processor/transform] Update With*Functions to make possible to use…

c8101cf

… multiple times.

[processor/transform] Fix unnecessary update.

a2eeed7

[processor/transform] Add api to change_logs.

a755aa1

Co-authored-by: Edmo Vamerlatti Costa <[email protected]>

[processor/transform] Update With*Functions comments, non-default fun…

6e101a0

…ctions log messages and fix typo.

[process/transform] Update test to add cases without default functions.

664778a

[processor/transform] Update to uppercase OTTL.

d4af5a1

[processor/transform] Add log with fields to know which context regis…

7d3c29c

…tered non-default functions.

[process/transform] Nit, remove s from traces.

825d3e4

franciscovalentecastro force-pushed the fcovalente-with-add-ottl-funcs branch from 6d4f449 to 825d3e4 Compare June 12, 2025 14:40

Merge branch 'main' into fcovalente-with-add-ottl-funcs

5e2a5ad

evan-bradley approved these changes Jun 16, 2025

View reviewed changes

edmocosta approved these changes Jun 18, 2025

View reviewed changes

Merge branch 'main' into fcovalente-with-add-ottl-funcs

9836ba1

evan-bradley merged commit 7513025 into open-telemetry:main Jun 20, 2025
177 checks passed

github-actions bot added this to the next release milestone Jun 20, 2025

This was referenced Jun 25, 2025

[processor/filter] Create With*Functions factory options to register custom OTTL funcs programatically. #40933

Merged

[processor/filter] Register programatically custom OTTL funcs in the filter processor. #40948

Closed

franciscovalentecastro mentioned this pull request Sep 8, 2025

REQUEST: New membership for franciscovalentecastro open-telemetry/community#2976

Closed

6 tasks

Uh oh!

[processor/transform] Create With*Functions factory options to provide custom OTTL funcs programatically. #39966

[processor/transform] Create With*Functions factory options to provide custom OTTL funcs programatically. #39966

Uh oh!

Conversation

franciscovalentecastro commented May 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Usage

Link to tracking issue

Testing

Documentation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

franciscovalentecastro commented May 16, 2025

Uh oh!

evan-bradley left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

edmocosta commented May 19, 2025

Uh oh!

quentinmit commented May 20, 2025

Uh oh!

edmocosta commented May 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

evan-bradley commented May 22, 2025

Uh oh!

edmocosta commented May 22, 2025

Uh oh!

franciscovalentecastro commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

evan-bradley May 23, 2025

Choose a reason for hiding this comment

Uh oh!

franciscovalentecastro Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Before Last Update

After Last Update

Uh oh!

evan-bradley Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

evan-bradley left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

evan-bradley commented Jun 10, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

evan-bradley commented Jun 16, 2025

Uh oh!

Uh oh!

franciscovalentecastro commented Jun 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[processor/transform] Create `With*Functions` factory options to provide custom OTTL funcs programatically. #39966

[processor/transform] Create `With*Functions` factory options to provide custom OTTL funcs programatically. #39966

franciscovalentecastro commented May 9, 2025 •

edited

Loading

evan-bradley left a comment •

edited

Loading

edmocosta commented May 21, 2025 •

edited

Loading

franciscovalentecastro commented May 23, 2025 •

edited

Loading

franciscovalentecastro Jun 4, 2025 •

edited

Loading

evan-bradley Jun 10, 2025 •

edited

Loading