Interceptors for SQL generation

[BalassaMarton]

Note: #28505 to allow interception of the LINQ expression tree has been split off from this and implemented in EF7.

Use cases

• Generating DB commands for operations not currently supported by EF (e.g. ExecuteUpdate/Delete (AKA bulk update, without loading data into memory) #795)
• Extending with provider-specific LINQ operators (like how linq2db allows a wide variety of SQL constructs)
• More flexible and robust query filters
• Query validation and transformation

The current query generation pipeline is hard to extend without relying on implementation details.

Proposal

Enable intercepting the translation steps in QueryCompilationContext.CreateQueryExecutor.
API details are to be discussed here, I don't have a concrete proposal yet.

[roji]

@BalassaMarton the query pipeline is already extensible in many different ways, thanks to the DI infrastructure of EF Core:

• Providers (and plugins) can easily add method and member translators
• Providers (and even users) can provide their own implementation of IRelationalSqlTranslatingExpressionVisitorFactory to translate other types of expressions
• Providers (and even users) can add query pipeline preprocessors (which run at the very beginning) and postprocessors (which run after translations). We do have plans to improve the API for inserting new preprocessing/postprocessing visitors for 5.0.

Admittedly we currently lack documentation on how to do the above - we definitely plan on making all this more discoverable for 5.0 (see dotnet/EntityFramework.Docs#681 for example, as well as other issues).

Are there any specific scenarios you have in mind, where you're unsure how to proceed? Is your interest more as a provider writer, or as an end user?

[BalassaMarton]

I'm only interested as a user, but I'll look deeper into the visitors and try to understand how it can be extended, thank you.

[ajcvickers]

Note from team discussion: we need to document this (see dotnet/EntityFramework.Docs#951 and dotnet/EntityFramework.Docs#500) but following that we should look at adding sugar for common patterns to avoid the need to understand expression visitors for common custom translations.

[smitpatel]

@BalassaMarton & @powermetal63 - Can you provide details of exactly what you want to change in expression tree for which you need extensibility? We can determine based on that what could be right hook for it and if it is missing, add it. Query pipeline is a complex system which has multiple parts end to end. As @roji mentioned above, there are hooks a long the way depending what kind of task need to be done.

[powermetal63]

@smitpatel I would start with the ability to preprocess/transform the IQueryable.Expression before EF Core infrastructure. This will be very useful for lambda injection extensions like NeinLinq, LinqKit and similar which currently have to implement custom IQueryProvider and IQueryable just to do that. The end user is required to call a custom method like ToInjectable(), AsExpandanle(), which can easily be forgotten, and also custom query providers doesn't play well with EF Core queryable/async extensions.

Also it will be useful if we want to add "fixes" to some things you are not willing to (or have no time, or will do it in future version we cannot afford waiting for etc.), like the one here #19930 (comment).

I guess all this falls into bullet #3 - Easy way to add query pipeline preprocessors. By easy I mean to be able to do that without implementing custom options in order to add services etc. Something as easy to use as the current ReplaceService.

[smitpatel]

Something as easy to use as the current ReplaceService.

That already works for everything you want to inject during compilation phase.

Anything before pre-funcletization can be done outside of EF before passing expression into EF.

[powermetal63]

@smitpatel But we don't want it do be done externally. I already gave you several examples. We don't want to ask the user to call custom methods like ToInjectable, AsExpandable, ConvertGroupJoins etc. in order get the functionality. Because we are extending/fixing EF/C# query limitations, and want these pure query expression transformations to be executed automatically for each EF query "compilation".

So what we want is an easy way to plug something similar to your current QueryTranslationPreprocessor.Process - like GroupJoinConverter from #19930 (comment), or LinqKit Expand functionality. Or in other words, something similar to AddInterceptors, but for query preprocessors which then run before EF.

[smitpatel]

@powermetal63 - You can already replace QueryTranslationPreprocessor by using ReplaceService API on IQueryTranslationPreprocessorFactory service. Did you look at that functionality?

[powermetal63]

@powermetal63 - You can already replace QueryTranslationPreprocessor by using ReplaceService API on IQueryTranslationPreprocessorFactory service. Did you look at that functionality?

@smitpatel Sure I know it, but that's the problem - all I can do is to replace it. Let say I replace it and return custom QueryTranslationPreprocessor. Which one QueryTranslationPreprocessor should I inherit - QueryTranslationPreprocessor, RelationalQueryTranslationPreprocessor or something else - at that moment I have no idea which is the default (current) IQueryTranslationPreprocessorFactory, what dependencies it has and what type of QueryTranslationPreprocessor. it creates. And since EF relies on that query preprocessing, we can easily break it unintentionally.

Shortly, IMHO this factory service approach is quite limited - I'm not sure why you created it and what it will be used for internally by EF, but it definitely is not something I (or 3rd party extensions creators) want to dial with. We don't need to replace something existing, we need something additive like recently added interceptors.

---

P.S. Please don't get me wrong. I'm doing EF Core hacks/workarounds from the very beginning. Here we are discussing a good public way of extending EF Core. Cheers.

[smitpatel]

Here we are discussing a good public way of extending EF Core. Cheers.

Infrastructure to extend EF already exists as asked by OP. It is personal preference to use it or not. That has no affect on this issue whatsoever.