Skip to content

Learned filter; learned compare (for resolve and join) #187

@shreyashankar

Description

@shreyashankar

In the optimizer; we should be able to fit a logistic regression model to the embeddings to try to learn the binary classification function. If we learn this with good accuracy; we could replace the LLM call with the logistic regression model---or we could implement a model cascade, if the logistic regression model is well-calibrated, we can fall-back to the LLM call if the probability is uncertain.

Metadata

Metadata

Assignees

No one assigned

    Labels

    efficiencyMaking docetl operations run fasterenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions