Skip to content

Conversation

david-sitsky
Copy link
Contributor

Addresses issue raised from https://deepjavalibrary.slack.com/archives/C01AURG857U/p1724804411722599.

Essentially batchifying / unbatchifying OrtNDArrays produce PtNDArrays, and this causes downstream issues, particularly in OrtSymbolBlock:114:

            OrtSession.Result results = session.run(container);
            NDList ret = evaluateOutput(results);
            ret.attach(inputs.head().getManager());

Because of the batchify call during Translator processInput() call, the initial OrtNDArrays end up as PtNDArrays, and then the ret.attach call incorrectly sets the ret NDList's main (and alternate manager) both to PtNDManager. This inhibits a lot of post-processing code from working as the returned NDArrays have the wrong manager set.

Maybe there is a more elegant way of fixing this within OrtSymbolBlock, but I couldn't see any easy way of passing in the Predict NDManager in.

@david-sitsky david-sitsky requested review from a team, frankfliu and zachgk as code owners August 30, 2024 05:24
@david-sitsky
Copy link
Contributor Author

I'm not really a fan of this change as I think it will introduce too much memory copying. Maybe there is a better way?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant