[Neo] Neo compilation/quantization script bugfixes #2115
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Changes the Neo output file format to better support requirements.txt and custom entry point files. The compiler output will be saved to a subdirectory and the input files are copied over to the output.
a-ys
changed the title
lanking520
approved these changes
Jul 10, 2024
tosterberg
pushed a commit
to tosterberg/djl-serving
that referenced
this pull request
Jul 18, 2024
) (cherry picked from commit 88f84ba)
tosterberg
added a commit
that referenced
this pull request
Jul 18, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR includes various fixes for the Neo Neuron compilation & vLLM quantization scripts.
9253e42 Hard-codes
engine=Pythonin to the Neo Neuron compilation script so that errors in customer serving.properties do not cause compilation to fail.405ea55 Removes a hanging reference to
TARGET_INSTANCE_TYPEin the Neo Quantization script.36810cc Adds logic to pass through
engineandoption.entryPointto the outputted serving.properties. This is done so that when we compile with hardcoded valuesengine=Pythonandoption.entryPoint=djl_python.transformers_neuronx, customer values for these are passed through to support custom entrypoints.c1556ec Changes the output file format to this following:
optimized_modelmodel_id=./optimized_modelso that the compiled model is used during deployment.This is done to allow for custom entrypoint files & requirements files for serving. Introduces an issue of doubling the model size. This will be refined in future changes.
9ddd2c3 Adds a check to make tp_degree required in the Neo neuron compilation script.