Make sentencepiece and tiktoken optional deps #337
Merged
+15
−4
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Ahajha
force-pushed
the
trotta/optional-deps
branch
from
762f64b to
2fa4881
Compare
Ubospica
approved these changes
Jun 16, 2025
There was a problem hiding this comment.
Thanks for the contribution @Ahajha!
Both dependencies are only actually used if users are, so it seems pointless to depend on them directly
This argument makes sense to me. May I ask if you have encountered any specific problems with the current dependencies?
|
@Ubospica Yes, we (Modular) got this user report: modular/modular#4509. We don't use the dependency directly. |
|
Dependencies in general, worth noting is triton and pytorch, but those are likely difficult to get rid of. We are (in an extremely simplified sense) competing with pytorch, so not depending on them is an ideal goal for us, though we use it in other places in our stack so I wouldn't sweat trying to get rid of it. |
modularbot
pushed a commit
to modular/modular
that referenced
this pull request
Jun 20, 2025
ggml-org/llama.cpp#14200 mlc-ai/xgrammar#337 gguf immediately released a new version, waiting on xgrammar. In the meantime though, bump gguf and remove the explicit sentencepiece dependency, it isn't used anywhere. This should automatically propagate up towards users, no need to bump minimum required versions. MODULAR_ORIG_COMMIT_REV_ID: 27aa8a6ece6f56d253875294f8ae9988eecc8cc1
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Sentencepiece has not had a new release in over a year, and users are running into easily fixable install issues like CMake minimum versions, missing headers, and missing wheels for newer python versions. See recent issues: https://github.com/google/sentencepiece/issues, like this one for example: google/sentencepiece#1113
Tiktoken hasn't caused any particular issues, but I'm including it because it's used in the same capacity here.
Both dependencies are only actually used if users are, so it seems pointless to depend on them directly. This updates the support to silently skip over them if they are not installed.