Recommended way to handle long text inputs (Sliding Window for Token Classification in DJL)

[ivrisivris]

Hi DJL team,

we’re using DJL for token classification on long text inputs that often exceed the model’s maximum token length (e.g., 512 tokens).

The HuggingFaceTokenizer supports parameters like stride and withOverflowingTokens, but it seems that TokenClassificationTranslator does not currently handle overflowing segments automatically.

Could you please advise what is the recommended approach in DJL for processing long texts — ideally with a sliding window or chunking mechanism that merges overlapping predictions?

Also, is there any plan to include native sliding window support in future versions of the library?

Thanks!

Kamil Kočí

[frankfliu]

@ivrisivris
TokenClassificationTranslator doesn't support overflow tokens. You have to implement your own TokenClassificationTranslator. Are interested in contribute to DJL to enhance TokenClassificationTranslator?

[david-sitsky]

@frankfliu - slightly related, but what about TextEmbeddingTranslator handling large text inputs via chunking? At the moment I am doing the chunking "client-side" using the HuggingFace tokeniser with .optStride and sending the token IDs to my custom translators, but ideally I'd like this done "properly" in the standard DJL library. Would we create a ChunkingTextEmbeddingTranslator that returns a List<float[]> or something like that?