2-gram discount out of range for adjusted count

[ezubaric]

I have two files. One file works fine with kenlm, the other gives the following error:

jbg-hackintosh:qblearn jbg$ lmplz -o 2 -S 2G -T -kndiscount /tmp < bl > scratch/Literature/10393.comb.arpa
=== 1/5 Counting and sorting n-grams ===
Reading stdin
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100

---

Unigram tokens 2366 types 1168
=== 2/5 Calculating and sorting adjusted counts ===
Chain sizes: 1:14016 2:2147469568
/Users/jbg/repositories/kenlm/lm/builder/adjust_counts.cc:50 in void lm::builder::::StatCollector::CalculateDiscounts() threw BadDiscountException because `discounts_[i].amount[j] < 0.0 || discounts_[i].amount[j] > j'.
ERROR: 2-gram discount out of range for adjusted count 3: -0.402645
Abort trap: 6

The only difference between the two files is that one ends with the sentence:

Lord Melbourne offered him a lordship, which he declined

I've also sent the full files to Kenneth via e-mail.

[kpu]

Your 104-line corpus is too small to have all the the modified Kneser-Ney smoothing statistics, even for a bigram model. Specifically, there need to be more words with adjusted count >= 3 (i.e. >= 3 unique words that precede them). You might say the bug is that I should let users configure how many modified counts there are (i.e. binning for 0, 1, 2+ instead of 0, 1, 2, 3+). But as it stands, the Chen and Goodman paper simply fails if these assumptions are not met.

[alvations]

@kpu , I'm having some sort of the same problem but my corpus is rather big. I ran:

lmplz -o 5 -T /tmp < corpus.en | gzip > corpus.lm.en.arpa.gz

and got:

Unigram tokens 93423504 types 311346
=== 2/5 Calculating and sorting adjusted counts ===
Chain sizes: 1:3736152 2:10548410368 3:19778271232 4:31645231104 5:46149300224
/home/expert/mosesdecoder/lm/builder/adjust_counts.cc:53 in void lm::builder::{anonymous}::StatCollector::CalculateDiscounts() threw BadDiscountException because `discounts_[i].amount[j] < 0.0 || discounts_[i].amount[j] > j'.
ERROR: 5-gram discount out of range for adjusted count 3: -0.0885062

After I did a cat corpus.en | sort -u > corpus.uniq.en and retrain a LM based on the unique corpus, KenLM works. Did I lose anything when i only train the corpus on a unique corpus? Will I lose the repetition probabilities?

[aayushkubb]

You can use --discount_fallback option.

[bricksdont]

@aayushkubb But you probably should not, as the help text of lmplz explains:

--discount_fallback [=arg(=0.5 1 1.5)]
                                    The closed-form estimate for Kneser-Ney
                                    discounts does not work without
                                    singletons or doubletons.  It can also
                                    fail if these values are out of range.
                                    This option falls back to
                                    user-specified discounts when the
                                    closed-form estimate fails.  Note that
                                    this option is generally a bad idea:
                                    you should deduplicate your corpus
                                    instead.