Having better defaults for ChrF

[alvations]

The chrf++ is a better chrf than the default chrf6 used in sacrebleu, could we change the default to chrf++?

From the original ChrF creator: https://twitter.com/amelija16mp/status/1331288013880614913

[ozancaglayan]

This would require changing chrf_order param to chrf_char_order, and adding a new chrf_word_order argument. For Chrf++, I think the defaults are 6 and 2, respectively. The current chrF in sacreBLEU probably amounts to these params being 6 and 0. Can someone confirm?

[alvations]

@m-popovic should be the "authoritative" person to confirm that =)

[ozancaglayan]

I added this into my refactor2021 branch. I should note that chrF++.py does a primitive tokenization for word-level n-gram matching e.g. it only separates out punctuations from beginning and end of words. Can someone come up with a regexp to replace the below implementation that I've got from chrF++? (P.S: The below is actually quite fast but I am curious about the regexp as well)

def separate_punctuation(line):
    words = line.strip().split()
    tokenized = []
    for w in words:
        if len(w) == 1:
            tokenized.append(w)
        else:
            lastChar = w[-1] 
            firstChar = w[0]
            if lastChar in string.punctuation:
                tokenized += [w[:-1], lastChar]
            elif firstChar in string.punctuation:
                tokenized += [firstChar, w[1:]]
            else:
                tokenized.append(w)
    
    return tokenized

Also, because of the way the sentences are tokenized as above, languages like zh and ja would probably not benefit from having word-level ngrams, right?

[martinpopel]

it only separates out punctuations from beginning and end of words

Yes, only a single punctuation character, only at beginning and end of words, but if there is a punctuation at the end, no punctuation from the beginning is separated:

separate_punctuation('(hi)') == ['(hi', ')']

This seems very wrong, but if we need to replicate chrF++, we need to accept that.

I am curious about the regexp as well

The following code is so obfuscated and also about 3.3 times slower than the original, so I mention it just as a joke:

import string
punct = string.punctuation
ugly_re = f'\s+|((?<=\S)[{punct}](?=\s|$)|(?<=\s)[{punct}](?=\S*[^{punct}](?:\s|$)))'

def separate_punctuation_re(line):
    return list(filter(None, re.split(ugly_re, line)))

The following code seems to be 23% faster than the original (on a randomly chosen file):

import string
punct_set = set(string.punctuation)

def separate_punctuation4(line):
    tokenized = []
    for w in line.split():
        if len(w) == 1:
            tokenized.append(w)
        else:
            if w[-1] in punct_set:
                tokenized += (w[:-1], w[-1])
            elif w[0] in punct_set:
                tokenized += (w[0], w[1:])
            else:
                tokenized.append(w)
    return tokenized

[ozancaglayan]

Didn't notice that glitch about being first/last. Definitely weird, probably an elif / if confusion? I'll note it there as a comment.
Will integrate your last version, thanks!