Need census of Unicode characters in PG texts

[eshellman]

Here something that woud be really useful for PG development going forward: A count of the occurrence of unicode code points used in PG texts. A recent issue #271 notes that the 2em dash is often used but missing in many typefaces.

Applications:

• we could automatically add an embedded "polyfill" subsetted font to epubs to improve the appearances of characters with spotty coverage in typefaces, or we could recommend to users to use fonts with comprehensive coverage.
• we could produce a test epub for use by producers to see coverage of non-ascii code point.
• data science!

How:

• a zipped tarball of all the texts in PG is available at https://gutenberg.org/cache/epub/feeds/txt-files.tar.zip

[rbfrank]

I'd be happy to work on this. However, I need some clarification.

• Is a scan of all -0.txt (declared "UTF-8") files in the catalog sufficient?
• Do you need all of them or just a representative 10,000 or 40,000 or so?
• Sometimes the HTML and the text version of a file are different, with numeric entities being used in the HTML. Do you need to capture the HTML as well?
• Or maybe, since this is EPUB-related, do only need the code points used in the HTML?

Let me know and I'll get started.

[eshellman]

Use the generated txt files from the tarball. The -o files are not guaranteed to be UTF-8

Since we're interested in the outliers, it should be all of them.

No, we do not want the html. I know there may be differences, but looking at the html is a lot more complex.

[anttijuu]

Gave this a shot: https://github.com/anttijuu/PGUnicodeCharacters

But please consider that I am almost a total noob in Unicode so there may be issues...

[rbfrank]

Results are here: http://pglaf.org/~rfrank/ccreport.txt

I wrote a Go program for it to use all 14 cores in parallel. Here is the run:

$ go run main.go t/cache/epub
Scanning directory: t/cache/epub
Found 62316 text files
Processing completed in 29.11831986s
Found 28152 unique characters
Total runes processed: 24333338985
Report written to ccreport.txt

[eshellman]

These are both fascinating!
The 2 counts are very consistent for the most frequent characters - 2,270,276,746 occurrences of 'e' vs. 2,270,275,893, which could be one book differences in the sample sets.
But Á has 91,199 occurrences in one count vs 62,459 in the other count suggests that handling of characters with 2 byte UTF8 encoding may be incorrect in one of the 2 counts.
It looks like handling of encoding errors may account for many of the rare occurrences. if there are control characters, they should be fixed!

[srjfoo]

But Á has 91,199 occurrences in one count vs 62,459 in the other count suggests that handling of characters with 2 byte UTF8 encoding may be incorrect in one of the 2 counts.

Perhaps some have been normalized and some not?

It looks like handling of encoding errors may account for many of the rare occurrences. if there are control characters, they should be fixed!

One thing that I noticed was that variation selectors were reported as separate characters. Take, for instance, the emoji for the zodiac symbols. Anything coming from DP after the Symbols Collection character suite was added should have a VS15 character associated with the zodiac symbols to force it to not appear as a color emoji. (This is not a criticism of @rbfrank's list, just an observation that it might account for some discrepancies, also. I was also fascinated.)

[rbfrank]

Ok I don't know how much time I'm supposed to be spending on this but here are some updates.

1. Regarding the VS15 character, there are 864 of those used in the published books. I treat the VS15 as a separate character and count it even though some might think of it as a combining character with the zodiac symbol.
2. I see there are discrepancies between my Go code and the Swift version. To help understand the differences, I rewrote the program entirely in Python. The results agreed very closely with the GO program run.
3. Regarding no U+0000 characters, it looks like there just aren't any. Neither my GO program or my Python program found any, and neither did the Swift program. I don't see U+0000 in any of the reports. My suspicion is that it is getting mapped to U+FFFD [INVALID UTF-8]. Notice that the Swift implementation and my two versions all report 4145 instances of the U+FFFD character.
4. Finally regarding the large count difference for “Á”, I'm guessing it's because I am counting raw code points in Go and Python and Swift is counting extended grapheme clusters instead. I guess if I were a purist, I'd be combining code points to get a true count of what would "look like" an “Á”. I'd have to deal with NFC and NFD forms.

I hope what I've done is useful. If someone needs more, rattle my cage and let's discuss it.