Releases · CUNY-CL/wikipron

Big scrape for 2022. (#464)
Added the --fresh flag to data/scrape/scrape.py to facilitate running the big scrape in batches. (#464)
Added the --exclude flag for excluding one or more languages in data/scrape/scrape.py. (#460)
Added data/src/normalize.py. (#356)
Updated README.md. (#360)
Added data/cg/tsv/geo.tsv. (#367)
Added data/morphology. (#369)
Added SIGMORPHON 2021 morphology data. (#375)
Added data/cg/tsv/jpn_hira.tsv. (#384)
Enforced final newlines. (#387)
Adds all UniMorph languages to morphology. (#393)
Added data/covering_grammar/tsv/fre_latn_phonemic.tsv (#398)
Added data/covering_grammar/lib/make_test_file.py (#396, #399)
Added Komi-Zyrian (kpv). (#400)
Added Makasar (mak). (#415, #419)
Added Zou (zom). (#421)
Added Wiyot (wiy). (#422)
Added Sidamo (sid). (#423)
Added Central Atlas Tamazight (tzm). (#429)
Added Chibcha (chb). (#430)
Added Kashmiri (kas). (#431)
Added Malayalam (mal). (#434)
Added Dhivehi (div). (#437)
Added Akkadian (akk). (#441)
Added Central Nahuatl (nhn). (#443)
Added Etruscan (ett). (#444)
Added Gujarati (guj). (#445)
Added Kannada (kan). (#446)
Added Karelian (krl). (#447)
Added Romagnol (rgn). (#448)
Added Southern Yukaghir (yux). (#449)
Added Urak Lawoi' (urk). (#451)
Added Hausa (ha). (#452)
Added Kashubian (csb). (#453)
Added Tabaru (tby). (#455)
Added West Makian (mqs). (#457)
Added Amharic (amh). (#458)
Added Livvi (olo). (#459)
Added Kalmyk (xal). (#472)
Added Ternate (tft). (#473)
Added Abkhaz (abk). (#474)
Added Farefare (gur). (#475)
Added Iban (iba). (#476)
Added Laz (lzz). (#477)

Changed

Switched to ISO 639-3 language codes. (#468)
Updated scraped data in preparation for the SIGMORPHON 2022 shared task:
swe nno ger dut ita rum ukr bel tgl ceb ben asm per pus tha lwl. (#461)
Made scripts under data/frequencies/ and data/morphology/ more flexible,
especially for the purposes of preparing data for a shared task. (#461)
Fixed the --restriction flag for specifying multiple languages in data/scrape/scrape.py. (#460)
Added covering grammar coverage error log and specified error_type in error_analysis.py. (#424)
Added error log writing in error_analysis.py. (#420)
Added new columns in summary tables. (#365)
Fixed broken paths in data/src/generate_phones_summary.py and in
data/phones/HOWTO.md. (#352)
Added Atong (India) (aot). (#353)
Added Egyptian Arabic (arz). (#354)
Added Lolopo (ycl). (#355)
Fixed Unicode normalization in data/phones/slv_phonemic.phones and
re-scraped Slovenian data. (#356)
Updated data/phones/HOWTO.md to include instructions on applying the
NFC Unicode normalization (#357)
Updated data/src/normalize.py to be more efficient. (#358)
Fixed inaccuracies in data/phones/geo_phonemic.phones. (#367)
Fixed typo in data/cg/tsv/geo.tsv and added missing character. (#370)
Morphology URLs are now provided as a list. (#376)
Configured and scraped Yamphu (ybi). (#380)
Configured and scraped Khumi Chin (cnk). (#381)
Made summary generation in common_characters.py optional. (#382)
Fixed phone counting in data/src/generate_phones_summary.py (#390, #392)
Reorganizes scraping scripts under data/scrape (#394)
Reorganizes .phones files and related scripts under data/phones (#395)
Reorganizes CG files and related scripts under data/covering_grammar (#395)
Reorganized data/phones/phones/fre_phonemic.phones (#398)
Removed data/src/ (#401)
Renamed TSV files and phonelists to use the terms "broad"/"narrow" instead
of "phonemic"/"phonetic" (#389, #402, #405)
Fixed typo in README.md (#407)
Fixed column ordering of the test file read by the script in
data/covering_grammar/lib/error_analysis.py (#411)
Fixed Common character collection in common_characters.py (#419)
Scraping test fixed for blt. (#436)
Changed URLs to point at CUNY-CL repo, where applicable. (#438)

Under `wikipron/` and elsewhere

Added

Added ckb in languagecodes.py. (#464)
Added support for Python 3.10. (#462)
Added test of phones list generation in test_data/test_summary.py (#363)
Added Min Nan extraction function. (#397)
Added Tai Dam extraction function, configuration and initial scrape. (#435)
Added test of casefold value for languages in data/scrape/lib/languages.json (#442)
Added support for Python 3.11. (#479)
Added checks for the Python source distribution and wheel on CI. (#479)
Turned on tests for Windows on CI. (#479)

Removed

Dropped support for Python 3.6. (#462)
Dropped support for Python 3.7. (#479)

Changed

Switched to ISO 639-3 language codes. (#468)
Converted setup.py to pyproject.toml. (#479)

Assets 2

30 Jan 20:59

kylebgorman

v1.2.0

a586f58

[1.2.0] - 2021-01-30

Under `data/`

Added

Added Afrikaans whitelists, filtered TSV file, rescraped phonemic and phonetic TSV files. (#311)
Added German whitelists and filtered TSV file. (#285)
Added whitelisting capabilities to postprocess. (#152)
Added whitelists for Dutch, English, Greek, Latin, Korean, and Spanish.
(#158, etc.)
Logged dialect configuration if specified. (#133)
Added typing to big scrape code. (#140)
Added argparse to allow limiting 'big scrape' to individual languages
with --restriction flag. (#154)
Added Manchu (mnc). (#185)
Added Polabian (pox). (#186)
Added aar, bdq, jje, and lsi. (#202)
Added tyv to languagecodes.py (#203, #205)
Added bcl, egl, izh, ltg, azg, kir and mga to languagecodes.py. (#205)
Added nep to languagecodes.py. (#206)
Added Ingrian (izh). (#215)
Added French phoneme list and filtered TSV file. (#213, #217)
Added Corsican (cos). (#222)
Added Middle Korean (okm). (#223)
Added Middle Irish (mga). (#224)
Added Old Portuguese (opt). (#225)
Added Serbo-Croatian phoneme list and filtered TSV files. (#227)
Added Tuvan (tyv). (#228)
Added Shan (shn) with custom extraction. (#229)
Added Northern Kurdish (kmr). (#243)
Added a script to facilitate the creation of a .phones file. (#246)
Added IPA validity checks for phonemes. (#248)
Split multiple pronunciations joined by tilde in eng_us_phonetic.
Added Italian phoneme list and filtered TSV file. (#260, #261)
Added Adyghe phone list and filtered TSV file. (#262, #263)
Added Bulgarian phoneme list and filtered TSV file. (#264, #267)
Added Icelandic phoneme list and filtered TSV file. (#269, #270)
Added Slovenian phoneme list and filtered TSV file. (#271, #273)
Added normalization to list_phones.py. Corrected errors relating to
ipapy (#275)
Added Welsh .phones lists and filtered TSV files. (#274, #276)
Added draft of covering grammar script. (#297)
Updated data/phones/README.md with instructions to re-scrape. (#279, #281)
Added Vietnamese .phones files and re-scraped and filtered .tsv files.
(#278, #283)
Added Hindi .phones files and the re-scraped and filtered .tsv files.
(#282, #284)
Added Old Frisian (ofs). (#294)
Added Dungan (dng). (#293)
Added Latgalian (ltg). (#296)
Added draft of covering grammar script. (#297)
Added Portuguese .phones files and re-scraped data. (#290, #304)
Added Japanese .phones files and re-scraped data. (#230, #307)
Added Moksha (mdf). (#295)
Added Azerbaijani .phones files and re-scraped data. (#306, #312)
Added Turkish .phones file and re-scraped data. (#313, #314)
Added Maltese .phones file and re-scraped data. (#317, #318)
Added Latvian .phones file and re-scraped data. (#321, #322)
Added Khmer .phones file and re-scraped data. (#324, #327)
Added Østnorsk (Bokmål) .phones file and re-scraped data. (#324, #327)
Several languages added to languagecodes.py. (#334)

Changed

Edited the arm_e_phonetic.phones and arm_w_phonetic.phones files. (#298)
Improved printing in the README table. (#145)
Renamed data directory data. (#147)
Split may into Latin and Arabic files. (#164)
Split pan into Gurmukhi and Shahmukhī. (#169)
Split uig into Perso-Arabic and Cyrillic. (#173)
Only allowed Latin spellings in Maltese lexicon. (#166).
Split mon into Cyrillic and Mongol Bichig (#179).
Merged whitelist.py into 'big scrape' script. src scrape.py now checks for
existence of whitelist file during scrape to create second filtered TSV.
New TSV placed under tsv/\*\_filtered.tsv. (#154).
Updated generate_summary.py to reflect presence of 'filtered' tsv. (#154)
Imperial Aramaic (arc) split into three scripts properly. (#187)
Flattened data directory structure. (#194)
Updated Georgian (geo) to take advantage of upstream bot-based
consistency fixes. (#138)
Split arm into Eastern and Western dialects. (#197)
Rescraped files with new whitelists. (#199)
Updated logging statements for consistency. (#196)
Renamed .whitelist file extension name as .phones. (#207)
Split ban into Latin and Balinese scripts. (#214)
Split kir into Cyrillic and Arabic. (#216)
Split Latin (lat) into its dialects. (#233)
Added MyPy coverage for wikipron, tests and data directories. (#247)
Modified paths in codes.py, scrape.py, and split.py. (#251, #256)
Modified config flags in languages.json and scrape.py. (#258)
Edited Serbo-Croatian .phones file to list all vowel/pitch accent
combinations. Re-scraped Serbo-Croatian data. (#288)
Moved list_phones.py to parent directory. (#265, #266)
Moved list_phones.py to src directory. (#297)
Frequencies code no longer overwrites TSV files. (#320)
Updated data/phones/README.md to specify that .phones files should be
in NFC normalization form. (#333)
Kurdish (kur) and Opata (opt) removed from languages.json. (#334)
Re-scraped Armenian data. Fixed an error in West Armenian phone list.
(#338)

Fixed

Fixed path issue with phonetic whitelisted files. (#195)

Under `wikipron/` and Elsewhere

Added

Added positive flags for stress, syllable boundaries, tones, segment to cli.py. (#141)
Added positive flags for space skipping to cli.py. (#257)
Added two Vietnamese dialects to languages.json. (#139)
Handled additional language codes. (#132, #148)
Added --no-skip-spaces-word and --no-skip-spaces-pron flag. (#135)
Allowed ASCII apostrophes (0x27) in spellings. (#172).
Added Vietnamese extraction function. (#181).
Modified pron selector in Latin extraction function. (#183).
Added --no-tone flag. (#188)
Customized extractor and new scraped prons for khb. (#219)
Added tests/test_data directory containing two tests. (#226, #251)
Added HTTP User-Agent header to API calls to Wiktionary. (#234)
Added support for python 3.9 (#240)
Added black style formatting to .circleci/config.yml. (#242)
Added logging for scraping a language with --dialect specified
that requires its custom extraction logic. (#245)
Improved CircleCI workflow with orbs. (#249)
Added test_split.py to tests/test_data. (#256)
Handled Cantonese for scraping. (#277)
Added exclusion for reconstructions. (#302)
Added Vietnamese contour tone grouping test in tests/test_config.py (#308)
Added restart functionality. (#340)

Changed

Renamed arguments to positive statements in wikipron/config.py and edited _get_process_pron function accordingly. (#141, #257)
Changed testing values used in tests/test_config.py in order to accomodate the addition of positive flags. (#141)
Specified UTF-8 encoding in handling text files. (#221)
Moved previous contents of tests into tests/test_wikipron (#226)
Updated the packages version numbers in requirements.txt to their latest according to PyPI (#239)
Updated the default pron selector to also look for IPA strings under paragraphs in addition to list items. (#295)
Updated segments package version to 2.2.0 (#308)

Removed

Moved Wiktionary querying functions from test_languagecodes.py to codes.py (#205)

Assets 2

03 Mar 15:37

jacksonllee

v1.1.0

7ea49af

v1.1.0

[1.1.0] - 2020-03-03

Added

Added the extraction function for Mandarin Chinese and its scraped data. (#124)
Integrated Wortschatz frequencies. (#122)

Changed

Updated the Japanese extraction function and Japanese data. (#129)
Updated all scraped Wiktionary data and frequency data. (#127, #128)
Generalized the splitting script in the big scrape. (#123)
Moved small file removal to generate_summary.py. (#119)
Updated Russian data. (#115)

Fixed

Avoided and logged error in case of pron processing failure. (#130)

Assets 2

Releases: CUNY-CL/wikipron

v1.3.3

Uh oh!

Release [1.3.2] - 2024-07-17

Under data/

Changed

Under src/ and elsewhere

Added

Changed.

Uh oh!

[1.3.1] - 2024-03-02

Under data/

Under wikipron/ and elsewhere

Uh oh!

[1.3.0] - 2022-11-28

Under data/

Added

Changed

Under wikipron/ and elsewhere

Added

Removed

Changed

Uh oh!

[1.2.0] - 2021-01-30

Under data/

Added

Changed

Fixed

Under wikipron/ and Elsewhere

Added

Changed

Removed

Uh oh!

v1.1.0

[1.1.0] - 2020-03-03

Added

Changed

Fixed

Uh oh!

Under `data/`

Under `src/` and elsewhere

Under `data/`

Under `wikipron/` and elsewhere

Under `data/`

Under `wikipron/` and elsewhere

Under `data/`

Under `wikipron/` and Elsewhere