Urdu, Kashmiri and Maithili Support

[anuragshas]

I would like to contribute for Urdu and Kashmiri language which are also one of the official languages in India and has Indian origins.
I have started working on Urdu Language using repo NLP for Marathi. I have gathered around 350K wikipedia articles link and in process of scraping those articles. I have also added multiprocessing support for gathering articles

[goru001]

Thanks for the initiative, shout out if you need any help!

[anuragshas]

I am falling short of memory while creating TextLMDataBunch with only 100K articles and 32K vocabulary. How much memory is required to create the data for language model?

[goru001]

It's advisable to not go beyond vocab length of 30k. Are you talking about GPU memory ? I've GTX 1080 Ti with 11 GB memory on which I trained all the models. You can train your models on Google Colab if you're having trouble with memory on your own gpu. If you were talking about RAM, i think 16gb should be fine!

[anuragshas]

Thank you for the information. The issue was that single file was having over 350K character which was unable to tokenized and numericalized at once and loaded into main memory so I had to select fewer sentences and it worked.
I have also created LM for Maithili language having perplexity of 50. In search of news for classification task.
Side by side I am training on Urdu language with 150K articles.
Kashmiri Language has only 350 articles and I think it won't be enough to create language model

[goru001]

Good to know that it worked.
Yes, 350 articles seems too less. Try if you can get data from somewhere else.. news articles/govt. Websites etc.

[anuragshas]

I have completed for Urdu and here is the link

Resources for Kashmiri language is very scarce and some of them are paid, there are epaper websites having images. I am searching for more resources if possible or else I will have to drop it.

I am working on scraping maithili language news websites

[goru001]

@anuragshas Thanks for the contribution! Would you like to raise a PR to add your model to iNLTK (I can help you with the process)

[anuragshas]

You are welcome. I am really happy that I will be able to raise my first PR on github.
After going through the the code, I guess i will have to change config.py file but I am in doubt will I have to fine tune the all_languages_identifying_model

[goru001]

@anuragshas don't worry about all_languages_identifying_model. I will be fine tuning it to add Tamil language to iNLTK , I will tune it for Urdu as well.

As far as LM is concerned, we'll not be able to add it to iNLTK in the current form. Can you follow these instructions and upload the saved model on dropbox and share it's link with me?

Shout out if you need any help!

[goru001]

@anuragshas You've been working on LM for Maithili as well, right? Can you share the Wikipedia Dataset you would've prepared for it? Because tuning language-classifier model again for Maithili will take time, I was about to train it for Tamil, Urdu, Telugu, so thought if we can add Maithili as well, that would be great!

[anuragshas]

@goru001 Here is the link of MaithiliWikiArticles.
I have been busy searching for job, I will create PR for urdu lm as soon as I get free.

[goru001]

@anuragshas No issues! Good luck :).

[anuragshas]

@goru001 I have uploaded the urdu model using the instructions mentioned here. Please let me know what changes shall I make to create PR