[Breaking] Switch text loading to multi-threaded PyArrow loading

[thomwolf]

Test if we can get better performances for large-scale text datasets by using multi-threaded text file loading based on Apache Arrow multi-threaded CSV loader.

If it works ok, it would fix #546.

Breaking change:
The text lines now do not include final line-breaks anymore.

[lhoestq]

Awesome !
Also I was wondering if we should try to make the hashing of the data_files faster (it is used to build the cache directory of datasets like text or json). Right now it reads each file and hashes all of its data. We could simply hash the path and some metadata including the time last modified tag no ? Apparently we can get this tag with os.path.getmtime(path)

[lhoestq]

I just rebased from master to include the hashing changes from #573

[thomwolf]

I think this is ready to merge, no?

[lhoestq]

Indeed it's ready to merge :)

[thomwolf]

Ok added the breaking change info and we can merge indeed.