Twitter Dataset

[igorbrigadir]

This isn't totally public because you have to have an approved app, but Twitter just released an endpoint that provides the COVID-19 firehose: all tweets matching this list https://developer.twitter.com/en/docs/labs/covid19-stream/filtering-rules can be streamed live: https://blog.twitter.com/developer/en_us/topics/tools/2020/covid19_public_conversation_data.html

Since the data is live streamed, the dataset is not complete yet - but i imagine it might be useful to collect the tweet IDs as a dataset for this project too! Social media posts would also make a good addition.

[wildlewis]

I like the idea of social content being added. Twitter has some restrictions on use, so that may affect the utility and shareability of the data. Although maybe Twitter can be convinced to change their data sharing restrictions? From: Igor Brigadir <notifications@github.com> Sent: Thursday, April 30, 2020 8:04 AM To: neulab/covid19-datashare <covid19-datashare@noreply.github.com> Cc: Subscribed <subscribed@noreply.github.com> Subject: [neulab/covid19-datashare] Twitter Dataset (#5) This isn't totally public because you have to have an approved app, but Twitter just released an endpoint that provides the COVID-19 firehose: all tweets matching this list https://developer.twitter.com/en/docs/labs/covid19-stream/filtering-rules<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdeveloper.twitter.com%2Fen%2Fdocs%2Flabs%2Fcovid19-stream%2Ffiltering-rules&data=02%7C01%7Cwilewis%40microsoft.com%7C0feda83402824d5f3db208d7ed17a527%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637238558129635312&sdata=z%2BXvzVvy9FeE8oPcM0ht%2FtmxXeLUDEw9%2BuPNv7hizxY%3D&reserved=0> can be streamed live: https://blog.twitter.com/developer/en_us/topics/tools/2020/covid19_public_conversation_data.html<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.twitter.com%2Fdeveloper%2Fen_us%2Ftopics%2Ftools%2F2020%2Fcovid19_public_conversation_data.html&data=02%7C01%7Cwilewis%40microsoft.com%7C0feda83402824d5f3db208d7ed17a527%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637238558129640303&sdata=vKmSSE%2Fl%2FSmbwJCASR0YmMckZ2baltuPUuwdAoQMoMU%3D&reserved=0> Since the data is live streamed, the dataset is not complete yet - but i imagine it might be useful to collect the tweet IDs as a dataset for this project too! Social media posts would also make a good addition. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fneulab%2Fcovid19-datashare%2Fissues%2F5&data=02%7C01%7Cwilewis%40microsoft.com%7C0feda83402824d5f3db208d7ed17a527%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637238558129640303&sdata=DzGT5N9ALCUHtDahs4Gzj1l%2F3usnRvYbxgl%2B%2B3axEoM%3D&reserved=0>, or unsubscribe<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAESY4JKH4TFGFCMKRCQFKTRPGHMFANCNFSM4MVU2PSA&data=02%7C01%7Cwilewis%40microsoft.com%7C0feda83402824d5f3db208d7ed17a527%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637238558129645293&sdata=uT8csucfXG%2F%2Bekg9CvTsKkYJx2%2FtAXaAuu%2FYqdlAy7M%3D&reserved=0>.

[igorbrigadir]

Yeah the restrictions are documented here: https://developer.twitter.com/en/developer-terms/agreement-and-policy and there are additional restrictions on use cases here: https://developer.twitter.com/en/developer-terms/more-on-restricted-use-cases

basically you're allowed to share Tweet IDs only, and must re-download when using them again - this ensures that anyone in the dataset retains some control over how their tweets will be used - deleted tweets will simply fail to download again. From a data sharing perspective, this means distributing additional code or instructions for "hydrating" the data like https://github.com/DocNow/twarc#hydrate for example.

It's not certain if the output of this special COVID-19 Stream will be any different - maybe there are additional freedoms or restrictions.