The previous release (v1.21) failed to add cohere-training-data-crawler to robots.txt because of a bug, which is now hopefully fixed.

Full Changelog: v1.20...v1.21

What's Changed

• Clarify our rationale by @glyn in #54
• Create workflow to upload robots.txt file as release artefact by @norwd in #55
• Allow facebookexternalhit by @glyn in #56
• Restore attribution by @fabianegli in #58

New Contributors

• @norwd made their first contribution in #55

Full Changelog: v1.19...v1.20

What's Changed

• Update robots.json with Claude respect link by @lxjv in #43
• Sort the content of robots.json by keys, case-insensitively by @isagalaev in #46
• Update the FAQ by @glyn in #45
• log the diff in the update actions by @fabianegli in #47
• PHP diagnostics by @glyn in #49
• Fix typo and trigger rerun of main job by @glyn in #50
• PHP to Python plus tests and stuff by @fabianegli in #51

New Contributors

• @lxjv made their first contribution in #43
• @isagalaev made their first contribution in #46
• @fabianegli made their first contribution in #47

Full Changelog: v1.18...v1.19

Adds ISSCyberRiskCrawler. Thanks to Brian at Skewray Research for calling this one out!

What's Changed

• Add Sidetrade indexer bot (Thank you to Brian at Skewray Research for calling this one out.)
• Add instructions for AI bot blocking on Vercel by @cramforce in #36
• Update README.md by @cityrolr in #41
• feat: make CCBot entry more accurate by @wumpus in #42
• Add an additional resource - README.md by @urvish-p80 in #38

New Contributors

• @cramforce made their first contribution in #36
• @cityrolr made their first contribution in #41
• @wumpus made their first contribution in #42
• @urvish-p80 made their first contribution in #38

Full Changelog: v1.16...v1.17

Thank you to Brian at Skewray Research for calling this one out.

What's Changed

• Improve main workflow by @nisbet-hubbard in #35

Full Changelog: v1.15...v1.16

Thank you to Brian at Skewray Research for calling this one out.

Full Changelog: v1.14...v1.15

What's Changed

• Adds the Webzio-Extended bot
• only build on changes to robots.json by @jbowdre in #29
• Add FAQ by @glyn in #31
• Tracking Dark Visitors Automatically by @ChenghaoMou in #32
• Add links for reporting and FAQ to README.md by @nisbet-hubbard in #33

New Contributors

• @jbowdre made their first contribution in #29
• @ChenghaoMou made their first contribution in #32

Full Changelog: v1.13...v1.14

What's Changed

• Fix Imagesift user agent by @jsheard in #27
• Add Cloudflares first-party scraper blocking to FAQ by @jsheard in #28

New Contributors

• @jsheard made their first contribution in #27

Full Changelog: v1.12...v1.13