Skip to content

Add FAQ #31

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Aug 6, 2024
Merged

Add FAQ #31

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b4d25bf
Add FAQ
glyn Aug 6, 2024
83d9397
Removing previously generated files
Aug 6, 2024
6d2285f
Add FAQ
Aug 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion FAQ.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,13 @@

## How do we know AI companies/bots respect `robots.txt`?

The short answer is that we don't. `robots.txt` is a well-established standard but compliance is voluntary. There is no enforcement mechanism.
The short answer is that we don't. `robots.txt` is a well-established standard, but compliance is voluntary. There is no enforcement mechanism.

## Why might AI web crawlers respect `robots.txt`?

Larger and/or reputable companies developing AI models probably wouldn't want to damage their reputation by ignoring `robots.txt`.

Also, given the contentious nature of AI and the possibility of legislation limiting its development, companies developing AI models will probably want to be seen to be behaving ethically, and so should (eventually) respect `robots.txt`.

## Can we block crawlers based on user agent strings?

Expand Down