URLHeadBear should use robots.txt

[jayvdb]

https://en.wikipedia.org/wiki/Robots_exclusion_standard

Any request that isnt allowed by robots.txt should be reported as such.

[meetmangukiya]

(participating)

[refeed]

We can use https://docs.python.org/2/library/robotparser.html for this, and with some enhancement like cache robots.txt for each site, etc..