Add support for anchor tags

[lenisha]

Links that point to non existent anchors https://github.com/lenisha/linkchecker-test/blob/main/about.html#L92 are not detected

<a href="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/Microsoft/onnxruntime#api-documentation" target="_blank" class="link">

[pawroman]

At this point, they're not supported.

This is a bit of a tricky one. Anchors are really the properties of the document contents, not really being integral parts of the URLs.

What I really mean by that is this:

• You can't tell if an anchor is valid by merely making a request (you can only check the URL path without the anchor)
• To verify anchors, you'd have to parse the retrieved HTML and check they're there (but we already do it for link extraction using html5ever, a browser grade parser)

So this might not be very straightforward, but should be perfectly doable. I'd welcome such a feature 😃

[MichaIng]

To verify anchors, you'd have to parse the retrieved HTML and check they're there (but we already do it for link extraction using html5ever, a browser grade parser)

HTMLs are parsed for link extraction on the input HTML documents, but not for the target HTML documents. So it would required to first check whether the target is actually an HTML document, then download it fully (currently it is only partially downloaded or not at all when using -X head) and then parse the target HTML for a matching id attribute.

When support for internal links via file existence check is added (#21), then it could be at least cheaply added for those:

• If the link/URL is an internal one, check only if the target file exists, relative to the input HTML file location or relative to the base path, when given and it is an absolute internal link.
• Then, if the target is an HTML file, or a directory that contains an index.htm/index.html, check for a contained matching id attribute.

In other cases, I think it's difficult if one does not want to imply much overhead for checking the URLs document type and downloading it completely. But nevertheless, the idea/feature itself would be awesome of course 🙂.

[MichaIng]

For reference the hint by @polarathene: #21 (comment)

Another case where the difficulties with target downloads etc do not apply, and probably this is what you had in mind anyway, is to check for scroll links within the input HTMLs. so if there is an <a href="#heading">, to check whether there is an id="heading" as well. If such is wanted, it's good to create a generic method/function for it, so that URL fragments can be generally detected and HTMLs generally scanned for matching IDs, but at first implemented for such same-page scroll links, then depending on #21 for locally stored target HTML files, and gradually for online HTML documents, when this is found to be important enough to fully download the target, based on content-type: text/html or so. But I'm not sure if the content type header is reliable enough of if there is a risk that a large archive or video file is unintentionally downloaded 😄.

[polarathene]

But I'm not sure if the content type header is reliable enough of if there is a risk that a large archive or video file is unintentionally downloaded

Response headers should provide info about content size, completing a request in full (eg downloading a large resource) shouldn't be necessary and could be terminated early AFAIK?

[MichaIng]

completing a request in full (eg downloading a large resource) shouldn't be necessary and could be terminated early AFAIK?

If you want to check whether an URL fragment is valid or not, you need to download the whole document of course, which is the whole issue I see here, as long as it's not about same-page links or local files. Currently the download is terminated early, reasonably, so currently it is impossible to apply this feature for remote HTML documents without changing the download behaviour based on such things like HTTP headers, which is why I think one needs to evaluate carefully if the effort is worth it or not 😉.

[polarathene]

you need to download the whole document of course, which is the whole issue I see here

Downloading a complete HTML document to operate on isn't much of a concern vs larger files which I thought was being raised as the concern but could be avoided.

You make a good point though, especially with some HTML documents that are dependent on JS to populate the real content, such as React pages without any pre-rendering process involved. That'd require actually loading and running the JS..

I haven't used lychee myself yet, so I'm not sure if it supports that well (anchor tag feature aside).

[lebensterben]

If other attributes of 'a' tag is provided, you can use that to help you determine whether to download that webpage, or whether it links to webpage, etc. But it gets complicated.

https://html.spec.whatwg.org/multipage/links.html#links