docs: add noindex tag to non-stable pages (Eventual-Inc#5105)

## Changes Made

Adds noindex tag to all docs that are not `/stable/`

As per https://aioseo.com/docs/when-to-use-noindex-or-the-robots-txt/

> The biggest difference to understand is that if you want search
engines to not include content in search results, then you MUST use the
NOINDEX tag and you MUST allow search engines to crawl the content. If
search engines CANNOT crawl the content then they CANNOT see the NOINDEX
meta tag and therefore CANNOT exclude the content from search results.
> 
> So if you want content not to be included in search results, then use
NOINDEX. If you want to stop search engines crawling a directory on your
server because it contains nothing they need to see, then use “Disallow”
directive in your robots.txt file.

i.e. we should:

1. Allow for crawling of all docs (fix robots.txt) `NOTE: this is not
done in this PR, see below for why`
2. Add a `canonical` link for everything that is not `stable` `NOTE:
this is done already`
3. Add a <noindex> tag for everything that is not `stable`

This PR just does (3). This means that starting from 0.6.0, all
versioned docs will have the `noindex` tag. However, it is not yet safe
to modify our robots.txt because all docs <0.6.0 still do not contain
`noindex`. We should only modify the robots.txt once we have that
guarantee.

## Testing

* Confirmed that <noindex> tag now appears in this build:
https://getdaft-docs--5105.org.readthedocs.build/en/5105/quickstart/
* Previously not appearing in
https://docs.daft.ai/en/v0.5.22/quickstart/

Author: jaychia

docs/hooks.py

@@ -0,0 +1,7 @@
+import os
+
+
+def on_config(config):
+    """Set Read the Docs version information in config."""
+    config.extra["rtd_version"] = os.environ.get("READTHEDOCS_VERSION", "unknown")
+    return config

docs/overrides/main.html

@@ -25,6 +25,9 @@
 {% block site_meta %}
 {{ super() }}
 <meta name="readthedocs-addons-api-version" content="1" />
+{% if config.extra.rtd_version != 'stable' %}
+<meta name="robots" content="noindex">
+{% endif %}
 {% endblock %}
 
 <!-- Remove automatic appending of site name to page title -->

mkdocs.yml

@@ -209,3 +209,5 @@ plugins:
 - nav-hide-children:
     hide_children:
     - ["Python API", "Functions"]
+hooks:
+- docs/hooks.py