Update robots.txt to hide older versions of the docs

[tynandebold]

Description

Resolves #2980.

Development notes

We're being more explicit now about what we're allowing and disallowing.

Notably, changing Disallow: * to Disallow: / disallows crawling of the entire website. Under that rule, we then state what we want crawled, which @stichbury outlined here.

Checklist

• Read the contributing guidelines
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes
• Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

[stichbury]

This looks great to me, thanks.

Looking forward, we'll need to remember to add each release as it arises, so when it is no longer the 'stable' release, it is still indexed. I have added to the Release+step-by-step+checklist+Kedro on Confluence

[merelcht]

LGTM! 👍

[astrojuanlu]

Gave a quick test with https://technicalseo.com/tools/robots-txt/ and looks good!