Remove custom robots.txt and sitemap

[DimedS]

Description

Related to #3741.

We decided to use the default robots.txt and sitemap.xml from RTD, because we embedded the nonindex tag in all old docs versions, preventing search engines from indexing them.

See: #4516

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

• Read the contributing guidelines
• Signed off each commit with a Developer Certificate of Origin (DCO)
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes
• Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

[astrojuanlu]

Thanks @DimedS ! Should we hold this off a few days to give Google some time to pick up the changes of #4516 ?

[astrojuanlu]

Nah scratch that. Let's proceed with this. Old pages cannot pick up the noindex tag precisely because they're blocked by our custom robots.txt 👍🏼

[DimedS]

Thanks @DimedS ! Should we hold this off a few days to give Google some time to pick up the changes of #4516 ?

Thanks, @astrojuanlu ! I also want to highlight that nothing will change after #4516 because the script we incorporated locally in our docs folder in that PR is not activated. I think we can ask RTD to activate it only after the release. Currently, the same script is still running from Google Cloud. Since this script has been running for over a week, I believe Google has already updated all the changes from it.