updated robots.txt Non latest documentation fixes#1960

[rajasurya-rjs]

Summary of Changes:

• Updated robots.txt file to ensure that only the latest version of the Taipy documentation is indexed by search engines.
• Allow search engines to index the latest version: /en/latest/
• Disallow search engines from indexing all other versions: /en/*/
• Sitemap reference added: https://docs.taipy.io/sitemap.xml
  fixes #1960

Purpose of These Changes:

• Prevent older documentation versions from appearing in search results when users search for Taipy documentation (e.g., currently, the /en/release-3.0/ version is showing instead of the latest).
• Ensure that search engines prioritize the latest version (/en/latest/) of the documentation.

Next Steps (For Maintainers):

• After merging this PR, it would be helpful to resubmit the robots.txt file and the sitemap via Google Search Console to expedite the reindexing process.
• Consider using the URL removal tool in Google Search Console to request removal of outdated versions from search results to further improve the discoverability of the latest documentation.

Testing:

• The changes have been tested locally by inspecting the robots.txt file and verifying the presence of the canonical tags in the HTML headers of older versions.

Please review the changes, and let me know if any adjustments are needed. I’m happy to assist with further improvements!

[jrobinAV]

I have a question: Why do we need a * for the disallow line, but not for the allow line?

Would it be better to have something like:

Allow: /en/* # Allow all robots to access the latest version of the documentation
Disallow: /en/release-*/ # Disallow all other versions except 'latest'

[rajasurya-rjs]

In a robots.txt file, the * wildcard is used in the Disallow line to block a range of URLs matching a pattern, such as Disallow: /en/release-*/, which prevents access to all release versions of the documentation except the latest. On the other hand, the Allow line often doesn’t need a wildcard, as it generally applies to all URLs under a specified path by default. For instance, Allow: /en/ would allow access to all pages starting with /en/, including the latest version, without needing a wildcard. Your proposed configuration of using Allow: /en/* and Disallow: /en/release-*/ is valid, allowing access to all documentation under /en/ while blocking specific release versions, except those under /en/latest/.

[github-actions]

This PR has been labelled as "🥶Waiting for contributor" because it has been inactive for more than 14 days. If you would like to continue working on this PR, then please add new commit or another comment, otherwise this PR will be closed in 14 days. For more information please refer to the contributing guidelines.