Redirect documention links of EOL versions to Apache Archive Service

[yaooqinn]

This PR redirects all the versions circled in red in the figure below to the Apache Archive Service.

Screen.Recording.2024-08-26.at.13.36.02.mov

0.6.x & 0.7.x are kept here as they seem not to be ASF releases
3.4.x, 3.5.x & 4.0.x are kept here as they are not EOL.

[dongjoon-hyun]

Thank you, @yaooqinn .

[dongjoon-hyun]

cc @srowen

[yaooqinn]

Redirecting…

Click here if you are not redirected.

[dongjoon-hyun]

+1, LGTM.

[yaooqinn]

Merged to asf-site.

Thank you @dongjoon-hyun

[allisonwang-db]

@yaooqinn it doesn't seem to redirect to the archive service and breaks Google search result:
https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.DataFrame.dropDuplicates.html


Also for keyword "pyspark groupby"

[yaooqinn]

Do you know why the Google searcher were looking for these items from 3.1 instead of newer ones such as 3.4 or 3.5

[dongjoon-hyun]

To @allisonwang-db , this PR doesn't mean that we implement a redirection from old Apache Spark links to the archive locations. I believe Google search engine index will be updated gradually.

[allisonwang-db]

I’m afraid that this could seriously impact the PySpark user experience, with an invalid documentation link appearing as the top Google search result for very common keywords.

cc @MrPowers, do you know how long it typically takes for Google’s search index to update?

[dongjoon-hyun]

I’m afraid that this could seriously impact the PySpark user experience, with an invalid documentation link appearing as the top Google search result for very common keywords.

cc @MrPowers, do you know how long it typically takes for Google’s search index to update?

To @allisonwang-db , we already discussed about the risk in the dev mailing list and decided to move only old End-Of-Support docs. Please see the scope of the PR. All live releases documents are available without any change. Given that, I don't think there is any real impact on the PySpark user experience due to this the End-Of-Support documentation link change. This is a legit maintenance activity on the end-of-support artifacts.

[HyukjinKwon]

I am fine with the PR itself but just probably a dumb question. Why doesn't the redirection work?

[yaooqinn]

The redirection pages, such as https://github.com/apache/spark-website/blob/asf-site/site/docs/0.8.0/index.html, are created only for the home pages of each version of docs

[yaooqinn]

I have created a follow-up #552 to enhance the UX. This is the best effort I can do for now.