Add support for Python3.13, fix running test on localhost, and move regression tests to pytest

[MonkeyCanCode]

Description

Fix #21

High level changes:

1. Add support for Python 3.13:

While trying to run regtest on local with Python 3.13, it failed on installing pydantic-core:

➜ regtests/run.sh
...
Package operations: 8 installs, 1 update, 0 removals
...
  • Installing pydantic-core (2.18.4): Failed
...
  ChefBuildError
  Backend subprocess exited when trying to invoke build_wheel
  Running `maturin pep517 build-wheel -i /private/var/folders/fl/1gq302cn7dd_w8mx24s7zhq40000gn/T/tmponjgdrat/.venv/bin/python --compatibility off`
  💥 maturin failed
    Caused by: Cargo metadata failed. Do you have cargo in your PATH?
    Caused by: No such file or directory (os error 2)
  Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/private/var/folders/fl/1gq302cn7dd_w8mx24s7zhq40000gn/T/tmponjgdrat/.venv/bin/python', '--compatibility', 'off'] returned non-zero exit status 1
...
Note: This error originates from the build backend, and is likely not a problem with poetry but with pydantic-core (2.18.4) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "pydantic-core (==2.18.4)"'.
...
Thu Dec 12 22:46:15 CST 2024: Starting pytest t_cli:test_cli.py
/Users/yong/polaris-venv/bin/python3: No module named pytest
...
Thu Dec 12 22:46:15 CST 2024: Test FAILED: t_cli:test_cli.py
...

This then prevents the installation of pytest which then fails all tests that are based on pytest.

2. Fix running test on localhost
   Also, regtests/t_cli/src/test_cli.py set default polaris host to polaris which can be problematic (as we are not documenting extra set for setting hostname anywhere for running test on local. Thus, better to set this to localhost instead.

3. Move all regression tests to pytests

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• Documentation update
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Detail changes summary:

1. Update README.md for inform users Polaris server should be start before local test can be run
2. Update README.md/docker-compose to remove AWS_TEST_BASE reference due to bad incomplete logic in the test cases (we should create a diff test case for this)
3. Refactor dockerfile for regtests
4. Add missing Azure variables in regtests/README.md
5. Include spark 3.5.3 via pip instead of setting up local spark
6. Bump the version of tox from >=3.9.0 to >=4.19.0 as that is the first version that support Python 3.13
7. Locked the version of platformdirs to 4.3.6 as virtualenv has >=3.9.1,<5 and will try to downgrade version to 3.11.0 during rerun of local test
8. Remove regtests/pyspark-setup.sh, regtests/run_spark_sql.sh and regtests/setup.sh as these two are no longer needed
9. Refactor regtests/runsh to perform regression tests on pytests only
10. Use localhost as default value for polaris host instead of polaris to support running regression tests not within docker (still do-able without this change if we want to add document to ask user to set POLARIS_HOST)
11. Refactor conftest.py and iceberg_spark.py
12. Reduce older ref/src regression tests with pytest

Things I didn't do

1. As I don't have access to Azure/GCP, I am not able to test 3 of the refactored regression tests related to them
2. I did some minor refactor on regtests/t_pyspark/src/test_spark_sql_s3_with_privileges.py. I can do another follow up PR to refactor all of them to more standard layout/format after current PR. Also, we should remove snowflake in this file in the next PR.

Test results:

# local test
➜ regtests/run.sh
...
Sat Dec 14 23:14:17 CST 2024: Running all tests
Sat Dec 14 23:14:17 CST 2024: Starting pytest t_cli:test_cli.py
========================================================================== test session starts ===========================================================================
...
=========================================================================== 9 passed in 35.10s ===========================================================================
Sat Dec 14 23:14:52 CST 2024: Test SUCCEEDED: t_cli:test_cli.py
Sat Dec 14 23:14:52 CST 2024: Starting test t_hello_world:hello_world.sh
Sat Dec 14 23:14:52 CST 2024: Test run concluded for t_hello_world:hello_world.sh
Sat Dec 14 23:14:52 CST 2024: Starting pytest t_pyspark:test_spark_sql_azure_blob.py
========================================================================== test session starts ===========================================================================
...
=========================================================================== 1 skipped in 0.14s ===========================================================================
Sat Dec 14 23:14:53 CST 2024: Test SUCCEEDED: t_pyspark:test_spark_sql_azure_blob.py
Sat Dec 14 23:14:53 CST 2024: Starting pytest t_pyspark:test_spark_sql_s3_with_privileges.py
========================================================================== test session starts ===========================================================================
...
============================================================== 14 passed, 27 warnings in 200.40s (0:03:20) ===============================================================
Sat Dec 14 23:18:14 CST 2024: Test SUCCEEDED: t_pyspark:test_spark_sql_s3_with_privileges.py
Sat Dec 14 23:18:14 CST 2024: Starting pytest t_pyspark:test_spark_sql_view.py
========================================================================== test session starts ===========================================================================
...
=========================================================================== 1 passed in 11.21s ===========================================================================
Sat Dec 14 23:18:26 CST 2024: Test SUCCEEDED: t_pyspark:test_spark_sql_view.py
Sat Dec 14 23:18:26 CST 2024: Starting pytest t_pyspark:test_spark_sql_gcp.py
========================================================================== test session starts ===========================================================================
...
=========================================================================== 1 skipped in 0.06s ===========================================================================
Sat Dec 14 23:18:27 CST 2024: Test SUCCEEDED: t_pyspark:test_spark_sql_gcp.py
Sat Dec 14 23:18:27 CST 2024: Starting pytest t_pyspark:test_spark_sql_basic.py
========================================================================== test session starts ===========================================================================
...
=========================================================================== 1 passed in 9.18s ============================================================================
Sat Dec 14 23:18:36 CST 2024: Test SUCCEEDED: t_pyspark:test_spark_sql_basic.py
Sat Dec 14 23:18:36 CST 2024: Starting pytest t_pyspark:test_spark_sql_s3.py
========================================================================== test session starts ===========================================================================
...
=========================================================================== 1 passed in 30.70s ===========================================================================
Sat Dec 14 23:19:08 CST 2024: Test SUCCEEDED: t_pyspark:test_spark_sql_s3.py
Sat Dec 14 23:19:08 CST 2024: Starting pytest t_pyspark:test_spark_sql_azure_dfs.py
========================================================================== test session starts ===========================================================================
...
=========================================================================== 1 skipped in 0.10s ===========================================================================
Sat Dec 14 23:19:09 CST 2024: Test SUCCEEDED: t_pyspark:test_spark_sql_azure_dfs.py
Sat Dec 14 23:19:09 CST 2024: Starting pytest t_pyspark:test_spark_sql_s3_cross_region.py
========================================================================== test session starts ===========================================================================
...
=========================================================================== 1 passed in 17.49s ===========================================================================
Sat Dec 14 23:19:27 CST 2024: Test SUCCEEDED: t_pyspark:test_spark_sql_s3_cross_region.py
Sat Dec 14 23:19:27 CST 2024: Tests completed with 0 successes and 0 failures

# docker test
➜ docker compose up --build --exit-code-from regtest
...
regtest-1  | Sun Dec 15 05:13:05 UTC 2024: Test SUCCEEDED: t_pyspark:test_spark_sql_s3.py
regtest-1  | Sun Dec 15 05:13:05 UTC 2024: Tests completed with 0 successes and 0 failures
regtest-1 exited with code 0
Aborting on container exit...

Checklist:

Please delete options that are not relevant.

• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• If adding new functionality, I have discussed my implementation with the community using the linked GitHub issue