feat: arrow based table state and checkpoint handling #4567

Job	Run time
lint	2m 38s
Python Benchmark	7m 9s
Python Build (Python 3.7 PyArrow 8.0.0)	12m 55s
Running with Python 3.7	7m 6s
Python Build (Python 3.10 PyArrow latest)	11m 56s
Running with Python 3.8	7m 22s
PySpark Integration Tests	4m 35s
Running with Python 3.9	6m 21s
Running with Python 3.10	6m 27s
Running with Python 3.11	7m 38s
	1h 14m 7s

Provide feedback