feat: arrow based table state and checkpoint handling #4571

Job	Run time
Python Benchmark	4m 2s
lint	2m 42s
Python Build (Python 3.10 PyArrow latest)	2m 52s
Running with Python 3.7	4m 10s
Python Build (Python 3.7 PyArrow 8.0.0)	5m 10s
PySpark Integration Tests	4m 10s
Running with Python 3.8	6m 54s
Running with Python 3.9	7m 11s
Running with Python 3.10	4m 9s
Running with Python 3.11	4m 4s
	45m 24s

Provide feedback