Low hanging fruit changes:

• polars update to conda env
• try with bigger memory partition on midway (will take a few days to process)

Refactoring changes:
-inner joins instead of left joins - change logic to start with sales

• minimize # of cols before the join (i.e., we only need a couple columns from prop).
• ensure that everything is filtered before the join to reflect criteria in the final universe.
• (last priority change) break up the joins into separate single scan + join + sink of each file creating multiple staging tables (instead of running multiple joins at the same time)

with these changes, a scan + sink + join pattern should work!

Next steps:

• get local version working again with these changes (clarity on join logic)
• deploy small batch on midway
• set up deployment of larger file on bigmem