Read from multiple partitions with s3.read_json using the parition_filter

[AdrianoNicolucci]

In S3, I have a dataset with multiple partitions:

s3:/mybucket/raw/auth_logs/auth_date=20220115/file1.json
s3:/mybucket/raw/auth_logs/auth_date=20220116/file2.json
s3:/mybucket/raw/auth_logs/auth_date=20220117/file3.json

I would like to return all records from multiple partitions into a single pandas dataframe. When I attempt to return the data using the partition filter below, I only get data from a single partition instead of both. What am I missing here?

s3_path = 's3:/mybucket/raw/auth_logs/'
partition_query= lambda x: x["auth_date"] in ['20220115', '20220116']
df = wr.s3.read_json(path=s3_path, dataset=True, partition_filter=partition_query)

[AdrianoNicolucci]

I made a mistake opening this ticket, the above filter in fact does work for querying data from multiple partitions