select_union insert values writer type fails with buffered writing

[jorritsandbrink]

dlt version

0.4.7

Describe the problem

User report: https://dlthub-community.slack.com/archives/C04DQA7JJN6/p1711839048557919

The cause of this issue is the fact that buffer chunk separation does not differentiate between default and select_union insert values writes type (only logic for default is implemented).

Expected behavior

select_union insert values writer type works properly, also when the number of items exceed the configured max_buffer_size.

Steps to reproduce

os.environ["DATA_WRITER__BUFFER_MAX_ITEMS"] = "5"  # set buffer size less than number of data items
pipeline = dlt.pipeline(pipeline_name="bug", destination="synapse", full_refresh=True)
pipeline.run([{"id": i} for  i in range(10)], table_name="items") 

Operating system

Windows

Runtime environment

Local

Python version

3.8

dlt data source

No response

dlt destination

No response

Other deployment details

No response

Additional information

No response