You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As understand it Parquet is a write once thing. So mutating data inside Parquet files is not an option. Now there is a new cross EU law coming in effect May 2018 that requires companies to delete data pertaining a customer if being asked to do so.
Our case is quite simple, our biggest parquet tables collect 7.5 billion rows a month. So removing data by duplicating this table whilst filtering out the unwanted customer data is not feasible.
Perhaps there is some way to remove particular data? Or perhaps there is an efficient way to do read/filter/write? Perhaps zeroing the data is an idea to not change the layout of the files.
Not sure if this is the right platform to start this discussion but I think more people will have this issue once it becomes clear that data needs to be deleted in all places, also in parquet files. Companies fase multi million dollar fines if they don't comply with GDPR.
Kai Liu:
Hi, Machiel,
Did you get any attraction on this issue? And would you mind share the approach you are taking to address GDPR requirement in your system now?
Machiel Groeneveld:
Hi [~zjumad] there is no news from the Parquet side. Though a recent development in the community to deal with this problem is Delta Lake. They add a layer on top of parquet to allow for deletions, although parquet is still read only.
Fokko Driesprong / @Fokko:
I don't see this being implemented in Apache Parquet. However, Delta lake, and also Apache Iceberg can solve the issue that you're describing. With Delta, make sure that you vacuum, otherwise the data will be still on the disks :-)
As understand it Parquet is a write once thing. So mutating data inside Parquet files is not an option. Now there is a new cross EU law coming in effect May 2018 that requires companies to delete data pertaining a customer if being asked to do so.
Our case is quite simple, our biggest parquet tables collect 7.5 billion rows a month. So removing data by duplicating this table whilst filtering out the unwanted customer data is not feasible.
Perhaps there is some way to remove particular data? Or perhaps there is an efficient way to do read/filter/write? Perhaps zeroing the data is an idea to not change the layout of the files.
Not sure if this is the right platform to start this discussion but I think more people will have this issue once it becomes clear that data needs to be deleted in all places, also in parquet files. Companies fase multi million dollar fines if they don't comply with GDPR.
Reporter: Machiel Groeneveld
Note: This issue was originally created as PARQUET-1155. Please see the migration documentation for further details.
The text was updated successfully, but these errors were encountered: