Thoughts on Frictionless

[davidgasquez]

Sharing here some thoughts on Frictionless Specs, Framework and other tooling.

So far I've enjoyed a lot playing with Frictionless and learning more about the internal design and specs.

Pros

• The minimal implementation (simple JSON) is super lightweight and flexible enought! If the modern data tooling understood them, it could help a lot with the interoperability.
• Human and machine readable!
• I can package someone else open datasets without moving the data and add them to the Fricitonless ecosystem.

Cons

• The specs can be convoluted at some levels. Perhaps doing too many things? (See https://github.com/datopian/datahub-next/discussions/90#discussioncomment-5462884)
• Would be cool if it integrated with things like fsspec to read/write in lots of places (already considering) and Arrow as the backend to allow smarter data handling/quering and reduce the surface layer of Frictionless (IO is fsspec, data is Arrow, frictionless is just mapping).
• As far as I understand, there is no way to keep track of the dataset state over time. If the data files are on git, it could be recovered but it would be awesome if the protocol had this in mind from the start and made it simple to keep the history for the user. That's one of the coolest things from ODF. Even if you're only thinking about files and doing updates in place, the protocol is taking care of saving things properly without you having to think about it. I highly recommend this talk from the founder on the topic.

Overall I think Frictionless is right in many abstractions and would like to continue using and learning about it.

PS: The situation reminds me of Data Engineering a while back. We had the Singer Specs and people started to build tooling/connectors around it. Over time it got a lot of traction and larger products like Meltano started to spawn around it. Nowadays, thanks to the specs you have a wide variety of ways to "use" these specs (e.g: Alto). All of this to say that, from my point of view, the missing part on Frictionless now is the integration with the broader data ecosystem. The specs are there and are very helpful, we need more tooling/bridges for people to use it.

[davidgasquez]

Don't have any specific action item for this issue so feel free to close it if you don't have any follow ups @rufuspollock!

[rufuspollock]

The specs can be comvoluted at some levels. Perhaps doing too many things?

Could you say a bit more about this? Really appreciate feedback and i'm sure there are lots of places spec could be improved. 😄

[davidgasquez]

Sure, and sorry, that came off more pessimistic/agressive than I wanted. I really like the current specs. I was thinking more about the Frictionless framework specifically as I've been using it to interact with the specs.

When first encountering the framework I was confussed by some things like what are scheme/sourcess and how they differ from format (e.g: should SQL be source or a format?).

Now that I'm checking the specs I realize they don't cover things like Pipelines or Inquiries that were also making the framework rely on more concepts.

This is all generalizing from my experience (N=1) so take it with a grain of salt. Also worth mentioning again, I'm sure the specs are already pretty optimized and am very happy with the design choices made! Perhaps the improvements could come with more framework agnostic examples and documentation?

I've crossed off that point from cons as I don't think the specs are doing too many things. Thanks for the great question @rufuspollock!

[rufuspollock]

Sure, and sorry, that came off more pessimistic/agressive than I wanted. I really like the current specs. I was thinking more about the Frictionless framework specifically as I've been using it to interact with the specs.

Not at all, love ways that we can improve.

And btw: feedback on the Frictionless Framework also very useful ...

[davidgasquez]

feedback on the Frictionless Framework also very useful ...

Sure! I've been sharing all the feedback on the slack too!

[davidgasquez]

One thing I've been also thinking about the current Frictionless specs is that it doesn't have a way to diverge from it while keeping some core functionality. Not sure if that makes sense but let me share an example.

The Microsoft project Datashaper started using datapackage.json specs but they then diverged. I think I saw a couple of extra projects doing something similar. While this is great, it makes the files and standard a bit messy. You don't know which version of the datapackage the project is using. It would be interesting to have that reflected in some way in the JSON/YAML.

Taking inspiration from Kubernetes:

name: atmospheric-carbon-dioxide-trends
kind: FrictionlessDatapackage
kindVersion: 1.0.0
title: Atmospheric Carbon Dioxide Trends

Consumers/tools can check the kind and apply the proper verifications. It also promotes some kind of core properties without limiting the scope of the metadata.

I'm sure you've all considered this before and it might have undesired side effects so please share anything!