Structured Streaming

[nightscape]

Spark supports using SQL in Structured Streaming by registering the stream source as temp view.
This should in principle allow using DBT to create Spark Structured Streaming pipelines, right?
Did someone already try this? Am I missing something obvious?
Thank you for any and all input 😃

[nightscape]

@VersusFacit I saw you did some work on adapters. Would you know if there's something obvious that would make this harder than expected?

[dataders]

in theory, you're right that this is possible @nightscape! I'd be curious to hear the opinion of @Fokko and @JCZuurmond wrt both feasibility as well as importance

[nightscape]

Related proposal in Spark to allow defining streaming sources in SQL and the corresponding design docs.