Support sub-schemas in Mermaid flowchart

[MaxHalford]

The jaffle_shop example has a kpis.finance sub-schema. It would be cool if the Mermaid output handles any level of depth for (sub-)schemas.

[LeonardoNatale]

Is this what you have in mind?

flowchart TB
subgraph analytics
analytics.kpis(kpis)

subgraph finance
analytics.finance.kpis(kpis)
end
end

subgraph core
core.customers(customers)
core.orders(orders)
end

subgraph staging
staging.orders(orders)
staging.customers(customers)
staging.payments(payments)
end

    core.orders --> analytics.finance.kpis
    core.customers --> analytics.kpis
    core.orders --> analytics.kpis
    staging.customers --> core.customers
    staging.orders --> core.customers
    staging.payments --> core.customers
    staging.orders --> core.orders
    staging.payments --> core.orders

Loading

[MaxHalford]

Yes!

[LeonardoNatale]

I was thinking of some recursive approach and what data structure would support it.

If we had the data in such a format ⬇️

data = {
    "analytics": ["kpis", {"finance": ["kpis"]}],
    "core": ["customers", "orders"],
    "staging": ["orders", "customers", "payments"] 
}

then something like this would work well ⬇️

def format_data(data: dict[str, list[str | dict]]):
    for schema, values in data.items():
        print_schema(schema, values)

def print_schema(schema: str, values: list[str | dict], prefix: str = ""):
    print(f"\nsubgraph {schema}")
    if isinstance(values, list):
        for value in values:
            if isinstance(value, dict):
                (sub_schema, sub_values), = value.items()
                new_prefix = f"{prefix}.{schema}" if prefix else schema
                print_schema(sub_schema, sub_values, new_prefix) # RECURSIVE STEP
            else:
                path = ".".join(filter(None, [prefix, schema, value]))
                print(f"{path}({value})")
    print("end")

Output ⬇️

flowchart TB
subgraph analytics
analytics.kpis(kpis)

subgraph finance
analytics.finance.kpis(kpis)
end
end

subgraph core
core.customers(customers)
core.orders(orders)
end

subgraph staging
staging.orders(orders)
staging.customers(customers)
staging.payments(payments)
end

❓ @MaxHalford What do you think?
In particular:

• where would you suggest to make this formatting? I am a bit skeptical of putting everything inside private methods in the DAGOfViews class. I would rather have a util function somewhere to perform the first step (from set of dependencies -> to nested dict).

[MaxHalford]

where would you suggest to make this formatting? I am a bit skeptical of putting everything inside private methods in the DAGOfViews class. I would rather have a util function somewhere to perform the first step (from set of dependencies -> to nested dict).

I think it's ok to have this stuff in DagOfViews. The latter is already taking in a set of dependencies and turning it into a DAG. What you need is simply a different view of this DAG. If I were you, I would add a new private method to DagOfViews, and then use it in the _to_mermaid_views method. But feel free to do something else if you can come up with a better proposal :)

[MaxHalford]

Done in #13, example here. Thanks @LeonardoNatale!