feat(python): Support "indexed" dicts as input to from_dict and from_dicts

[alexander-beedie]

Came across some data organised like this recently. This PR allows for streamlined ingest so the caller doesn't have to unpack such "indexed" dicts manually (which, while not rocket science, can definitely be awkward/irritating to have to set up yourself every time).

Note that when indexed=True the default colname is "index", but you can also provide a string to customise/avoid clashes, eg: indexed="key").

(FYI: we can write this flavour of dataset from df.rows_by_key(…), so there's some nice symmetry in also being able to read that output back).

Example

• from_dict

  data={
      "a": {"x": [1, 2], "y": [0.5, 2.5]},
      "b": {"x": [5, 6], "y": [5.0, 1.0]},
  }
  
  df = pl.from_dict(data)
  # shape: (1, 2)
  # ┌─────────────────────┬─────────────────────┐
  # │ a                   ┆ b                   │
  # │ ---                 ┆ ---                 │
  # │ struct[2]           ┆ struct[2]           │
  # ╞═════════════════════╪═════════════════════╡
  # │ {[1, 2],[0.5, 2.5]} ┆ {[5, 6],[5.0, 1.0]} │
  # └─────────────────────┴─────────────────────┘
  
  df = pl.from_dict(data, indexed=True)
  # shape: (4, 3)
  # ┌───────┬─────┬─────┐
  # │ index ┆ x   ┆ y   │
  # │ ---   ┆ --- ┆ --- │
  # │ str   ┆ i64 ┆ f64 │
  # ╞═══════╪═════╪═════╡
  # │ a     ┆ 1   ┆ 0.5 │
  # │ a     ┆ 2   ┆ 2.5 │
  # │ b     ┆ 5   ┆ 5.0 │
  # │ b     ┆ 6   ┆ 1.0 │
  # └───────┴─────┴─────┘
  
  df = pl.from_dict(data, indexed="key")
  # shape: (4, 3)
  # ┌─────┬─────┬─────┐
  # │ key ┆ x   ┆ y   │
  # │ --- ┆ --- ┆ --- │
  # │ str ┆ i64 ┆ f64 │
  # ╞═════╪═════╪═════╡
  # │ a   ┆ 1   ┆ 0.5 │
  # │ a   ┆ 2   ┆ 2.5 │
  # │ b   ┆ 5   ┆ 5.0 │
  # │ b   ┆ 6   ┆ 1.0 │
  # └─────┴─────┴─────┘

• from_dicts

  data = {
      "a": [
          {"w": None, "x": 1, "y": 2.5, "z": None},
          {"x": 8, "y": 5.0, "w": None, "z": None},
      ],
      "b": [
          {"x": None, "y": 2.0, "w": 0, "z": 8},
          {"x": None, "y": 3.0, "w": 0, "z": 7},
      ],
      "c": [
          {"x": None, "y": None, "w": None, "z": None},
      ],
  }
  
  df = pl.from_dicts(data)
  # ShapeError: could not create a new DataFrame ...
  
  df = pl.from_dicts(data, indexed=True)
  # shape: (5, 5)
  # ┌───────┬──────┬──────┬──────┬──────┐
  # │ index ┆ w    ┆ x    ┆ y    ┆ z    │
  # │ ---   ┆ ---  ┆ ---  ┆ ---  ┆ ---  │
  # │ str   ┆ i64  ┆ i64  ┆ f64  ┆ i64  │
  # ╞═══════╪══════╪══════╪══════╪══════╡
  # │ a     ┆ null ┆ 1    ┆ 2.5  ┆ null │
  # │ a     ┆ null ┆ 8    ┆ 5.0  ┆ null │
  # │ b     ┆ 0    ┆ null ┆ 2.0  ┆ 8    │
  # │ b     ┆ 0    ┆ null ┆ 3.0  ┆ 7    │
  # │ c     ┆ null ┆ null ┆ null ┆ null │
  # └───────┴──────┴──────┴──────┴──────┘
  
  df = pl.from_dicts(data, indexed="key")
  # shape: (5, 5)
  # ┌─────┬──────┬──────┬──────┬──────┐
  # │ key ┆ w    ┆ x    ┆ y    ┆ z    │
  # │ --- ┆ ---  ┆ ---  ┆ ---  ┆ ---  │
  # │ str ┆ i64  ┆ i64  ┆ f64  ┆ i64  │
  # ╞═════╪══════╪══════╪══════╪══════╡
  # │ a   ┆ null ┆ 1    ┆ 2.5  ┆ null │
  # │ a   ┆ null ┆ 8    ┆ 5.0  ┆ null │
  # │ b   ┆ 0    ┆ null ┆ 2.0  ┆ 8    │
  # │ b   ┆ 0    ┆ null ┆ 3.0  ┆ 7    │
  # │ c   ┆ null ┆ null ┆ null ┆ null │
  # └─────┴──────┴──────┴──────┴──────┘

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79.13%. Comparing base (676f10d) to head (6ff2099).

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #18215   +/-   ##
=======================================
  Coverage   79.13%   79.13%           
=======================================
  Files        1572     1572           
  Lines      219839   219846    +7     
  Branches     2462     2464    +2     
=======================================
+ Hits       173961   173970    +9     
+ Misses      45310    45308    -2     
  Partials      568      568           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ritchie46]

I am not entirely sure about this one. Especially given that we do a full pass and a full reallocation first in Python. If we were to do this, I think it should be in a single pass.

[alexander-beedie]

I am not entirely sure about this one. Especially given that we do a full pass and a full reallocation first in Python. If we were to do this, I think it should be in a single pass.

True; in its current form it's a convenience to leverage the existing code rather than a great implementation - I'll revisit ;)