perf: use Narwhals expressions (and stable API)

[MarcoGorelli]

Hey, this does a couple of things:

• refactors the loop to use expressions, as that'll make the most out of backends like Polars (which, even with eager dataframes, can perform some optimisations when you use expressions as opposed to Series)
• uses the narwhals.stable.v1 api (see chore: use more robust dtype comparisons, use Narwhals stable API in tests altair#3670 for an example of where it ended up being really useful)

[MarcoGorelli]

separately from this PR, is this logic correct? note that dtype == nw.Datetime checks for any Datetime variant, including those which already have a time zone:

>>> nw.Datetime('us', 'Asia/Kathmandu') == nw.Datetime
True
>>> nw.Datetime('us') == nw.Datetime
True
>>> nw.Datetime('ns', 'UTC') == nw.Datetime
True

just like Polars does

I'm not saying that the logic here isn't correct, I'm just pointing this out in case there was any misunderstanding that can be corrected - dealing with time zones is always tricky 😄

[jonmmease]

Thanks, this was correct when I originally wrote it because VegaFusion previously only returned naive timestamps that were implicitly in UTC. But since then I've updated VegaFusion to work with timezone aware timestamps internally, and so it is now possible for it to return timezone aware timestamps.

So the correct logic now would be to use replace_time_zone only when it's a naive timestamp. What's the canonical way to test for a timezone aware timestamp in polars/narwhals?

[MarcoGorelli]

ah thanks for explaining, so is the desired logic:

• if it's tz-naive, then set to UTC and then convert to local_tz
• if it's tz-aware, then just convert to local_tz

?

If so, then just .dt.convert_time_zone(local_tz) would do the trick, as that converts as if from UTC for tz-naive data https://narwhals-dev.github.io/narwhals/api-reference/expr_dt/#narwhals.expr.ExprDateTimeNamespace.convert_time_zone

[jonmmease]

yeah, that's perfect!

[jonmmease]

Thanks!