core_eia860m__changelog_generators contains NULL valid_until_date values

[TrentonBush]

Describe the bug

core_eia860m__changelog_generators contains 950+ NULL values in the valid_until_date column. I don't think they should exist because this column is derived from report_date and none of those are NULL. Most of the NULLs (700+) are from the last report_date in the series (currently 2023-12-01), but ~200 others exist throughout the time span of the data. Maybe it has to do with the last date in each generator's series?

Bug Severity

This table is encoded as a changelog between two dates -- valid_until_date specifies one of the boundaries. It is hard to interpret the data without that boundary.

To Reproduce

SELECT
    valid_until_date,
    count(*) as count
FROM core_eia860m__changelog_generators
GROUP BY 1
ORDER BY 1 desc

Software Environment?

in pudl.sqlite version v2024.2.6

[cmgosnell]

Just checked and all of the max dates do indeed have null valid_until_date:

because of this line

i tried deleting it and now we have full coverage:

[TrentonBush]

eia860m_changelog["valid_until_date"] = (
    eia860m_changelog.sort_values(gens_idx, ascending=False)
    .groupby(gen_idx_no_date)["report_date"]
    .transform("shift")
    .fillna(eia860m_changelog.report_date_max)
    .where(eia860m_changelog["report_date"] != eia860m_changelog["report_date_max"])
)

I think that shift/fillna(max_date) combo assumes that all the timeseries are contiguous and end at the most recent reporting date. I think some of them end earlier or have reporting gaps, so the fillna() value might be a little more complicated

[cmgosnell]

we use pudl.helpers.expand_timeseries before this to make sure there is a continuous time series! and iirc the reason we added this last where clause is because we wanted to define the valid_until_date as the following:

"The record in the changelog is valid until this date. The record is valid from the report_date up until but not including the valid_until_date."

If we want to keep this definition then the where clause should probably stay as is and we should fill the records in the where with report_date + one month.

[TrentonBush]

If we want to keep this definition then the where clause should probably stay as is and we should fill the records in the where with report_date + one month.

That makes sense to me!