[v3.1.2] Cover letters created before migrating to v3 show up separatly in patch list

[alialnu]

Example:
This cover letter was part of this series before the migration, but it no longer is. Moreover, its state now says None.

There are many other covers with the same issue.

EDIT:
The series above actually has a cover letter as part of it, but the cover letter is listed again in the patch list:
This one is part of the series.
This one isn't.

This isn't consistent with patchsets that were created after the migration.

The issue appeared when I migrated from v2.2.6 to v3.1.2.

[alialnu]

New replies to cover letters created prior to the migration from v2 to v3 will not appear in the cover letter that is part of the series and will not trigger a cover-comment-created event.

[alialnu]

@stephenfin, do you think this issue could be resolved in a new migration? Rolling back could be problematic because the recent migrations aren't reversible. Any suggestion?

[stephenfin]

In theory, yes, migrations can fix anything in the database. I need to find time to look into this though. Perhaps this weekend.

Are new cover letters to new series (created after the migration) getting linked correctly?

[stephenfin]

First pass suggests I've missed something in the migration. That migration got rid of the Submission model, which Patch and Cover inherited from (read: JOINed to) in favour of wholly separate models. My guess is that we've ended up with a row in both tables.

[alialnu]

Are new cover letters to new series (created after the migration) getting linked correctly?

Yes new ones are linked correctly.
New cover letters (created after the migration) only have a /project/<Project>/cover/<Message-Id> web url.
Old cover letters (created before the migration) have /project/<Project>/cover/<Message-Id> and /project/<Project>/patch/<Message-Id> web urls.

When someone replies to old cover letters, the reply appears in the /project/<Project>/patch/<Message-Id> url, which isn't part of a series.

[stephenfin]

First pass suggests I've missed something in the migration. That migration got rid of the Submission model, which Patch and Cover inherited from (read: JOINed to) in favour of wholly separate models. My guess is that we've ended up with a row in both tables.

Yeah, looks like it's this: I forgot to clean up the old cover letter-related entries in the patchwork_submission table before it got renamed to patchwork_patch (migration 0043).

@alialnu Can you check something for me: do the following commands returns the same count?

SELECT COUNT(patch.id) FROM patchwork_patch patch WHERE ISNULL(patch.state_id);
SELECT COUNT(cover.id) FROM patchwork_cover cover;

And maybe for safety, check the exact IDs also.

SELECT patch.id FROM patchwork_patch patch WHERE ISNULL(patch.state_id);
SELECT cover.id FROM patchwork_cover cover;

Cover letters don't have a state but patches always do, so in theory any entry in patchwork_patch with a null state_id column should be left-over cover letters. I'd like to sanity check this against a real deployment though.

If this works, our apparent fix would be to identify any rows in patchwork_patch_comment that have foreign keys pointing to one of these rows in patchwork_patch, copy them to patchwork_covercomment, then delete the offending rows in patchwork_patch (which should cause the patchwork_patchcomment rows to be deleted since we have a CASCADE in place) after we've deleted the rows in patchwork_patchcomment since Django doesn't do database-level cascading. Sound reasonable?

EDIT: Effectively this query:

SELECT patch_comment.id
FROM patchwork_patchcomment patch_comment
WHERE patch_comment.patch_id IN (
    SELECT patch.id FROM patchwork_patch patch WHERE ISNULL(patch.state_id)
);

[stephenfin]

I think the following script should do the trick.

# copy any recent patch comments that should have been assigned to a cover letter instead
INSERT INTO patchwork_covercomment (msgid, date, headers, content, cover_id, submitter_id)
SELECT msgid, date, headers, content, patch_id, submitter_id
FROM patchwork_patchcomment
WHERE patchwork_patchcomment.patch_id IN (
    SELECT patch.id FROM patchwork_patch patch WHERE ISNULL(patch.state_id)
);

# delete the duplicate patch comments
DELETE patch_comment
FROM patchwork_patchcomment patch_comment
WHERE patch_comment.patch_id IN (
    SELECT patch.id FROM patchwork_patch patch WHERE ISNULL(patch.state_id)
);

# delete the duplicate patches
DELETE patch FROM patchwork_patch patch WHERE ISNULL(patch.state_id);

Assuming so, I can codify this in a migration script and push a new migration. The only thing is that Django doesn't appear to provide a mechanism to insert a migration between two older migrations (unlike Alembic) so you're either going to need to update to current HEAD, wait for a new release, or run it manually (it should be idempotent so there'll be no issues if you did it manually and later upgraded to a release with this migration in it)

[stephenfin]

I've opened a bug against Django since this isn't the behavior I expected. I expected the deletion of the CoverLetter model to drop all rows in the parent Submission table. This clearly didn't happen.

https://code.djangoproject.com/ticket/34796

[alialnu]

Thanks @stephenfin for helping with this,

@alialnu Can you check something for me: do the following commands returns the same count?

I forgot to mention that I'm using postgresql. I made some changes to the queries:

SELECT patch.id FROM patchwork_patch patch WHERE patch.state_id IS NULL;
   id
--------
  61292
  77881
  [..]
  129709
  129712
(8138 rows)

SELECT cover.id FROM patchwork_cover cover;
   id
--------
 1
 2
 [..]
 129709
 129712
(8222 rows)

They aren't returning the same count.

EDIT: Effectively this query:

SELECT patch_comment.id
FROM patchwork_patchcomment patch_comment
WHERE patch_comment.patch_id IN (
    SELECT patch.id FROM patchwork_patch patch WHERE patch.state_id IS NULL
);
   id
--------
 162805
 162984
 [..]
 163347
 163359
(26 rows)

I think the following script should do the trick.

Removing duplicate patch comments is failing for me:

# copy any recent patch comments that should have been assigned to a cover letter instead
INSERT INTO patchwork_covercomment (msgid, date, headers, content, cover_id, submitter_id)
SELECT msgid, date, headers, content, patch_id, submitter_id
FROM patchwork_patchcomment
WHERE patchwork_patchcomment.patch_id IN (
    SELECT patch.id FROM patchwork_patch patch WHERE patch.state_id IS NULL
);

Output:

INSERT 0 26

# delete the duplicate patch comments
DELETE FROM patchwork_patchcomment AS patch_comment
WHERE patch_comment.patch_id IN (
    SELECT patch.id FROM patchwork_patch patch WHERE patch.state_id IS NULL
);

Output:

ERROR:  update or delete on table "patchwork_patchcomment" violates foreign key constraint "patchwork_event_patch_comment_id_4123011e_fk_patchwork" on table "patchwork_event"
DETAIL:  Key (id)=(162805) is still referenced from table "patchwork_event".

[stephenfin]

Whoops, I forgot about the events. How about if you add the following after the first step?

# migrate any erroneous patch-comment-created events to cover-comment-created events
UPDATE patchwork_event
SET cover_id = patch_id, category = 'cover-comment-created', patch_id = NULL
WHERE patch_id IN (
    SELECT patch.id FROM patchwork_patch patch WHERE patch.state_id IS NULL
);

[stephenfin]

Thanks @stephenfin for helping with this,

@alialnu Can you check something for me: do the following commands returns the same count?

I forgot to mention that I'm using postgresql. I made some changes to the queries:

SELECT patch.id FROM patchwork_patch patch WHERE patch.state_id IS NULL;
   id
--------
  61292
  77881
  [..]
  129709
  129712
(8138 rows)

SELECT cover.id FROM patchwork_cover cover;
   id
--------
 1
 2
 [..]
 129709
 129712
(8222 rows)

They aren't returning the same count.

Ah, right, they will be different since you have new cover letters. You probably need to add a limit to this. We don't track our own created_at/updated_at fields so we need to use date as a stand-in. This should do the trick (or at least the PostgreSQL equivalent will):

SELECT patch.id FROM patchwork_patch patch WHERE patch.state_id IS NULL;

SELECT cover.id
FROM patchwork_cover cover
WHERE cover.date <= (
    SELECT MAX(patch.date) FROM patchwork_patch patch WHERE patch.state_id IS NULL
);

I'm pretty confident that filtering on null state_id will do the trick now though, with what you've already provided.