Upgrade from 9.1.2 -> 10.0.0 succeeds but org is lost #1240

brunis · 2022-01-05T11:46:48Z

Version

21.5.0

Steps to Reproduce

Upgrade 9.1.2 to 21.5.0 as per hard-stop docs (https://develop.sentry.dev/self-hosted/releases/#hard-stops)
Everything works (even Postgres db migration)

Expected Result

A running sentry with my data from Sentry 9.1.2

Actual Result

Fresh start, no data.

brunis · 2022-01-05T11:47:55Z

sentry-migrate-log.txt

brunis · 2022-01-06T09:20:57Z

Just retried the 9.1.2 -> 10.0.0 upgrade, it also succesfully migrates the database, but when you start up Sentry v10, the database is wiped/cleared/ignored or whatever it does.
This is the prompt from the cmdline after migration:

09:14:44 [WARNING] sentry: Cannot initiate onboarding for organization (1) due to missing owners
Created internal Sentry project (slug=internal, id=1)

Would you like to create a user account now? [Y/n]:

chadwhitacre · 2022-01-06T16:44:51Z

▶ Setting up / migrating database ...

These lines in sentry-migrate-log.txt suggest that your working copy is at a later version than 10.0.0. You should git checkout 10.0.0 before running install.sh. What does git log -n1 show?

brunis · 2022-01-07T07:08:08Z

▶ Setting up / migrating database ...
These lines in sentry-migrate-log.txt suggest that your working copy is at a later version than 10.0.0. You should git checkout 10.0.0 before running install.sh. What does git log -n1 show?

yes, the original post was not wrong, it was a 9.1.2 -> 21.5.0 upgrade.. as the hard-stop docs says..

My comment was about trying an intermediate upgrade -> 10.0.0 first.. hoping db migrations would work, but they already fail there.. so it doesn't matter what i upgrade to.. it can't migrate db properly.

brunis · 2022-01-07T07:19:15Z

▶ Setting up / migrating database ...
These lines in sentry-migrate-log.txt suggest that your working copy is at a later version than 10.0.0. You should git checkout 10.0.0 before running install.sh. What does git log -n1 show?

so what do i do now.. just accept that your upgrade path doesn't work? or i send you a copy of my database, so you can confirm the bugs and fix them?

chadwhitacre · 2022-01-07T15:09:37Z

At this point with just yourself reporting an issue, I need to leave the debugging to you until it's clearer that there is indeed a bug on our side. Until then I'm happy to be a rubber duck and help you whittle down a test case! 😁

Since the misbehavior occurs with v10.0.0, let's focus on that version.

Just retried the 9.1.2 -> 10.0.0 upgrade, it also succesfully migrates the database, but when you start up Sentry v10, the database is wiped/cleared/ignored or whatever it does.

Based on my reading of this comment, are we looking at a bug report like this, then ... ?

Steps to Reproduce

Start with 9.1.2 and a populated database—can we use some SELECT count(*) from events or whatever as a baseline test?
git checkout 10.0.0
install.sh - finishes successfully (yes? not clear to me yet ...)
Confirm that the database is successfully migrated—how are we confirming this? same SQL as (1)?
Start up Sentry with docker compose up.

Expected Result

Database stays populated, i.e., SELECT count(*) FROM events returns NNN.

Actual Result

Database is wiped, i.e., SELECT count(*) FROM events returns 0.

brunis · 2022-01-10T05:50:35Z

At this point with just yourself reporting an issue, I need to leave the debugging to you until it's clearer that there is indeed a bug on our side. Until then I'm happy to be a rubber duck and help you whittle down a test case! 😁

That is perfectly fine!

Since the misbehavior occurs with v10.0.0, let's focus on that version.

Agreed.

Just retried the 9.1.2 -> 10.0.0 upgrade, it also succesfully migrates the database, but when you start up Sentry v10, the database is wiped/cleared/ignored or whatever it does.

Based on my reading of this comment, are we looking at a bug report like this, then ... ?

Steps to Reproduce
1. Start with 9.1.2 and a populated database—**can we use some `SELECT count(*) from events` or whatever as a baseline test?**

You're the expert, if events survives a migration, i'll do a count on that one. I'll have to figure out how to get into the pgsql container and do that from the commandline.

2. `git checkout 10.0.0`

3. `install.sh` - finishes successfully (**yes? not clear to me yet ...**)

Yes, it finishes, but the issue i mentioned here #1240 (comment) tell's me something doesn't migrate properly, sounds like the organization looses ownership.

4. Confirm that the database is successfully migrated—**how are we confirming this? same SQL as (1)?**

Yeah, I will also test any organization and project tables to see if they are still there.

5. Start up Sentry with `docker compose up`.
Expected Result

Database stays populated, i.e., SELECT count(*) FROM events returns NNN.

Actual Result

Database is wiped, i.e., SELECT count(*) FROM events returns 0.

Well, i don't think it's wiped (the data is still approx. 1GB uncompressed after migration), but something related to the organization and users loose relationship/ownership and it wants to start fresh.

I'll get back to you when i'm done.

brunis · 2022-01-10T06:10:20Z

There is no events table, but i looked at the team, organization and project tables and everything is intact, at least in those tables.

sentry=# select * from sentry_team;
 id |    slug     |     name     |          date_added           | status | organization_id
----+-------------+--------------+-------------------------------+--------+-----------------
  1 | sentry      | Sentry       | 2018-04-05 10:47:27.659706+00 |      0 |               1
  2 | innomate-as | INNOMATE a/s | 2018-04-05 10:59:14.329622+00 |      0 |               1
(2 rows)

sentry=# select * from sentry_organization;
 id |     name     | status |          date_added           |   slug   | flags | default_role
----+--------------+--------+-------------------------------+----------+-------+--------------
  1 | Innomate a/s |      0 | 2018-04-05 10:47:27.650583+00 | innomate |     1 | member
(1 row)

sentry=# select * from sentry_project;
 id |                name                | public |          date_added           | status |        slug        | organization_id |      first_event       | forced_co
lor | flags |  platform
----+------------------------------------+--------+-------------------------------+--------+--------------------+-----------------+------------------------+----------
----+-------+------------
  4 | PublicPages                        | f      | 2018-04-09 13:49:39.573184+00 |      0 | publicpages        |               1 | 2018-05-15 07:35:13+00 |
    |     1 | javascript
  1 | Internal                           | f      | 2018-04-05 10:47:27.661645+00 |      0 | internal           |               1 | 2018-04-05 20:52:19+00 |
    |     1 |
  6 | IntegrationService                 | f      | 2018-11-06 10:37:43.631511+00 |      0 | integrationservice |               1 | 2018-11-07 11:00:31+00 |
    |     0 | csharp
  7 | Azure AppService @ hr.innomate.net | f      | 2019-02-19 08:50:43.10292+00  |      0 | app_service        |               1 | 2019-02-21 12:26:48+00 |
    |     1 | csharp
  3 | INNOMATE@HR                        | f      | 2018-04-05 11:00:13.586215+00 |      0 | innomate_hr        |               1 | 2018-04-06 15:59:41+00 |
    |     1 | csharp
(5 rows)

sentry=# select * from auth_user;

and the user table contains our users and all their information, email, passwords, etc..

so, which tables can i look at to figure out why the installer wants to start fresh thinking there are no users..?

brunis · 2022-01-10T06:20:05Z

Unsure of the relevance of this one.

sentry=# select * from sentry_authidentity;
 id | user_id | auth_provider_id | ident | data | date_added | last_verified | last_synced
----+---------+------------------+-------+------+------------+---------------+-------------
(0 rows)

But this looks bad, every project should have an owner here..?

sentry=# select * from sentry_projectownership;
 id | project_id | raw | schema | fallthrough | date_created | last_updated | is_active
----+------------+-----+--------+-------------+--------------+--------------+-----------
(0 rows)

This looks correct:

sentry=# select * from sentry_projectteam;
 id | project_id | team_id
----+------------+---------
  1 |          1 |       1
  3 |          3 |       2
  4 |          4 |       2
  6 |          6 |       2
  7 |          7 |       2
(5 rows)

And this looks correct:

sentry=# select * from sentry_organizationmember;
 id | organization_id | user_id | type |          date_added           | email | has_global_access | flags |  role  |                              token
                 | token_expires_at
----+-----------------+---------+------+-------------------------------+-------+-------------------+-------+--------+-------------------------------------------------
-----------------+------------------
  1 |               1 |       1 |   50 | 2018-04-05 10:48:00.214304+00 |       | t                 |     0 | owner  |
                 |
  2 |               1 |       2 |   50 | 2018-04-09 11:55:30.861505+00 |       | t                 |     0 | admin  | bd2e8c4c6f6a46e882949126b69c43589080ef1ee0fb4c1a
af4d8b5a8621a3c9 |
  3 |               1 |       3 |   50 | 2018-04-09 11:55:53.221391+00 |       | t                 |     0 | member | 44e0e2eff06b421786724f1396deb259396373f5686a4e44
ba3ece52adc6834b |
  6 |               1 |       4 |   50 | 2018-04-12 07:08:09.990141+00 |       | t                 |     0 | member | c80100dbb03c40ae9c74b4c8c8698615df5f43903e46442e
b927a34ecac5e894 |
  7 |               1 |       5 |   50 | 2018-04-12 07:08:41.368408+00 |       | t                 |     0 | member | 868ebf78a11b4b9683a032215bff7594780363ac43a34b9d
acf78702886f8b27 |
(5 rows)

brunis · 2022-01-10T11:25:10Z

ok, restored my old 9.1.2 PG_VERSION=9.5 db to check.. and the content of the tables are exactly the same.. was trying to help but i don't know where to look or what to look for ..

brunis · 2022-01-10T13:11:42Z

But as promised:
PG 9.5 DB Sentry 9.1.2

sentry=# select count(*) from sentry_eventuser;
 count
-------
  8307
(1 row)

sentry=# select count(*) from sentry_eventmapping;
 count
--------
 317507
(1 row)

PG 9.6 DB Sentry 10.0.0:

sentry=# select count(*) from sentry_eventuser;
 count
-------
  8307
(1 row)

sentry=# select count(*) from sentry_eventmapping;
 count
--------
 317507
(1 row)

chadwhitacre · 2022-01-19T15:07:21Z

@bergpb I added #1240 as a second report in the description on this ticket.

Now to whittle down the repro so we can identify the issue ...

chadwhitacre · 2022-01-19T15:25:49Z

Eep, just kidding! #1252 has a different actual result (missing issues). 😬

Well, i don't think it's wiped (the data is still approx. 1GB uncompressed after migration), but something related to the organization and users loose relationship/ownership and it wants to start fresh.

Right, so the report is:

Steps to Reproduce

Start with 9.1.2 and a populated database
git checkout 21.5.0
install.sh

Expect Result

install.sh completes successfully without prompting for a new user.

Actual Result

09:14:44 [WARNING] sentry: Cannot initiate onboarding for organization (1) due to missing owners
Created internal Sentry project (slug=internal, id=1)

Would you like to create a user account now? [Y/n]:

Yes?

Looking at the code, here is the query that is failing:

Organization.objects.get(id=project.organization_id).get_default_owner().

Maybe there is a clue in there? What is the value of [a SQL approximation of] that query before and after running install.sh? If you see a different before and after, can you bisect the migrations (run them manually using the sentry django CLI?) to narrow down which migration is causing the unexpected change?

Another idea: you could try installing a clean 9.1.2 and then upgrading to 21.5.0, and (assuming it doesn't lose the org) inspecting the before/after to ensure we understand the "good" state.

Also, have you reviewed the search results for the error message to see if there are some useful tips or clues there?

brunis · 2022-01-20T08:55:26Z

@chadwhitacre yes, the changes you made are correct, we just can't afford spending more time on it, so we've decided to just start fresh. We'll keep our old 9.1.2 docker images/database if we should need to investigate old issues.

chadwhitacre · 2022-01-20T14:30:41Z

Sounds reasonable. Good luck @brunis! :)

Aleksazhko · 2022-01-31T13:38:35Z

Hello there, guys!
I've tried to find out the way i should resolve my task, but no luck.
Currently i have standalone Sentry 7.7.0 installation with huge amount of data. I need to upgrade it to the latest version, but i have to save all the projects and preferences we have. Old history and another events are not needed. I know about hard stops, but no idea regarding where to start.
Thanx in advance!

chadwhitacre · 2022-01-31T13:52:35Z

If you know about the hard stops then you probably know about our upgrade documentation. Where better place to start? :)

If you hit specific problems please open a new ticket. Good luck!

Aleksazhko · 2022-01-31T14:09:26Z

Appreciate your quick answer. But what if a have non-Docker version? Documentation looks like docker-related only

chadwhitacre · 2022-01-31T15:31:18Z

We only support the Docker version in this repo and in our documentation. Best to seek support from whomever made the version you are using, I think.

getsantry bot added the Status: Untriaged label Jan 5, 2022

brunis mentioned this issue Jan 5, 2022

ERROR: for relay Container "60f6f0106686" is unhealthy. #1235

Closed

chadwhitacre added the Type: Support Request label Jan 5, 2022

getsantry bot removed the Status: Untriaged label Jan 5, 2022

brunis changed the title ~~Upgrade from 9.1.2 -> 21.5.0 succeeds but no data~~ Upgrade from 9.1.2 -> 10.0.0 succeeds but no data Jan 10, 2022

chadwhitacre mentioned this issue Jan 12, 2022

Lost Issues/Events After upgrade from version 9.0 to 21.5 #1252

Closed

chadwhitacre changed the title ~~Upgrade from 9.1.2 -> 10.0.0 succeeds but no data~~ Upgrade from 9.1.2 -> 10.0.0 succeeds but org is lost Jan 19, 2022

chadwhitacre closed this as completed Jan 20, 2022

github-actions bot locked and limited conversation to collaborators Feb 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade from 9.1.2 -> 10.0.0 succeeds but org is lost #1240

Upgrade from 9.1.2 -> 10.0.0 succeeds but org is lost #1240

brunis commented Jan 5, 2022 •

edited by chadwhitacre

Loading

brunis commented Jan 5, 2022

brunis commented Jan 6, 2022 •

edited

Loading

chadwhitacre commented Jan 6, 2022 •

edited

Loading

brunis commented Jan 7, 2022

brunis commented Jan 7, 2022

chadwhitacre commented Jan 7, 2022

brunis commented Jan 10, 2022

Steps to Reproduce

Expected Result

Actual Result

brunis commented Jan 10, 2022 •

edited

Loading

brunis commented Jan 10, 2022 •

edited

Loading

brunis commented Jan 10, 2022

brunis commented Jan 10, 2022

chadwhitacre commented Jan 19, 2022

chadwhitacre commented Jan 19, 2022 •

edited

Loading

brunis commented Jan 20, 2022

chadwhitacre commented Jan 20, 2022

Aleksazhko commented Jan 31, 2022

chadwhitacre commented Jan 31, 2022

Aleksazhko commented Jan 31, 2022 •

edited

Loading

chadwhitacre commented Jan 31, 2022

Upgrade from 9.1.2 -> 10.0.0 succeeds but org is lost #1240

Upgrade from 9.1.2 -> 10.0.0 succeeds but org is lost #1240

Comments

brunis commented Jan 5, 2022 • edited by chadwhitacre Loading

Version

Steps to Reproduce

Expected Result

Actual Result

brunis commented Jan 5, 2022

brunis commented Jan 6, 2022 • edited Loading

chadwhitacre commented Jan 6, 2022 • edited Loading

brunis commented Jan 7, 2022

brunis commented Jan 7, 2022

chadwhitacre commented Jan 7, 2022

Steps to Reproduce

Expected Result

Actual Result

brunis commented Jan 10, 2022

Steps to Reproduce

Expected Result

Actual Result

brunis commented Jan 10, 2022 • edited Loading

brunis commented Jan 10, 2022 • edited Loading

brunis commented Jan 10, 2022

brunis commented Jan 10, 2022

chadwhitacre commented Jan 19, 2022

chadwhitacre commented Jan 19, 2022 • edited Loading

Steps to Reproduce

Expect Result

Actual Result

brunis commented Jan 20, 2022

chadwhitacre commented Jan 20, 2022

Aleksazhko commented Jan 31, 2022

chadwhitacre commented Jan 31, 2022

Aleksazhko commented Jan 31, 2022 • edited Loading

chadwhitacre commented Jan 31, 2022

brunis commented Jan 5, 2022 •

edited by chadwhitacre

Loading

brunis commented Jan 6, 2022 •

edited

Loading

chadwhitacre commented Jan 6, 2022 •

edited

Loading

brunis commented Jan 10, 2022 •

edited

Loading

brunis commented Jan 10, 2022 •

edited

Loading

chadwhitacre commented Jan 19, 2022 •

edited

Loading

Aleksazhko commented Jan 31, 2022 •

edited

Loading