Skip to content

DB structure change + pending connection = bad sync #96

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wonder-sk opened this issue May 31, 2021 · 1 comment
Closed

DB structure change + pending connection = bad sync #96

wonder-sk opened this issue May 31, 2021 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@wonder-sk
Copy link
Contributor

There is a sync problem if the following conditions are met:

  1. using WAL mode
  2. user has modified structure of the geopackage file
  3. there is a reader (or writer) connected to the file (could be a conn from GDAL conn pool)

When client tries to checkpoint the database, it will fail because of the other reader/writer - see (1, 4, 0) return value (the first value should be zero on success), keeping the original DB file unchanged, all changes stuck in the -wal file. Also create_changeset() fails, causing upload of the raw gpkg file (without changes) + update of basefile by file copy (also without changes). 🌩️

Condition 1 can be simulated like this:

    con = sqlite3.connect(test_gpkg)
    cursor = con.cursor()
    cursor.execute('CREATE TABLE test (fid SERIAL, txt TEXT);')
    cursor.execute('INSERT INTO test VALUES (123, \'hello\');')
    cursor.execute('COMMIT;')

Condition 2 can be simlated like this:

    con2 = sqlite3.connect(test_gpkg)
    cursor2 = con2.cursor()
    cursor2.execute('select count(*) from simple;')

A sample client log:

2021-05-28 23:39:18,889 --- start push martin/test_gpkg_change
2021-05-28 23:39:20,668 got project info: local version v1 / server version v1
2021-05-28 23:39:20,668 checkpoint - going to add it in /tmp/test_gpkg_change/test.gpkg
2021-05-28 23:39:25,679 checkpoint - return value: (1, 4, 0)
2021-05-28 23:39:25,693 checkpoint - new size 98304 checksum 6ff613aaf318f8e1bdeab6abb0d9a3848b43e2eb
2021-05-28 23:39:25,698 GEODIFF: Table names are not matching between the input databases.
Base:     simple
Modified: simple, test
2021-05-28 23:39:25,699 failed to create changeset for test.gpkg
2021-05-28 23:39:25,700 push changes:
{'added': [],
 'removed': [],
 'renamed': [],
 'updated': [{'checksum': '6ff613aaf318f8e1bdeab6abb0d9a3848b43e2eb',
              'chunks': ['86acddbd-9984-45ad-b8d7-f7108b0a9bc7'],
              'mtime': datetime.datetime(2021, 5, 28, 23, 39, 18, 758328, tzinfo=tzlocal()),
              'origin_checksum': '4e567044949c26899f10a9985b3f80b7e5477111',
              'path': 'test.gpkg',
              'size': 98304}]}
2021-05-28 23:39:28,125 got transaction ID 5773573e-14e4-4f07-a0aa-cd241cd91b76
2021-05-28 23:39:28,126 will upload 1 items with total size 98304
2021-05-28 23:39:28,129 Uploading /tmp/test_gpkg_change/test.gpkg part=0
2021-05-28 23:39:30,520 Upload finished: /tmp/test_gpkg_change/test.gpkg
2021-05-28 23:39:30,521 Finishing transaction 5773573e-14e4-4f07-a0aa-cd241cd91b76
2021-05-28 23:39:32,134 updating basefile (copy) for: test.gpkg
2021-05-28 23:39:32,134 --- push finished - new project version v2
@wonder-sk
Copy link
Contributor Author

Fixed by #30 and verified with #100

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant