Update Django Framework from 2.2.27 to 4.2 LTS

[chris-bateman]

How did you install WebODM (docker, installer, etc.)?

N/A

What's your browser and operating system? (Copy/paste the output of https://www.whatismybrowser.com/)

N/A

What is the problem?

Django framework is EOL. WebODM uses 2.2.27 and the current LTS is 4.2
Placeholder issue to begin discussion on upgrading the framework.

How can we reproduce this? (What steps trigger the problem? What parameters are you using for processing? Include screenshots. If you are having issues processing a dataset, you must include a copy of your dataset uploaded on Dropbox, Google Drive or https://dronedb.app)

Review Django version here -

WebODM/requirements.txt

Line 7 in 8150b82

Django==2.2.27

4.2 LTS makes sense support-wise, but v5 is also out.

[chris-bateman]

EOL notes - https://www.djangoproject.com/download/ and https://endoflife.date/django

[Saijin-Naib]

Do you know of any gotchas? API breaks?

Keeping current on supported dependencies would be wonderful.

[chris-bateman]

So this doco covers the process - https://docs.djangoproject.com/en/5.0/howto/upgrade-version/

Example incompatible notes - https://docs.djangoproject.com/en/5.0/releases/4.2/#backwards-incompatible-changes-in-4-2

And a good list here as well - https://docs.djangoproject.com/en/5.0/internals/deprecation/

[chris-bateman]

I started to make a little progress on this. I am playing dependency wack a mole.
So with that in mind it might be time to audit the current modules located in this file -
https://github.com/OpenDroneMap/WebODM/blob/master/requirements.txt

The main focus is identifying packages that are deprecated/abandoned with no clear support from their maintainer/community.
Some have suitable alternatives noted in their doco but others will require some research to find a suitable replacement.

django-guardian

rfc3987 - Github repo was archived on the 12th Jan 2024 but moved to a new repo recently.
https://codeberg.org/atufi/rfc3987/

djangorestframework-jwt - Github archived
Alternatives - jpadilla/django-rest-framework-jwt#484

itypes - Github archived

django-rest-framework-guardian

strict-rfc3339

openapi-codec - Github archived

django-codemirror2 - After 7 years of no activity has had a few updates this year.

piexif

appdirs - Deprecated

[smathermather]

This is great Chris. Anything you need to help, e.g. identifying alternatives? I would say, for anything that has a known upgrade path, feel free to upgrade and test, and send me a relevant repo to review and I'm happy to run in production on my end and help find any issues.

[NtskwK]

Do you know of any gotchas? API breaks?

Keeping current on supported dependencies would be wonderful.

Support for the previous LTS, Django 3.2, will end in April 2024. Then, upstream support for PostgreSQL 11 ends in November 2023. Django 4.2 supports PostgreSQL 12 and higher.

But the EOL of PostgreSQL 12 is coming soon in November 14, 2024.

It may be a lot of challenges.

Whether should try to upgrade them to the latest LTS version or others versions?

[NtskwK]

Hello everyone!

Great news! The migration work has made some progress, including Django 4.2 LTS, PostgreSQL 16, and more. We still need to conduct further testing on functions. Could you create a new branch for this purpose, please. I will creat a PR and advance the work to the next stage.

Thank you! @Saijin-Naib @smathermather

[smathermather]

That is fantastic news! Thanks for continuing on this path.

I defer to Piero on workflow, but we typically just keep the main branch except for hot fixes, and then merge from branches on forks when testing is complete. If you let me know what fork/branch you have for testing, I can build and test as a start.

[NtskwK]

I have solved a part compatibility problems to sure it can work in the development environment.

But there are also many different from django-rest-framework-jwt to djangorestframework-simplejwt. So it can not pass the test in webodm.sh.

I don't really understand about auth function. I'm sorry.

[NtskwK]

That is fantastic news! Thanks for continuing on this path.

I defer to Piero on workflow, but we typically just keep the main branch except for hot fixes, and then merge from branches on forks when testing is complete. If you let me know what fork/branch you have for testing, I can build and test as a start.

My apologies. I noticed that there is a actions for Build Docker Image and Public. I don't know whether it can bring dangers to other users with some Breaking change. In my opinions that we should be more careful.

If you think it doesn't matter, that I will do it soon.

[smathermather]

Thanks for the pull request!

We can run workflows on pull requests without merging, they just don't run by default. Once the pull request is far enough along and reviewed, I would expect we'll also test it in a workflow before merging.

[NtskwK]

#1543 @pierotofy
There is a problem with the recover database migrations.

django.db.utils.ProgrammingError: type "raster" does not exist
LINE 1: ...d_control_points" varchar(100) NULL, "orthophoto" raster NUL...

I found the field orthophoto in models.Task was been add in 2016, then deleted in 2017.

app\migrations\0001_initial.py

 79 |      ('orthophoto', django.contrib.gis.db.models.RasterField(blank=True, help_text='Orthophoto created by OpenDroneMap', null=True, srid=4326)),

app\migrations\0004_auto_20170707_1014.py

    operations = [
        migrations.RemoveField(
            model_name='task',
            name='orthophoto',
        ),

python manage.py migrate will perform all the operations in turn. It could have to remove database migrations files if we don't change app\migrations\0001_initial.py with create postgis_raster extension.
If you plan to use PostGIS raster functionality, you should also activate the postgis_raster extension.

[NtskwK]

TODO: update Ubuntu to 24.04 LTS or 22.04 LTS

I'll have a try. checked

PDAL has not ready to fit the Ubuntu 24.04, but 22.04 is ok.

apt of Ubuntu 24.04
E: Unable to locate package pdal

[NtskwK]

sed: can't read /usr/include/proj_api.h: No such file or directory

The behavior of libproj-dev did not inexpectations on 24.04 even 22.04. What is the purpose of this command? Is this necessary? @pierotofy @chris-bateman #1545

	tar -zxf postgis-$POSTGIS_VERSION.tar.gz; \
	sed -i 's/#error.*/#define ACCEPT_USE_OF_DEPRECATED_PROJ_API_H 1/' /usr/include/proj_api.h; \
        cd /staging/postgis-$POSTGIS_VERSION; \
	./configure --with-pgconfig=/usr/local/pgsql/bin/pg_config --with-gdalconfig=/usr/bin/gdal-config > /dev/null; \

subsequent:
Postgis can be build if this line was deleted. Have an impact here with webodm?

[NtskwK]

@pierotofy Django 4.2 only supports PostgreSQL 12 and higher. Whether to need to continue to push the work, but more conservative.