Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mise à jour Elixir 1.15 -> 1.16 et Erlang 24 -> 25 #3815

Merged
merged 33 commits into from
Apr 8, 2024
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
1e3b720
Bump to Erlang 25 and Elixir 1.16
vdegove Mar 15, 2024
f38c980
Add a first version of upgrade documentation
vdegove Mar 15, 2024
28c6bab
Bump also nodejs version
vdegove Mar 15, 2024
0b51c6f
Point to the right docker image
vdegove Mar 15, 2024
de7fb35
improve readme
vdegove Mar 15, 2024
b8ee5db
remove compilation warnings
vdegove Mar 15, 2024
dbf8a57
remove warnings about match on 0.0
vdegove Mar 15, 2024
8f4d806
Update credo
vdegove Mar 18, 2024
02cadb4
Merge branch 'master' into elixir-1-16
thbar Mar 19, 2024
50894f6
Fix Dialyxir errors
vdegove Mar 19, 2024
629800b
Merge branch 'elixir-1-16' of https://github.com/etalab/transport-sit…
vdegove Mar 19, 2024
ea0b2a9
Merge branch 'master' into elixir-1-16
vdegove Mar 19, 2024
67e0cee
improve documentation
vdegove Mar 19, 2024
f583cd2
Update to OTP 26 in .tool_versions
vdegove Mar 19, 2024
b68c1c1
Order of map operations changed in OTP26
vdegove Mar 19, 2024
d771a9e
Temporary OTP26 dockerfile
vdegove Mar 19, 2024
7e74929
Revert "Update to OTP 26 in .tool_versions"
vdegove Mar 19, 2024
a9c5460
Revert "Order of map operations changed in OTP26"
vdegove Mar 19, 2024
f07acc9
Revert "Temporary OTP26 dockerfile"
vdegove Mar 19, 2024
72e16fd
Update comment in credo config
vdegove Mar 21, 2024
6ed3686
Merge branch 'master' into elixir-1-16
thbar Mar 21, 2024
cf67225
Merge branch 'master' into elixir-1-16
thbar Mar 22, 2024
c0e1700
Merge branch 'master' into elixir-1-16
thbar Mar 25, 2024
8c8ed74
Improve badges code in organizations
vdegove Mar 25, 2024
6112039
Merge branch 'elixir-1-16' of https://github.com/etalab/transport-sit…
vdegove Mar 25, 2024
a34566e
Revert credo config
vdegove Mar 25, 2024
43ffe86
Merge branch 'master' into elixir-1-16
thbar Mar 26, 2024
b817a1c
Merge branch 'master' into elixir-1-16
thbar Mar 28, 2024
f4c0b14
Merge branch 'master' into elixir-1-16
thbar Mar 30, 2024
677fd70
Merge branch 'master' into elixir-1-16
vdegove Apr 2, 2024
a579a4a
Merge branch 'master' into elixir-1-16
thbar Apr 7, 2024
1864fa2
Bump OS to latest
thbar Apr 8, 2024
8b89caf
Merge branch 'master' into elixir-1-16
thbar Apr 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ default_docker: &default_docker
defaults: &defaults
parameters:
base_image:
default: ghcr.io/etalab/transport-ops:elixir-1.15.5-erlang-24.3.4.13-ubuntu-focal-20230126-transport-tools-1.0.7
default: ghcr.io/etalab/transport-ops:elixir-1.16.2-erlang-25.3.2.10-ubuntu-focal-20240123-transport-tools-1.0.7
thbar marked this conversation as resolved.
Show resolved Hide resolved
type: string
# useful to invalidate the build cache manually by bumping the version
build_cache_key:
Expand Down
7 changes: 3 additions & 4 deletions .credo.exs
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,6 @@
{Credo.Check.Readability.ModuleNames},
{Credo.Check.Readability.ParenthesesOnZeroArityDefs},
{Credo.Check.Readability.ParenthesesInCondition},
{Credo.Check.Readability.PredicateFunctionNames},
{Credo.Check.Readability.PreferImplicitTry},
{Credo.Check.Readability.RedundantBlankLines},
{Credo.Check.Readability.StringSigils},
Expand Down Expand Up @@ -129,10 +128,10 @@

# Deprecated checks (these will be deleted after a grace period)
#
{Credo.Check.Readability.Specs, false}
{Credo.Check.Readability.Specs, false},

# Custom checks can be created using `mix credo.gen.check`.
#
# Allowing for now functions like is_gtfs?
vdegove marked this conversation as resolved.
Show resolved Hide resolved
{Credo.Check.Readability.PredicateFunctionNames, false}
thbar marked this conversation as resolved.
Show resolved Hide resolved
]
}
]
Expand Down
7 changes: 4 additions & 3 deletions .tool-versions
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,13 @@
# - https://hexdocs.pm/elixir/compatibility-and-deprecations.html
# - https://github.com/elixir-lang/elixir/releases
# - `asdf list all elixir`
elixir 1.15.5-otp-24
elixir 1.16.2-otp-25

# See:
# - https://github.com/erlang/otp/releases
# - Blog, e.g. https://www.erlang.org/blog/my-otp-25-highlights/
# - https://github.com/erlang/otp/blob/master/otp_versions.table
# - `asdf list all erlang`
erlang 24.3.4.13
erlang 25.3.2.10

nodejs 18.17.1
nodejs 20.11.1
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM ghcr.io/etalab/transport-ops:elixir-1.15.5-erlang-24.3.4.13-ubuntu-focal-20230126-transport-tools-1.0.7
FROM ghcr.io/etalab/transport-ops:elixir-1.16.2-erlang-25.3.2.10-ubuntu-focal-20240123-transport-tools-1.0.7

RUN mkdir phoenixapp
WORKDIR /phoenixapp
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile.dev
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM ghcr.io/etalab/transport-ops:elixir-1.15.5-erlang-24.3.4.13-ubuntu-focal-20230126-transport-tools-1.0.7
FROM ghcr.io/etalab/transport-ops:elixir-1.16.2-erlang-25.3.2.10-ubuntu-focal-20240123-transport-tools-1.0.7

RUN apt-get install -y git inotify-tools postgresql-client>=11

Expand Down
2 changes: 1 addition & 1 deletion apps/shared/lib/hasher.ex
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ defmodule Hasher do

def get_file_hash(file_path) do
file_path
|> File.stream!([], 2048)
|> File.stream!(2048)
vdegove marked this conversation as resolved.
Show resolved Hide resolved
|> compute_checksum(:sha256)
end

Expand Down
2 changes: 1 addition & 1 deletion apps/transport/lib/db/contact.ex
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ defmodule DB.Contact do
def organization_name(orgs) do
certified_orgs =
orgs
|> Enum.filter(fn %DB.Organization{badges: badges} -> Enum.any?(badges, &match?(&1, %{"kind" => "certified"})) end)
|> Enum.filter(fn %DB.Organization{badges: badges} -> Enum.any?(badges, &match?(%{"kind" => "certified"}, &1)) end)
vdegove marked this conversation as resolved.
Show resolved Hide resolved
vdegove marked this conversation as resolved.
Show resolved Hide resolved

case certified_orgs do
[] -> orgs |> List.first() |> Map.fetch!(:name)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ defmodule TransportWeb.Backoffice.PageView do
def unaccent(nil), do: ""

def unaccent(value) when is_binary(value) do
~R<\p{Mn}>u
~r/\p{Mn}/u
thbar marked this conversation as resolved.
Show resolved Hide resolved
|> Regex.replace(value |> :unicode.characters_to_nfd_binary(), "")
|> :unicode.characters_to_nfc_binary()
end
Expand Down
22 changes: 12 additions & 10 deletions apps/transport/test/transport/jobs/dataset_quality_score_test.exs
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ defmodule Transport.Test.Transport.Jobs.DatasetQualityScoreTest do

assert %{
format: "GTFS",
freshness: 0.0,
freshness: +0.0,
thbar marked this conversation as resolved.
Show resolved Hide resolved
raw_measure: %{end_date: _, start_date: _},
resource_id: _
} = resource_freshness(resource)
Expand Down Expand Up @@ -127,7 +127,7 @@ defmodule Transport.Test.Transport.Jobs.DatasetQualityScoreTest do
metadata: %{"feed_timestamp_delay" => 1000}
)

assert %{format: "gbfs", freshness: 0.0, raw_measure: 1000, resource_id: _} = resource_freshness(resource)
assert %{format: "gbfs", freshness: +0.0, raw_measure: 1000, resource_id: _} = resource_freshness(resource)
end

test "GBFS resource without metadata for today" do
Expand Down Expand Up @@ -178,7 +178,7 @@ defmodule Transport.Test.Transport.Jobs.DatasetQualityScoreTest do
insert(:resource_unavailability, start: hours_ago(2), resource: resource)

assert %{
availability: 0.0,
availability: +0.0,
raw_measure: (22 / 24 * 100) |> Float.floor(1),
resource_id: resource.id
} == resource_availability(resource)
Expand Down Expand Up @@ -240,7 +240,7 @@ defmodule Transport.Test.Transport.Jobs.DatasetQualityScoreTest do
details: %{
resources: [
%{availability: 1.0, raw_measure: nil, resource_id: r1.id},
%{availability: 0.0, raw_measure: 0, resource_id: r2.id}
%{availability: +0.0, raw_measure: 0, resource_id: r2.id}
]
},
score: 0.5
Expand Down Expand Up @@ -269,7 +269,7 @@ defmodule Transport.Test.Transport.Jobs.DatasetQualityScoreTest do

assert [] == dataset |> DB.Repo.preload(:resources) |> Map.fetch!(:resources)

assert %{score: 0.0, details: %{resources: []}} == current_dataset_availability(dataset.id)
assert %{score: +0.0, details: %{resources: []}} == current_dataset_availability(dataset.id)
end
end

Expand Down Expand Up @@ -298,7 +298,7 @@ defmodule Transport.Test.Transport.Jobs.DatasetQualityScoreTest do
},
%{
format: "gtfs-rt",
freshness: 0.0,
freshness: +0.0,
raw_measure: 1000,
resource_id: ^resource_id_2
}
Expand All @@ -319,12 +319,12 @@ defmodule Transport.Test.Transport.Jobs.DatasetQualityScoreTest do

# average freshness for only 1 resource with freshness information available
assert %{
score: 0.0,
score: +0.0,
details: %{
resources: [
%{
format: "GTFS",
freshness: 0.0,
freshness: +0.0,
raw_measure: %{end_date: _, start_date: _},
resource_id: ^resource_id
},
Expand Down Expand Up @@ -369,7 +369,9 @@ defmodule Transport.Test.Transport.Jobs.DatasetQualityScoreTest do
assert %{
score: 0,
details: %{
resources: [%{compliance: 0.0, raw_measure: %{"has_errors" => true}, resource_id: geojson_resource.id}]
resources: [
%{compliance: +0.0, raw_measure: %{"has_errors" => true}, resource_id: geojson_resource.id}
]
}
} == current_dataset_compliance(dataset.id)
end
Expand All @@ -395,7 +397,7 @@ defmodule Transport.Test.Transport.Jobs.DatasetQualityScoreTest do
score: 0.5,
details: %{
resources: [
%{compliance: 0.0, raw_measure: %{"max_error" => "Error"}, resource_id: gtfs_1.id},
%{compliance: +0.0, raw_measure: %{"max_error" => "Error"}, resource_id: gtfs_1.id},
%{compliance: 1.0, raw_measure: %{"max_error" => "Warning"}, resource_id: gtfs_2.id}
]
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ defmodule Transport.Jobs.LowEmissionZonesToGeoDataTest do
coordinates: [
[[{102.0, 2.0}, {103.0, 2.0}, {103.0, 3.0}, {102.0, 3.0}, {102.0, 2.0}]],
[
[{100.0, 0.0}, {101.0, 0.0}, {101.0, 1.0}, {100.0, 1.0}, {100.0, 0.0}],
[{100.0, +0.0}, {101.0, +0.0}, {101.0, 1.0}, {100.0, 1.0}, {100.0, +0.0}],
[{100.2, 0.2}, {100.2, 0.8}, {100.8, 0.8}, {100.8, 0.2}, {100.2, 0.2}]
]
],
Expand Down
32 changes: 32 additions & 0 deletions docs/upgrade_versions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
## Gestion des versions et des images Docker

Le setup standard de développement s’appuie sur ASDF pour la gestion des versions. Cependant, la CI/CD (CircleCI) et la production s’appuient sur Docker.

Une première image Docker est mise à disposition par le dépôt tranport-ops : elle récupère une image Docker standard sur le hub Docker avec déjà la bonne version de Elixir et Erlang, rajoute Node et les outils annexes (transport-tools), et est publiée sur le dépôt d’images Docker de Github.

L’application transport-site récupère l’image Docker de transport-ops, et y rajoute uniquement son propre code, le compile, lance le serveur, les migrations, et expose le serveur à l’extérieur.

CircleCI pour les tests va donc faire ces étapes là de compilation du code Elixir, mais sans devoir reconstruire toute l’image (puisque s’appuyant sur l’image de transport-ops). Quand on pousse sur les serveurs de CleverCloud, un premier serveur va créer l’image Docker finale (mêmes étapes que sur CircleCI, donc un temps de compilation existe) puis déployer l’image résultante. Potentiellement, on pourrait réduire à l’avenir le temps de déploiement en fournissant directement l’image Docker avec l’application précompilée, et que la même image serve pour les tests et pour la production.

## Processus de mise à jour

Les versions de ASDF et de Docker doivent être mises à jour dans la même branche de transport-site (sinon des tests ratent). Il faut donc en préalable du merge de la branche qu’une image avec les bonnes versions de Elixir, Erlang/OTP et Node et les outils transport soit déjà présente, donc qu’une branche correspondante sur transport-ops ait été mergée et un package crée.

Dans les grandes lignes : on teste d’abord avec ASDF que tout va bien, puis on fait le nécessaire sur le repo transport-ops pour créer un package, et on finalise la branche sur transport-site avec la référence de l’image.

1. faire une branche, mettre à jour tool versions
2. Voir ce qu’il y a dans tool versions => doc sur où sont les versions disponibles, plus deprecations elixir.
3. Soit bump mineur, soit bump majeur.
4. Gaffe à OTP 26 => issue.
5. Allez aussi chercher Erlang compatible
6. Plus Nodejs si possible, mais mieux vaut le faire après. Prendre la version LTS sur nodejs.org
7. => Le mieux est une image qui bumbe tout, mais dans ma branche y aller par étapes.
8. Vérifier que les tests passent. Le build test va pas passer, c’est normal. Il est possible qu’il y ait des warnings à traiter. Traiter les warnings. Astuce : mix test | tee upgrade.log
9. Possible de créer un ticket, voir https://github.com/etalab/transport-site/issues/3307
10. On peut faire la MAJ Elixir et pas OTP 26 https://github.com/etalab/transport-site/issues/3315
11. Si c’est tout smooth et que ça marche bien, cloner transport-ops en local. Voir https://github.com/etalab/transport-ops
12. Github workflow du repository : aller voir les tests pour protéger
13. Donc : une fois branche en local qui marche bien, modifier transport-ops pour faire correspondre, vérifier que tout marche bien, puis faire une release (avec le nom bien formaté). Pour faire une release, voir le readme.
14. Puis enfin, PR de transport-site à mettre à jour avec l’image.
15. Ne pas tester en local transport-site avec l’image docker : les tests sufiront + prochainement. Il n’y a pas de grosse diff entre passage asdf et docker.
16. Mettre à jour les readme :) => créer une doc sur trnasport-site sur le workflow.
Loading