images are pulled on machines where they are not needed #1208

koenhandekyn · 2024-11-09T09:07:50Z

koenhandekyn
Nov 9, 2024

i'm having one app server (app-01) running development, staging an demo instances of my app. i'm having another server (app-02) running production instances (one multitenant shared instance, but also some dedicated instances for customers that want their own instance).

when i'm deploying development - it only references the app-01 - somehow kamal 'finds out' (remembers?) that there is also an app-02, and though the app doesn't have to be booted there, it still triggers a pull of the image to that server.

it's not breaking anything but it's kind of wasting time and bandwidth.

it seems like a possible improvement to optimize this ?

tuladhar · 2024-11-12T10:18:59Z

tuladhar
Nov 12, 2024

Could you provide details on how the deploy config file is structured for dev, staging, and prod environments?

0 replies

koenhandekyn · 2024-11-12T10:33:15Z

koenhandekyn
Nov 12, 2024
Author

we have this list

deploy.yml is the 'root' and does not mention any servers
deploy.development.yml contains the specific development details (the actual server, the actual proxy hosts, some specific env vars in the clear)
deploy.staging.yml contains the specific staging details (the actual server, the actual proxy hosts, some specific env vars in the clear)
deploy.prod.yml contains the specific prod details (the actual server, the actual proxy hosts, some specific env vars in the clear)
etc

for the secrets it's kind of mirrored, but there the inheritence is somewhat different (kamal has the common approach there) but that part is working fine and we basically just are explicit for the secrets per environment

so the strange behaviour is that when we deploy to prod, i expect it reads deploy.prod.yml and deploy.yml, where it should not find any reference to the development/staging server. but still it is pulling the image on the development/staging server.

let me know if you need more

1 reply

tuladhar Nov 12, 2024

In deploy.yml, I see web role in servers pointing to some apps-, which hosts is it targetting?

Have you tried removing servers section from deploy.yml and try deploying to prod.

kamal deploy -d production

koenhandekyn · 2024-11-12T11:31:02Z

koenhandekyn
Nov 12, 2024
Author

sorry i was unclear i guess with the screenshots here are the actual files and details of the issue in the logs well indicated hope it clarifies deploy.yml with no roles/servers mentioned ``` service: allcrux-corpware image: koenhandekyn/allcrux registry: server: ghcr.io username: koenhandekyn password: - KAMAL_REGISTRY_PASSWORD builder: arch: x86_64 cache: type: gha # remote: ***@***.*** # secrets: # - BUILDER_ROOT_PASSWORD env: clear: ADMIN_SAML_SERVICE_URL: 'https://development.de.allcrux.com/admin/saml/auth' ALLCRUX_PREVIEW_REPORTS_BUCKET: 'allcrux-preview-reports' ALLCRUX_REPORTS_BUCKET: 'allcrux-reports' APP_REVISION: '<%= ENV['GIT_COMMIT_HASH'] || `git rev-parse HEAD`.strip %>' # this is used by appsignal AWS_REGION: 'eu-west-1' CLOUD_PROVIDER: 'hetzner' DEFAULT_FROM_ADDRESS: ***@***.*** DEPLOY_TIMESTAMP_ISO8601: '<%= Time.now.utc.strftime("%Y-%m-%dT%H:%M:%SZ") %>' EXTERNAL_PROTOCOL: 'https' MAILGUN_API_HOST: 'api.eu.mailgun.net' OPENAI_MODEL: 'gpt-4o' RAILS_ENV: 'production' RAILS_LOG_TO_STDOUT: 1 SESSION_MINUTES: 525600 secret: - ADMIN_OPENID_CLIENT_ID - ADMIN_OPENID_SECRET - ADMIN_OPENID_TENANT_ID - APPSIGNAL_PUSH_API_KEY - AWS_ACCESS_KEY_ID - AWS_SECRET_ACCESS_KEY - DATABASE_URL - MAILGUN_API_KEY - MAILGUN_WEBHOOK_API_KEY - OPENAI_API_KEY - RAILS_MASTER_KEY - SECRET_KEY_BASE - SIGNREQUEST_TOKEN - TWO_FACTOR_SECRET_KEY aliases: console: app exec --interactive --reuse "bin/rails console" shell: app exec --interactive --reuse "bash" logs: app logs -f dbc: app exec --interactive --reuse "bin/rails dbconsole" dblogs: accessory logs db -f readiness_delay: 5 deploy_timeout: 120 # require_destination: true retain_containers: 2 accessories: db: image: postgres:16 port: 5432:5432 host: apps-01.de.allcrux.com env: secret: - POSTGRES_USER - POSTGRES_PASSWORD directories: - data:/var/lib/postgresql/data db-prod: image: postgres:16 port: 5432:5432 host: apps-02.de.allcrux.com env: secret: - POSTGRES_USER - POSTGRES_PASSWORD directories: - data:/var/lib/postgresql/data ``` for example : deploy.staging.yml with reference to the server where it should run (similar for prod that references apps-02) ``` servers: web: - apps-01.de.allcrux.com proxy: ssl: true hosts: - staging.de.allcrux.com - staging.allcrux.com env: clear: HOST: 'staging.allcrux.com' S3_BUCKET: 'allcrux-staging' ALLCRUX_WATERMARK: 'staging' APPSIGNAL_APP_NAME: 'allcrux-corpware-staging' APPSIGNAL_APP_ENV: 'staging' MAILGUN_DOMAIN: 'mg-staging.allcrux.com' SESSION_MINUTES: 525600 ``` and the issue below e.g. when i instruct to deploy to staging, there should be NO reference to apps-02 (this only exists in deploy.prod.yml and i assume when targeting staging kamal should only read deploy.staging.yml and deploy.yml) and still kamal is doing work on both apps-01(correct) and apps-02 (not needed), for example deploying latest-development to staging, kamal is logging into apps-02 and pulling the image on apps-02 the good thing is that kamal is just restarting the app and proxy on apps-01, so that’s fine but kamal is doing a lot of work on apps-02, e.g. logging in and pulling the image, getting/releasing a lock etc which is not needed and actually also not desired. hope this clarifies ❯ kamal deploy -P --version=latest-development -d staging Log into image registry... INFO [3aba3afe] Running docker login ghcr.io -u [REDACTED] -p [REDACTED] on apps-01.de.allcrux.com INFO [73e41da1] Running docker login ghcr.io -u [REDACTED] -p [REDACTED] on apps-02.de.allcrux.com <http://apps-02.de.allcrux.com/> <<< NOT NEEDED INFO [73e41da1] Finished in 1.544 seconds with exit status 0 (successful). INFO [3aba3afe] Finished in 6.703 seconds with exit status 0 (successful). Pull app image... INFO [1a17e4d1] Running docker info --format '{{index .RegistryConfig.Mirrors 0}}' on apps-01.de.allcrux.com INFO [8f6632f0] Running docker info --format '{{index .RegistryConfig.Mirrors 0}}' on apps-02.de.allcrux.com <http://apps-02.de.allcrux.com/> <<< NOT NEEDED INFO [8d70153c] Running docker image rm --force ghcr.io/koenhandekyn/allcrux:latest-development on apps-02.de.allcrux.com INFO [9a1d0340] Running docker image rm --force ghcr.io/koenhandekyn/allcrux:latest-development on apps-01.de.allcrux.com INFO [9a1d0340] Finished in 0.222 seconds with exit status 0 (successful). INFO [ea3d4e98] Running docker pull ghcr.io/koenhandekyn/allcrux:latest-development on apps-01.de.allcrux.com INFO [ea3d4e98] Finished in 0.802 seconds with exit status 0 (successful). INFO [93f5cdc4] Running docker inspect -f '{{ .Config.Labels.service }}' ghcr.io/koenhandekyn/allcrux:latest-development | grep -x allcrux-corpware || (echo "Image ghcr.io/koenhandekyn/allcrux:latest-development is missing the 'service' label" && exit 1) on apps-01.de.allcrux.com INFO [93f5cdc4] Finished in 0.232 seconds with exit status 0 (successful). INFO [8d70153c] Finished in 4.398 seconds with exit status 0 (successful). INFO [8854f2ed] Running docker pull ghcr.io/koenhandekyn/allcrux:latest-development on apps-02.de.allcrux.com <http://apps-02.de.allcrux.com/> <<< NOT NEEDED (uses bandwidth, takes time, etc) INFO [8854f2ed] Finished in 51.395 seconds with exit status 0 (successful). INFO [0a212cba] Running docker inspect -f '{{ .Config.Labels.service }}' ghcr.io/koenhandekyn/allcrux:latest-development | grep -x allcrux-corpware || (echo "Image ghcr.io/koenhandekyn/allcrux:latest-development is missing the 'service' label" && exit 1) on apps-02.de.allcrux.com INFO [0a212cba] Finished in 0.130 seconds with exit status 0 (successful). INFO [670f6643] Running /usr/bin/env mkdir -p .kamal on apps-01.de.allcrux.com INFO [c59b724a] Running /usr/bin/env mkdir -p .kamal on apps-02.de.allcrux.com <http://apps-02.de.allcrux.com/> <<< NOT NEEDED INFO [670f6643] Finished in 0.104 seconds with exit status 0 (successful). INFO [c59b724a] Finished in 0.167 seconds with exit status 0 (successful). Acquiring the deploy lock... Ensure kamal-proxy is running... INFO [17af717c] Running docker network create kamal on apps-01.de.allcrux.com INFO [f9b7ecdb] Running docker network create kamal on apps-02.de.allcrux.com <http://apps-02.de.allcrux.com/> <<< NOT NEEDED INFO [adf0b937] Running docker login ghcr.io -u [REDACTED] -p [REDACTED] on apps-01.de.allcrux.com INFO [adf0b937] Finished in 0.609 seconds with exit status 0 (successful). INFO [9d1b17a8] Running docker inspect kamal-proxy --format '{{.Config.Image}}' | cut -d: -f2 on apps-01.de.allcrux.com INFO [9d1b17a8] Finished in 0.219 seconds with exit status 0 (successful). INFO [56725679] Running docker container start kamal-proxy || docker run --name kamal-proxy --network kamal --detach --restart unless-stopped --volume kamal-proxy-config:/home/kamal-proxy/.config/kamal-proxy $(cat .kamal/proxy/options || echo "--publish 80:80 --publish 443:443 --log-opt max-size=10m") basecamp/kamal-proxy:v0.8.2 on apps-01.de.allcrux.com INFO [56725679] Finished in 0.215 seconds with exit status 0 (successful). Detect stale containers... INFO [58d082db] Running docker ps --filter label=service=allcrux-corpware --filter label=destination=staging --filter label=role=web --format "{{.Names}}" | while read line; do echo ${line#allcrux-corpware-web-staging-}; done on apps-01.de.allcrux.com INFO [58d082db] Finished in 0.200 seconds with exit status 0 (successful). INFO [af799103] Running /usr/bin/env sh -c 'docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=allcrux-corpware --filter label=destination=staging --filter label=role=web --filter status=running --filter status=restarting --filter ancestor=$(docker image ls --filter reference=ghcr.io/koenhandekyn/allcrux:latest-staging --format '\''{{.ID}}'\'') ; docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=allcrux-corpware --filter label=destination=staging --filter label=role=web --filter status=running --filter status=restarting' | head -1 | while read line; do echo ${line#allcrux-corpware-web-staging-}; done on apps-01.de.allcrux.com INFO [af799103] Finished in 0.400 seconds with exit status 0 (successful). Start container with version latest-development (or reboot if already running)... INFO [d42cd34c] Running docker container ls --all --filter name=^allcrux-corpware-web-staging-latest-development$ --quiet on apps-01.de.allcrux.com INFO [d42cd34c] Finished in 0.245 seconds with exit status 0 (successful). INFO [6e17a241] Running /usr/bin/env sh -c 'docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=allcrux-corpware --filter label=destination=staging --filter label=role=web --filter status=running --filter status=restarting --filter ancestor=$(docker image ls --filter reference=ghcr.io/koenhandekyn/allcrux:latest-staging --format '\''{{.ID}}'\'') ; docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=allcrux-corpware --filter label=destination=staging --filter label=role=web --filter status=running --filter status=restarting' | head -1 | while read line; do echo ${line#allcrux-corpware-web-staging-}; done on apps-01.de.allcrux.com INFO [6e17a241] Finished in 0.388 seconds with exit status 0 (successful). INFO [52b22328] Running /usr/bin/env mkdir -p .kamal/apps/allcrux-corpware-staging/env/roles on apps-01.de.allcrux.com INFO [52b22328] Finished in 0.174 seconds with exit status 0 (successful). INFO Uploading .kamal/apps/allcrux-corpware-staging/env/roles/web.env 100.0% INFO [fb90a182] Running docker run --detach --restart unless-stopped --name allcrux-corpware-web-staging-latest-development --network kamal --hostname apps-01.de.allcrux.com-d0f3d126a136 -e KAMAL_CONTAINER_NAME="allcrux-corpware-web-staging-latest-development" -e KAMAL_VERSION="latest-development" --env ADMIN_SAML_SERVICE_URL="https://development.de.allcrux.com/admin/saml/auth" --env ALLCRUX_PREVIEW_REPORTS_BUCKET="allcrux-preview-reports" --env ALLCRUX_REPORTS_BUCKET="allcrux-reports" --env APP_REVISION="502832f9cc52ab63e2e79b988f55dbc894500288" --env AWS_REGION="eu-west-1" --env CLOUD_PROVIDER="hetzner" --env ***@***.***" --env DEPLOY_TIMESTAMP_ISO8601="2024-11-12T11:24:12Z" --env EXTERNAL_PROTOCOL="https" --env MAILGUN_API_HOST="api.eu.mailgun.net" --env OPENAI_MODEL="gpt-4o" --env RAILS_ENV="production" --env RAILS_LOG_TO_STDOUT="1" --env SESSION_MINUTES="525600" --env HOST="staging.allcrux.com" --env S3_BUCKET="allcrux-staging" --env ALLCRUX_WATERMARK="staging" --env APPSIGNAL_APP_NAME="allcrux-corpware-staging" --env APPSIGNAL_APP_ENV="staging" --env MAILGUN_DOMAIN="mg-staging.allcrux.com" --env-file .kamal/apps/allcrux-corpware-staging/env/roles/web.env --log-opt max-size="10m" --label service="allcrux-corpware" --label role="web" --label destination="staging" ghcr.io/koenhandekyn/allcrux:latest-development on apps-01.de.allcrux.com INFO [fb90a182] Finished in 0.752 seconds with exit status 0 (successful). INFO [6d77a2b4] Running docker container ls --all --filter name=^allcrux-corpware-web-staging-latest-development$ --quiet on apps-01.de.allcrux.com INFO [6d77a2b4] Finished in 0.227 seconds with exit status 0 (successful). INFO [085c7574] Running docker exec kamal-proxy kamal-proxy deploy allcrux-corpware-web-staging --target="6c930e2527c3:80" --host="staging.de.allcrux.com" --host="staging.allcrux.com" --tls --deploy-timeout="120s" --drain-timeout="30s" --buffer-requests --buffer-responses --log-request-header="Cache-Control" --log-request-header="Last-Modified" --log-request-header="User-Agent" on apps-01.de.allcrux.com INFO [085c7574] Finished in 30.383 seconds with exit status 0 (successful). INFO First web container is healthy on apps-01.de.allcrux.com, booting any other roles INFO [e47cf783] Running docker container ls --all --filter name=^allcrux-corpware-web-staging-latest-staging_replaced_21bbe4282f0183b2$ --quiet | xargs docker stop on apps-01.de.allcrux.com INFO [e47cf783] Finished in 1.440 seconds with exit status 0 (successful). INFO [cbf1b3de] Running docker tag ghcr.io/koenhandekyn/allcrux:latest-development ghcr.io/koenhandekyn/allcrux:latest-staging on apps-02.de.allcrux.com <http://apps-02.de.allcrux.com/> <<< NOT NEEDED INFO [521fd3d1] Running docker tag ghcr.io/koenhandekyn/allcrux:latest-development ghcr.io/koenhandekyn/allcrux:latest-staging on apps-01.de.allcrux.com INFO [cbf1b3de] Finished in 0.192 seconds with exit status 0 (successful). INFO [521fd3d1] Finished in 0.226 seconds with exit status 0 (successful). Prune old containers and images... INFO [fbce7b4c] Running docker ps -q -a --filter label=service=allcrux-corpware --filter status=created --filter status=exited --filter status=dead | tail -n +3 | while read container_id; do docker rm $container_id; done on apps-02.de.allcrux.com <http://apps-02.de.allcrux.com/> <<< NOT NEEDED INFO [eef60474] Running docker ps -q -a --filter label=service=allcrux-corpware --filter status=created --filter status=exited --filter status=dead | tail -n +3 | while read container_id; do docker rm $container_id; done on apps-01.de.allcrux.com INFO [fbce7b4c] Finished in 0.117 seconds with exit status 0 (successful). INFO [eef60474] Finished in 0.285 seconds with exit status 0 (successful). INFO [09a07d5f] Running docker image prune --force --filter label=service=allcrux-corpware on apps-02.de.allcrux.com <http://apps-02.de.allcrux.com/> <<< NOT NEEDED INFO [c140c38c] Running docker image prune --force --filter label=service=allcrux-corpware on apps-01.de.allcrux.com INFO [09a07d5f] Finished in 0.185 seconds with exit status 0 (successful). INFO [e341315b] Running docker image ls --filter label=service=allcrux-corpware --format '{{.ID}} {{.Repository}}:{{.Tag}}' | grep -v -w "$(docker container ls -a --format '{{.Image}}\|' --filter label=service=allcrux-corpware | tr -d '\n')ghcr.io/koenhandekyn/allcrux:latest-staging\|ghcr.io/koenhandekyn/allcrux:<none>" | while read image tag; do docker rmi $tag; done on apps-02.de.allcrux.com <http://apps-02.de.allcrux.com/> <<< NOT NEEDED INFO [c140c38c] Finished in 0.228 seconds with exit status 0 (successful). INFO [99bb4260] Running docker image ls --filter label=service=allcrux-corpware --format '{{.ID}} {{.Repository}}:{{.Tag}}' | grep -v -w "$(docker container ls -a --format '{{.Image}}\|' --filter label=service=allcrux-corpware | tr -d '\n')ghcr.io/koenhandekyn/allcrux:latest-staging\|ghcr.io/koenhandekyn/allcrux:<none>" | while read image tag; do docker rmi $tag; done on apps-01.de.allcrux.com INFO [e341315b] Finished in 0.209 seconds with exit status 0 (successful). INFO [99bb4260] Finished in 4.393 seconds with exit status 0 (successful). Releasing the deploy lock... Finished all in 105.3 seconds

…

On 12 Nov 2024, at 12:13, Puru ***@***.***> wrote: In deploy.yml, I see web role in servers pointing to some apps-, which hosts is it targetting? Have you tried removing servers section from deploy.yml and try deploying to prod. kamal deploy -d production — Reply to this email directly, view it on GitHub <#1208 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFNM5RT3FF7Y2CTOOHNDDL2AHPG3AVCNFSM6AAAAABRO5XLYSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMRSGQ3DKMY>. You are receiving this because you authored the thread.

1 reply

tuladhar Nov 12, 2024

Can you try adding -p option to only run it on primary role (web) instead of all, like this:

kamal deploy -P --version=latest-development -d staging -p

koenhandekyn · 2024-11-12T19:56:49Z

koenhandekyn
Nov 12, 2024
Author

ha that worked ! so in your analysis the current behavior is then correct? for me it feels confusing based on the documentation - kamal reads deploy.yml - kamal reads destination specific deploy.staging.yml so it’s kind of surprising that kamal ‘discovers’ ’non primary nodes’ in the other files ? the ticket was meant as a suggestion for possible improvement i’ll use the tip / trick with -p :) i’m hoping (will test soon) that this will also work when deploying to production :) AH : another small one, when deploying like below to staging like that, the image is not tagged as latest-staging, which would actually be very nice in my opinion to support a dev/staging/prod promotion flow ... kamal deploy -P --version=latest-development -d staging -p

…

On 12 Nov 2024, at 13:20, Puru ***@***.***> wrote: Can you try adding -p option to only run it on primary role (web) instead of all, like this: kamal deploy -P --version=latest-development -d staging -p — Reply to this email directly, view it on GitHub <#1208 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFNM5TTG2VMYCGDVCH2BW32AHXAFAVCNFSM6AAAAABRO5XLYSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMRSGU2DCOA>. You are receiving this because you authored the thread.

0 replies

tuladhar · 2024-11-13T06:12:55Z

tuladhar
Nov 13, 2024

so it’s kind of surprising that kamal ‘discovers’ ’non primary nodes’ in the other files ?

In this case, it detected the hosts from accessory in deploy.yml.

0 replies

koenhandekyn · 2024-11-13T06:28:55Z

koenhandekyn
Nov 13, 2024
Author

ah interesting, then i’ll try putting them into a deploy.accessory-non-prod.yml and deploy.accessory-prod.yml file :) but from ‘outside’ the box it doesn’t feel very intuitive, in some context i would deploy my db for example on to a dedicated machine, and i wouldn’t expect it to be included in my app cluster ... just sharing my intuition from my point of view with respect for other views ! thanks for sharing the analysis and source, it makes a bit more sense now what i was observing

…

On 13 Nov 2024, at 07:13, Puru ***@***.***> wrote: so it’s kind of surprising that kamal ‘discovers’ ’non primary nodes’ in the other files ? In this case, it detected the hosts from accessory in deploy.yml. — Reply to this email directly, view it on GitHub <#1208 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFNM5XMAB3B3EY725RQOBT2ALUXZAVCNFSM6AAAAABRO5XLYSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMRTGU3TAMI>. You are receiving this because you authored the thread.

0 replies

koenhandekyn · 2024-11-13T06:36:42Z

koenhandekyn
Nov 13, 2024
Author

confirming it works i created 2 destination files deploy.accessory-01.yml the db for app-01 deploy.accessory-01.yml the db for app-02 now kamal deploy -P --version=latest-development -d staging purely works with app-01 as intended HOWEVER now somewhat unexpectedly i do need to mention a list of servers in my accessory deploy yml files even though i’m purely using these deploy files then to manage the accessories, so to get it working i now have this deploy.accessory-01.yml file concretely servers: web: - apps-01.de.allcrux.com accessories: db: image: postgres:16 port: 5432:5432 host: apps-01.de.allcrux.com env: secret: - POSTGRES_USER - POSTGRES_PASSWORD directories: - data:/var/lib/postgresql/data

…

On 13 Nov 2024, at 07:28, koen handekyn ***@***.***> wrote: ah interesting, then i’ll try putting them into a deploy.accessory-non-prod.yml and deploy.accessory-prod.yml file :) but from ‘outside’ the box it doesn’t feel very intuitive, in some context i would deploy my db for example on to a dedicated machine, and i wouldn’t expect it to be included in my app cluster ... just sharing my intuition from my point of view with respect for other views ! thanks for sharing the analysis and source, it makes a bit more sense now what i was observing > On 13 Nov 2024, at 07:13, Puru ***@***.***> wrote: > > > so it’s kind of surprising that kamal ‘discovers’ ’non primary nodes’ in the other files ? > > In this case, it detected the hosts from accessory in deploy.yml. > > — > Reply to this email directly, view it on GitHub <#1208 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFNM5XMAB3B3EY725RQOBT2ALUXZAVCNFSM6AAAAABRO5XLYSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMRTGU3TAMI>. > You are receiving this because you authored the thread. >

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

images are pulled on machines where they are not needed #1208

{{title}}

Replies: 7 comments 2 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

images are pulled on machines where they are not needed #1208

koenhandekyn Nov 9, 2024

Replies: 7 comments · 2 replies

tuladhar Nov 12, 2024

koenhandekyn Nov 12, 2024 Author

tuladhar Nov 12, 2024

koenhandekyn Nov 12, 2024 Author

tuladhar Nov 12, 2024

koenhandekyn Nov 12, 2024 Author

tuladhar Nov 13, 2024

koenhandekyn Nov 13, 2024 Author

koenhandekyn Nov 13, 2024 Author

koenhandekyn
Nov 9, 2024

Replies: 7 comments 2 replies

tuladhar
Nov 12, 2024

koenhandekyn
Nov 12, 2024
Author

koenhandekyn
Nov 12, 2024
Author

koenhandekyn
Nov 12, 2024
Author

tuladhar
Nov 13, 2024

koenhandekyn
Nov 13, 2024
Author

koenhandekyn
Nov 13, 2024
Author