Running oro:platform:update in multi server environment

[LRotherfield]

Hi Guys

We are running the Oro Platform on multiple servers and are running into an issue when we deploy.

To insure migrations are run and new assets are generated at the same time as the new code is deployed, we run oro:platform:update on each server on each deploy. Occasionally one of our servers will fail to deploy because of a race condition in oro:platform:update which results in:

[Oro\Bundle\EntityConfigBundle\Exception\RuntimeException]                                          
  A model for "Oro\Bundle\AddressBundle\Entity\AddressType" was not found. Config models are locked.  
                                                                                                      
    oro:entity-config:cache:warmup ...

I raised another ticket about load balanced environments and was pointed to https://oroinc.com/orocrm/doc/current/dev-guide/scale-nodes but it does not seem to deal with deployments and how you would run oro:platform:update.

What is the suggested method for making sure assets are generated and migrations run in a multi server deploy without running into the above (and potentially other) race conditions?

Cheers
Luke

[anyt]

Do you use a shared database for all instances?

[LRotherfield]

Yes

[MikeParkin]

@anyt - We're load balancing for performance + redundancy of a single instance of OroPlatform, hence the shared database.

[anyt]

Why are you run oro:platform:update on multiple instances?
if cache and database are shared I don't see why it's needed

[anyt]

To only install assets, if they are not shared you can use this commands:

app/console fos:js-routing:dump --env=prod
app/console  oro:localization:dump --env=prod
app/console oro:assets:install --symlink --env=prod
app/console assetic:dump --env=prod
app/console oro:translation:dump --env=prod
app/console oro:requirejs:build --env=prod

About migrations from my perspective it's enough to run platform:update only on single instance as they affects only database and caches

[anyt]

But I'd recommend sharing assets as well.
It's possible by using CDN or some sort of local CDN just by moving them to separate domain.
Here is example configuration for configuring CDN on top of assetic:
https://symfony.com/blog/new-in-symfony-2-7-the-new-asset-component#configuration-changes

[LRotherfield]

@anyt, we understand that running platform:update would be better on only one instance, is there a way to configure the platform to do that though? Or is that something that you write a separate script for in your deployment tool?

[anyt]

There are several rules:

1. All resources that can be modified at runtime should be shared.
2. During deployment, an application must be in maintenance mode so no one will trigger cache rebuild by loading the webpage when an application is not ready, as this can lead to some unexpected errors.
3. It's just more reliable to prepare single instance and then clone all the code to other instances, as there is no need to warmup cache or build assets several times from different apps. For example platform:update will run migrations, build assets, warmup cache etc. on each application you run it, but it's enough to do it once.

[anyt]

I don't have any ready to use scripts to share, just recommendations.

[MikeParkin]

@anyt on typical CI/Automated deployment we do: 1. Push code, 2: Build zip in build environment, 3: deploy.

With Oro Platform we cannot do a full build during build time (part 2) because it requires access to the database for entity config and we don't want to give our build server access to production database. We also then we might need to also run migrations on the server during step 3.

Our infrastructure is really common, one load balancer, multiple web servers then 1 database. We have NFS mounts for all "shared" points (which is basically just media/uploads) and we keep cache in redis.

When you make a deploy using AWS to multiple servers, it's not possible to say "deploy on this server first and run oro:platform:update, then, deploy on every other server without running platform update". Also, doing it like that would increase the amount of time where different versions of the software are running connected to the same database.

This feels like the most common usecase for running in load balanced environment and we can't find any guidance on documentation for how to run such a setup. Surely your hosted enterprise solution encounters many of the same issues? Would it be possible to get some guidance/recommendations on how this is done / what best practice recommendations are for such a scenario please?

[anyt]

would increase the amount of time where different versions of the software are running connected to the same database.

Currently, OroPlatform based application doesn't support running old code with the new database or search index or caches, and vice-versa. So during update maintenance mode should be enabled for all instances.

[MikeParkin]

@anyt ok fair enough, so when we start running push we enable maintenance mode. Far from ideal, but can accept that.

What is the recommended set of steps then.

"First server"
oro:platform:update

"all other servers"
app/console fos:js-routing:dump --env=prod
app/console oro:localization:dump --env=prod
app/console oro:assets:install --symlink --env=prod
app/console assetic:dump --env=prod
app/console oro:translation:dump --env=prod
app/console oro:requirejs:build --env=prod

?

[MikeParkin]

Also, this means every push is going to result in many minutes of downtime - from a CI perspective it's a bit of a nightmare no?

[anyt]

This is the regular update flow for single instance application https://oroinc.com/orocrm/doc/2.0/dev-guide/cookbook/how-to-upgrade-to-new-version .
Depending on what is shared on your setup you can adapt it to your needs.
oro:platform:update command just runs a bunch of other commands, you can check the list here https://github.com/oroinc/platform/blob/master/src/Oro/Bundle/InstallerBundle/Command/PlatformUpdateCommand.php

I just listed commands that required for rebuilding assets, but above list can be incomplete for your setup.