[Zookeeper] Upgrade workers to AL2023

[mreyescdl]

Current Stage workers

• uc3-mrtzk01x2-stg.cdlib.org
• uc3-mrtzk02x2-stg.cdlib.org
• uc3-mrtzk03x2-stg.cdlib.org

Current Production workers

• uc3-mrtzk01x2-prd.cdlib.org
• uc3-mrtzk02x2-prd.cdlib.org
• uc3-mrtzk03x2-prd.cdlib.org
• uc3-mrtzk04x2-prd.cdlib.org
• uc3-mrtzk05x2-prd.cdlib.org

Task list

• Parallel AL2023 workers for stage and production
• Modify SSM variables (/uc3/mrt/stg/inventory/zoo/queueService,/uc3/mrt/prd/inventory/zoo/queueService)
• Restart client services (Access Inventory, Ingest and Admin)

[mreyescdl]

Stage progress

• Updated SSM var

        --name /uc3/mrt/stg/inventory/zoo/queueService \
        --value "uc3-mrtzk-stg01.cdlib.org:2181,uc3-mrtzk-stg02.cdlib.org:2181,uc3-mrtzk-stg03.cdlib.org:2181"

[mreyescdl]

Restarted Access/Ingest/Inventory which will bring in new ZK workers
Running Integration test...
All tests passed.

@ashleygould New Zookeeper workers on Stage working okay. Can we run in Stage for a while before moving on to production? Also, please keep old workers around for this period in case we need to rollback

@terrywbrady Can you move Stage Admin UI to use the new ZK workers?

[mreyescdl]

@ashleygould Please decommission old Stage workers

• uc3-mrtzk01x2-stg.cdlib.org
• uc3-mrtzk02x2-stg.cdlib.org
• uc3-mrtzk03x2-stg.cdlib.org

[mreyescdl]

Production

Update SSM var

        --name /uc3/mrt/prd/inventory/zoo/queueService \
        --value "uc3-mrtzk-prd01.cdlib.org:2181,uc3-mrtzk-prd02.cdlib.org:2181,uc3-mrtzk-prd03.cdlib.org:2181,uc3-mrtzk-prd04.cdlib.org:2181,uc3-mrtzk-prd05.cdlib.org:2181"

[mreyescdl]

Switched over to AL2023 Zookeeper workers

@ashleygould Please keep old workers online for a week or so.