stop workers during online backup #856

evgeni · 2024-05-28T10:10:42Z

this ensures more consistent on-disk data

definitions/procedures/backup/stop_workers.rb

definitions/checks/pulpcore/not_running.rb

evgeni · 2024-06-07T07:33:25Z

# hammer job-invocation create --job-template "Run Command - Script Default" --inputs 'command=sleep 120' --search-query "name = $(hostname -f)" --async
Job invocation 5 created

# foreman-maintain backup online /var/tmp/b
Starting backup: 2024-06-07 07:29:04 +0000
Running preparation steps required to run the next scenarios
================================================================================
Make sure Foreman DB is up: 
/ Checking connection to the Foreman DB                               [OK]      
--------------------------------------------------------------------------------
Make sure Candlepin DB is up: 
- Checking connection to the Candlepin DB                             [OK]      
--------------------------------------------------------------------------------
Make sure Pulpcore DB is up: 
\ Checking connection to the Pulpcore DB                              [OK]      
--------------------------------------------------------------------------------


Running Backup
================================================================================
Check for running tasks:                                              [FAIL]
There are 2 active task(s) in the system.
Please wait for these to complete or cancel them from the Monitor tab.
--------------------------------------------------------------------------------
There are multiple steps to proceed:
1) Fetch tasks status and wait till they finish
2) Investigate the tasks via UI
Select step to continue, [n(next), q(quit)] 1
Fetch tasks status and wait till they finish:                                   
| Try checking status of running task(s)                                        
There are 2 running tasks.
| Try checking status of running task(s)                                        
There are 2 running tasks.
\ Try checking status of running task(s)                                        
There are 2 running tasks.
\ Try checking status of running task(s)                                        
There are 2 running tasks.
\ Try checking status of running task(s)                                        
There are 2 running tasks.
\ Try checking status of running task(s)                                        
There are 2 running tasks.
| Try checking status of running task(s)                                        
There are 2 running tasks.
| Try checking status of running task(s)                                        
There are 2 running tasks.
| Try checking status of running task(s)                                        
There are 2 running tasks.
| Try checking status of running task(s)                                        
There are 2 running tasks.
| Try checking status of running task(s)                                        
There are 2 running tasks.
| Try checking status of running task(s)                                        
There are 1 running tasks.
| Try checking status of running task(s)                                        
There are 1 running tasks.
| Try checking status of running task(s)                              [OK]      
--------------------------------------------------------------------------------
Rerunning the check after fix procedure
Check for running tasks:                                              [OK]
--------------------------------------------------------------------------------
Check for running pulpcore tasks:                                     [OK]

evgeni · 2024-06-07T08:29:49Z

# PULP_SETTINGS=/etc/pulp/settings.py pulpcore-manager shell < lol.py 
/pulp/api/v3/tasks/018ff1cc-1924-7108-9b17-0b7adc330e9b/

# pulp task list --state running
[
  {
    "pulp_href": "/pulp/api/v3/tasks/018ff1cc-1924-7108-9b17-0b7adc330e9b/",
    "pulp_created": "2024-06-07T08:23:55.941268Z",
    "pulp_last_updated": "2024-06-07T08:23:55.941279Z",
    "state": "running",
    "name": "pulpcore.app.tasks.test.sleep",
    "logging_cid": "lol",
    "created_by": "/pulp/api/v3/users/1/",
    "unblocked_at": "2024-06-07T08:23:55.952339Z",
    "started_at": "2024-06-07T08:23:55.972970Z",
    "finished_at": null,
    "error": null,
    "worker": "/pulp/api/v3/workers/018ff195-c89d-70bd-8262-16b5431226bc/",
    "parent_task": null,
    "child_tasks": [],
    "task_group": null,
    "progress_reports": [],
    "created_resources": [],
    "reserved_resources_record": [
      "shared:/pulp/api/v3/domains/018fb72c-f3d2-7e49-9b59-20a4c2f7da44/"
    ]
  }
]

# foreman-maintain backup online /var/tmp/b
Starting backup: 2024-06-07 08:24:01 +0000
Running preparation steps required to run the next scenarios
================================================================================
Make sure Foreman DB is up: 
/ Checking connection to the Foreman DB                               [OK]      
--------------------------------------------------------------------------------
Make sure Candlepin DB is up: 
- Checking connection to the Candlepin DB                             [OK]      
--------------------------------------------------------------------------------
Make sure Pulpcore DB is up: 
\ Checking connection to the Pulpcore DB                              [OK]      
--------------------------------------------------------------------------------


Running Backup
================================================================================
Check for running tasks:                                              [OK]
--------------------------------------------------------------------------------
Check for running pulpcore tasks:                                     [FAIL]
There are 1 active task(s) in the system.
Please wait for these to complete.
--------------------------------------------------------------------------------
Continue with step [Fetch tasks status and wait till they finish]?, [y(yes), n(no), q(quit)] y
Fetch tasks status and wait till they finish:                                   
- waiting for tasks to finish                                                   
There are 1 tasks.
/ Waiting 10 seconds before retry.                                              
There are 1 tasks.
\ Waiting 10 seconds before retry.                                              
There are 1 tasks.
/ Waiting 10 seconds before retry.                                              
There are 1 tasks.
| Waiting 10 seconds before retry.                                              
There are 1 tasks.
- Waiting 10 seconds before retry.                                              
There are 1 tasks.
| Waiting 10 seconds before retry.                                    [OK]      
--------------------------------------------------------------------------------
Rerunning the check after fix procedure
Check for running pulpcore tasks:                                     [OK]

lol.py being:

from django_guid import set_guid 
from pulpcore.tasking.tasks import dispatch 
from pulpcore.app.util import get_url
set_guid('lol') 
task = dispatch("pulpcore.app.tasks.test.sleep", args=(60,)) 
print(get_url(task))

evgeni · 2024-06-14T11:25:03Z

lib/foreman_maintain/cli/backup_command.rb

@@ -57,6 +57,7 @@ def common_backup_options
          option '--features', 'FEATURES',
            "#{proxy_name} features to include in the backup. " \
                'Valid features are tftp, dns, dhcp, openscap, and all.', :multivalued => true
+          option '--wait-for-tasks', :flag, 'Wait for running tasks to complete instead of aborting'


Right now I've implemented this as a boolean flag, but am wondering if instead we want to give users the ability to configure the timeout. (the current one is 300, so 5 minutes, for both pulp and foreman, so a maximum of 10 minutes waiting)

Yes - however we could make it explicit (options to the CLI) or "hidden" (environment variables).

I think that is the only one that is left undecided.
I must admit I kinda find env vars bad for discovery, but on the other hand we'd have to add the cli command to update/upgrade too if we'd go with that (to be consistent).
But I also really have no feeling how often people would actually want to override that.

So, maybe, start with env, and then see how many complaints we get?

evgeni · 2024-06-14T11:29:47Z

should the whole "wait for tasks before backup" part be split out in an own commit/PR?

definitions/checks/pulpcore/no_running_tasks.rb

evgeni · 2024-06-17T09:43:23Z

split the big patch into four logical commits, enjoy!

ehelms · 2024-07-12T20:56:29Z

Needs a rebase!

evgeni · 2024-07-13T09:58:47Z

got a rebase!

evgeni · 2024-07-13T10:00:58Z

hm, need to adjust tests. later.

definitions/checks/pulpcore/no_running_tasks.rb

definitions/scenarios/satellite_upgrade.rb

optionally allow waiting for them to complete

this ensures more consistent on-disk data

evgeni force-pushed the stop-workers-for-backup branch 3 times, most recently from 68572a8 to b27654d Compare May 29, 2024 07:07

evgeni commented May 29, 2024

View reviewed changes

definitions/procedures/backup/stop_workers.rb Outdated Show resolved Hide resolved

evgeni force-pushed the stop-workers-for-backup branch 2 times, most recently from b844757 to f2a3be6 Compare May 29, 2024 08:26

evgeni commented May 29, 2024

View reviewed changes

definitions/checks/pulpcore/not_running.rb Outdated Show resolved Hide resolved

evgeni force-pushed the stop-workers-for-backup branch 2 times, most recently from a3cc938 to 91cb8bb Compare May 29, 2024 16:11

evgeni force-pushed the stop-workers-for-backup branch from 91cb8bb to 6d4e875 Compare June 7, 2024 07:12

evgeni force-pushed the stop-workers-for-backup branch 3 times, most recently from 9595a47 to cd5e6cb Compare June 7, 2024 08:23

evgeni force-pushed the stop-workers-for-backup branch 10 times, most recently from 73b53e6 to 239f42c Compare June 14, 2024 11:21

evgeni commented Jun 14, 2024

View reviewed changes

evgeni marked this pull request as ready for review June 14, 2024 11:29

ehelms reviewed Jun 14, 2024

View reviewed changes

definitions/checks/pulpcore/no_running_tasks.rb Outdated Show resolved Hide resolved

ehelms reviewed Jun 14, 2024

View reviewed changes

definitions/checks/pulpcore/no_running_tasks.rb Outdated Show resolved Hide resolved

evgeni force-pushed the stop-workers-for-backup branch from 6a9403a to 5059588 Compare June 14, 2024 14:08

evgeni mentioned this pull request Jun 14, 2024

enforce better variable name than e for exceptions via rubocop #885

Open

evgeni force-pushed the stop-workers-for-backup branch 2 times, most recently from 5766e06 to db07fba Compare June 17, 2024 09:39

evgeni force-pushed the stop-workers-for-backup branch from db07fba to bacba01 Compare June 19, 2024 13:28

evgeni force-pushed the stop-workers-for-backup branch 2 times, most recently from d66c6ad to 6ba62e7 Compare July 13, 2024 09:58

evgeni force-pushed the stop-workers-for-backup branch 3 times, most recently from cd13c57 to 2897a6c Compare July 17, 2024 08:49

ehelms reviewed Jul 17, 2024

View reviewed changes

definitions/checks/pulpcore/no_running_tasks.rb Show resolved Hide resolved

ehelms reviewed Jul 17, 2024

View reviewed changes

definitions/checks/pulpcore/no_running_tasks.rb Show resolved Hide resolved

ehelms reviewed Jul 17, 2024

View reviewed changes

definitions/checks/pulpcore/no_running_tasks.rb Outdated Show resolved Hide resolved

evgeni force-pushed the stop-workers-for-backup branch from 2897a6c to b87f9b8 Compare July 18, 2024 06:58

ehelms reviewed Jul 18, 2024

View reviewed changes

definitions/scenarios/satellite_upgrade.rb Show resolved Hide resolved

evgeni added 5 commits July 22, 2024 09:07

add "no running tasks" check for Pulpcore

13ccc36

make wait_for_tasks configurable in NotRunning foreman tasks check

76070aa

check for running tasks before a backup is done

90d344c

optionally allow waiting for them to complete

stop workers during online backup

471a7a8

this ensures more consistent on-disk data

add Checks::Pulpcore::NoRunningTasks to update/upgrade scenarios

7d3c539

evgeni force-pushed the stop-workers-for-backup branch from b87f9b8 to 7d3c539 Compare July 22, 2024 07:35

This was referenced Jul 22, 2024

Consistently call the service "Pulp" #902

Open

Drop tags from checks/procedures #903

Open

ehelms approved these changes Aug 12, 2024

View reviewed changes

evgeni merged commit fdb2c09 into master Aug 13, 2024
8 checks passed

evgeni deleted the stop-workers-for-backup branch August 13, 2024 18:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stop workers during online backup #856

stop workers during online backup #856

evgeni commented May 28, 2024

evgeni commented Jun 7, 2024 •

edited

Loading

evgeni commented Jun 7, 2024 •

edited

Loading

evgeni Jun 14, 2024

ehelms Jun 14, 2024

evgeni Jul 22, 2024

evgeni commented Jun 14, 2024

evgeni commented Jun 17, 2024

ehelms commented Jul 12, 2024

evgeni commented Jul 13, 2024

evgeni commented Jul 13, 2024

stop workers during online backup #856

stop workers during online backup #856

Conversation

evgeni commented May 28, 2024

evgeni commented Jun 7, 2024 • edited Loading

evgeni commented Jun 7, 2024 • edited Loading

evgeni Jun 14, 2024

Choose a reason for hiding this comment

ehelms Jun 14, 2024

Choose a reason for hiding this comment

evgeni Jul 22, 2024

Choose a reason for hiding this comment

evgeni commented Jun 14, 2024

evgeni commented Jun 17, 2024

ehelms commented Jul 12, 2024

evgeni commented Jul 13, 2024

evgeni commented Jul 13, 2024

evgeni commented Jun 7, 2024 •

edited

Loading

evgeni commented Jun 7, 2024 •

edited

Loading