Skip to content

Commit

Permalink
Merge pull request #1 from mberacochea/feature/slurm
Browse files Browse the repository at this point in the history
First working version that supports LSF and Slurm
  • Loading branch information
mberacochea authored Feb 26, 2024
2 parents a2c4c9d + 279ba7d commit ec5edaa
Show file tree
Hide file tree
Showing 10 changed files with 736 additions and 341 deletions.
107 changes: 75 additions & 32 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,37 +1,80 @@
# mjobs

A little tool to make it easier to inspect LSF jobs.

```shell
Usage: mjobs [-h] [-q QUEUE] [-u USER] [-r] [-a] [-d] [-G USER_GROUP] [-g GROUP] [-m HOSTS] [-p] [-e] [-f FILTER] [-t] [-nh] [--bkill] [job_id ...]

bjobs but a bit nicer

Positional Arguments:
job_id Specifies the jobs or job arrays that bjobs displays.

Optional Arguments:
-h, --help show this help message and exit
-q QUEUE Displays jobs in the specified queue
-u USER Displays jobs in the specified user
-r Displays running jobs.
-a Displays information about jobs in all states, including jobs that finished recently.
-d Displays information about jobs that finished recently.
-G USER_GROUP Displays jobs associated with the specified user group.
-g GROUP Displays information about jobs attached to the specified job group.
-m HOSTS Displays jobs dispatched to the specified hosts.
-p Displays pending jobs, together with the pending reasons that caused each job not to be dispatched during the last dispatch turn.
-e Add the execution josts, output file and error file to the table.
-f FILTER Filter the jobs using the specified regex on the job name or pending reason.
-t No fancy table, a good ol' tsv
-nh Don't print the table header, useful to pipe the tsv output
--bkill Terminate found or filtered jobs with bkill.

```

## Example

![Alt text](images/mjobs-example.png?raw=true "mjobs example")
A little tool to make it easier to inspect [IBM Spectrum LSF](https://www.ibm.com/products/hpc-workload-management) (or just LSF) and [Slurm](https://slurm.schedmd.com/) jobs.

The program will auto detect [bjobs](https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=bjobs-options) under [LSF](https://www.ibm.com/products/hpc-workload-management), or [squeue](https://slurm.schedmd.com/squeue.html) in [Slurm](https://slurm.schedmd.com/).

## IBM LSF

mjobs doesn't not support all the options of bjobs, only a subset. It also adds a the `-f FILTER`, `--bkill`, `-ts` and `-nh` options.

<pre>[mbc@codon-login-02 ~]$ ./mjobs -h
<font color="#FF8700">Usage:</font> <font color="#808080">mjobs</font> [<font color="#05979A">-h</font>] [<font color="#05979A">-f</font> <font color="#00AF87">FILTER</font>] [<font color="#05979A">-ts</font>] [<font color="#05979A">-nh</font>] [<font color="#05979A">-q</font> <font color="#00AF87">QUEUE</font>] [<font color="#05979A">-u</font> <font color="#00AF87">USER</font>] [<font color="#05979A">-r</font>] [<font color="#05979A">-a</font>] [<font color="#05979A">-d</font>] [<font color="#05979A">-G</font> <font color="#00AF87">USER_GROUP</font>] [<font color="#05979A">-g</font> <font color="#00AF87">GROUP</font>] [<font color="#05979A">-m</font> <font color="#00AF87">HOSTS</font>] [<font color="#05979A">-p</font>] [<font color="#05979A">-e</font>] [<font color="#05979A">--bkill</font>] [<font color="#05979A">job_id</font> ...]

Just like bjobs but a bit nicer

<font color="#FF8700">Positional Arguments:</font>
<font color="#05979A">job_id</font> Specifies the jobs or job arrays that bjobs displays.

<font color="#FF8700">Optional Arguments:</font>
<font color="#05979A">-h</font>, <font color="#05979A">--help</font> show this help message and exit
<font color="#05979A">-f</font> <font color="#00AF87">FILTER</font> Filter the jobs using the specified regex on the job name or pending reason.
<font color="#05979A">-ts</font>, <font color="#05979A">--tsv</font> No fancy table, a good ol&apos; tsv
<font color="#05979A">-nh</font> Don&apos;t print the table header, useful to pipe the tsv output
<font color="#05979A">-q</font> <font color="#00AF87">QUEUE</font> Displays jobs in the specified queue
<font color="#05979A">-u</font> <font color="#00AF87">USER</font> Displays jobs in the specified user
<font color="#05979A">-r</font> Displays running jobs.
<font color="#05979A">-a</font> Displays information about jobs in all states, including jobs that finished recently.
<font color="#05979A">-d</font> Displays information about jobs that finished recently.
<font color="#05979A">-G</font> <font color="#00AF87">USER_GROUP</font> Displays jobs associated with the specified user group.
<font color="#05979A">-g</font> <font color="#00AF87">GROUP</font> Displays information about jobs attached to the specified job group.
<font color="#05979A">-m</font> <font color="#00AF87">HOSTS</font> Displays jobs dispatched to the specified hosts.
<font color="#05979A">-p</font> Displays pending jobs, together with the pending reasons that caused each job not to be dispatched during the last dispatch turn.
<font color="#05979A">-e</font> Add the execution hosts, output file and error file to the table.
<font color="#05979A">--bkill</font> Terminate found or filtered jobs with bkill.</pre>

### Example

<pre>[mbc@codon-login-02 ~]$ mjobs -u emgpr_wgs -f SRR12480298_mgnify_analysis
<i> LSF jobs for emgpr_wgs </i>
┏━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
┃<b> JobId </b>┃<b> Status </b>┃<b> JobName </b>┃<b> JobGroup </b>┃<b> User </b>┃<b> Queue </b>┃<b> Submit Time </b>┃<b> Start Time </b>┃<b> Finish Time </b>┃<b> Pending reason </b>┃
┡━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
│ 87346050 │ <font color="#89E234"><b>RUN </b></font> │ <font color="#EF2828"><b>SRR12480298_mgnify_analysis</b></font> │ ---- │ emgpr_wgs │ production │ Feb 25 08:00 │ Feb 25 08:03 │ Feb 28 08:03 L │ ---- │
└──────────┴────────┴─────────────────────────────┴──────────┴───────────┴────────────┴──────────────┴──────────────┴────────────────┴────────────────┘
</pre>

## Slurm

mjobs doesn't not support all the options of squeue, only a subset. It also adds a the `-f FILTER`, `-ts` and `-nh` options.

<pre>[mbc@codon-slurm-login-01 ~]$ ./mjobs -h
<font color="#FF8700">Usage:</font> <font color="#808080">mjobs</font> [<font color="#05979A">-h</font>] [<font color="#05979A">-f</font> <font color="#00AF87">FILTER</font>] [<font color="#05979A">-ts</font>] [<font color="#05979A">-nh</font>] [<font color="#05979A">-p</font> <font color="#00AF87">PARTITION</font>] [<font color="#05979A">-u</font> <font color="#00AF87">USER</font>]
[<font color="#05979A">-t</font> <font color="#00AF87">{pending,running,suspended,completed,cancelled,failed,timeout,node_fail,preempted,boot_fail,deadline,out_of_memory,completing,configuring,resizing,resv_del_hold,requeued,requeue_fed,requeue_hold,revoked,signaling,special_exit,stage_out,stopped} [{pending,running,suspended,completed,cancelled,failed,timeout,node_fail,preempted,boot_fail,deadline,out_of_memory,completing,configuring,resizing,resv_del_hold,requeued,requeue_fed,requeue_hold,revoked,signaling,special_exit,stage_out,stopped} ...]</font>]
[<font color="#05979A">-w</font> <font color="#00AF87">NODELIST [NODELIST ...]</font>] [<font color="#05979A">-e</font>]
[<font color="#05979A">job_id</font> ...]

Just like squeue but a bit nicer

<font color="#FF8700">Positional Arguments:</font>
<font color="#05979A">job_id</font> Specifies the jobs or job arrays that squeue displays.

<font color="#FF8700">Optional Arguments:</font>
<font color="#05979A">-h</font>, <font color="#05979A">--help</font> show this help message and exit
<font color="#05979A">-f</font> <font color="#00AF87">FILTER</font> Filter the jobs using the specified regex on the job name or pending reason.
<font color="#05979A">-ts</font>, <font color="#05979A">--tsv</font> No fancy table, a good ol&apos; tsv
<font color="#05979A">-nh</font> Don&apos;t print the table header, useful to pipe the tsv output
<font color="#05979A">-p</font>, <font color="#05979A">--partition</font> <font color="#00AF87">PARTITION</font>
Specify the partitions of the jobs or steps to view. Accepts a comma separated list of partition names.
<font color="#05979A">-u</font>, <font color="#05979A">--user</font> <font color="#00AF87">USER</font> Request jobs or job steps from a comma separated list of users. The list can consist of user names or user id numbers. Performance of the command can be measurably improved for systems with large numbers of jobs when a
single user is specified.
<font color="#05979A">-t</font>, <font color="#05979A">--states</font> <font color="#00AF87">{pending,running,suspended,completed,cancelled,failed,timeout,node_fail,preempted,boot_fail,deadline,out_of_memory,completing,configuring,resizing,resv_del_hold,requeued,requeue_fed,requeue_hold,revoked,signaling,special_exit,stage_out,stopped} [{pending,running,suspended,completed,cancelled,failed,timeout,node_fail,preempted,boot_fail,deadline,out_of_memory,completing,configuring,resizing,resv_del_hold,requeued,requeue_fed,requeue_hold,revoked,signaling,special_exit,stage_out,stopped} ...]</font>
Specify the states of jobs to view. Accepts a comma separated list of state names or &apos;all&apos;. If &apos;all&apos; is specified then jobs of all states will be reported. If no state is specified then pending, running, and completing jobs
are reported. See the JOB STATE CODES section below for a list of valid states. Both extended and compact forms are valid. Note the &lt;state_list&gt; supplied is case insensitive (&apos;pending&apos; and &apos;PENDING&apos; are equivalent).
<font color="#05979A">-w</font>, <font color="#05979A">--nodelist</font> <font color="#00AF87">NODELIST [NODELIST ...]</font>
Report only on jobs allocated to the specified node or list of nodes. This may either be the NodeName or NodeHostname as defined in slurm.conf(5) in the event that they differ. A node_name of localhost is mapped to the
current host name.
<font color="#05979A">-e</font> Add the execution nodes, stdoutput file and stderror file to the table.</pre>

# Installation - bundle the app

Expand Down
1 change: 1 addition & 0 deletions Taskfile.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,4 @@ tasks:
build:
cmds:
- pyinstaller mjobs/main.py --onefile --clean --name mjobs
- scp dist/mjobs codon:~/
Binary file removed images/mjobs-example.png
Binary file not shown.
90 changes: 90 additions & 0 deletions mjobs/base.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
#!/usr/bin/env python
# -*- coding: utf-8 -*-

# Copyright 2021-2024 - Martin Beracochea
#
# Licensed under the Apache License, Version 2.0 (the 'License');
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an 'AS IS' BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import argparse
import csv
import sys
from abc import ABC

from rich.console import Console
from rich.table import Table
from rich_argparse import RichHelpFormatter


class Base(ABC):
def __init__(self, console: Console, error_console: Console) -> None:
"""console is the default one to print content
error_console will be used to print error messages
"""
self.console = console
self.error_console = console
super().__init__()

def get_args(self, implementation_name: str):
"""Base arguments that every implementation should support"""
parser = argparse.ArgumentParser(
description=f"Just like {implementation_name} but a bit nicer",
formatter_class=RichHelpFormatter,
)
parser.add_argument(
"-f",
dest="filter",
required=False,
help="Filter the jobs using the specified regex on the job name or pending reason.",
)
parser.add_argument(
"-ts",
"--tsv",
dest="tsv",
action="store_true",
help="No fancy table, a good ol' tsv",
)
parser.add_argument(
"-nh",
dest="no_header",
action="store_true",
help="Don't print the table header, useful to pipe the tsv output",
)
return parser

def render(
self,
title: str,
columns: list[dict[str, any]],
rows: list[dict[str, any]],
):
"""Render the jobs"""
if self.args.tsv:
# print with no styles
writer = csv.writer(sys.stdout, delimiter="\t")
if not self.args.no_header:
writer.writerow([c.get("header") for c in columns])
writer.writerows(rows)
else:
# print the fancy table
table = Table(
title=title, show_lines=True, show_header=not self.args.no_header
)
for col in columns:
table.add_column(**col)
for row in rows:
table.add_row(*row)
self.console.print(table)

def main(self):
"""Main execution point.
Should handle the logic to get jobs, build the table and any other features"""
pass
Loading

0 comments on commit ec5edaa

Please sign in to comment.