- About Badfish
- Scope
- Features
- Requirements
- Setup
- Usage
- Usage via Podman
- Common Operations
- Enforcing an OpenStack Director-style interface order
- Enforcing a Foreman-style interface order
- Enforcing a Custom interface order
- Forcing a one time boot to a specific device
- Forcing a one time boot to a specific mac address
- Forcing a one time boot to a specific type
- Forcing a one-time boot to PXE
- Rebooting a System
- Power Cycling a System
- Power State Control
- Check Power State
- Resetting iDRAC
- BIOS factory reset
- Check current boot order
- Variable number of retries
- Firmware inventory
- Clear Job Queue
- List Job Queue
- List Network Interfaces
- List Memory
- List Processors
- Check Virtual Media
- Unmount Virtual Media
- Get SRIOV mode
- Set SRIOV mode
- Bulk actions via text file with list of hosts
- Verbose Output
- Log to File
- iDRAC and Data Format
- Contributing
- Contact
Badfish is a Redfish-based API tool for managing bare-metal systems via the Redfish API
You can read more about badfish at the QUADS website.
Right now Badfish is focused on managing Dell, SuperMicro and HPE systems, but can potentially work with any system that supports the Redfish API. Functionality may vary depending on the vendor Redfish implementation.
We're mostly concentrated on programmatically enforcing interface/device boot order to accommodate TripleO based OpenStack and OpenShift deployments while simultaneously allowing easy management and provisioning of those same systems via The Foreman. Badfish can be useful as a general standalone, unified vendor IPMI/OOB tool however as support for more vendors is added.
- Toggle and save a persistent interface/device boot order on remote systems
- Perform one-time boot to a specific interface, mac address or device listed for PXE booting
- Enforce a custom interface boot order
- Check current boot order
- Reboot host
- Reset iDRAC
- Clear iDRAC job queue
- Revert to factory settings
- Check/set SRIOV
- Get firmware inventory of installed devices supported by iDRAC
- Check/ummount virtual media en-masse across a set of systems
- Obtain limited hardware information (CPU, Memory, Interfaces)
- Bulk actions via plain text file with list of hosts for parallel execution
- Logging to a specific path
- Containerized Badfish image
- (Dell) iDRAC7,8,9 or newer
- (Dell) Firmware version
2.60.60.60or higher - iDRAC administrative account
- Python >=
3.6or podman as a container. - python3-devel >=
3.6(If using standalone below).
git clone https://github.com/redhat-performance/badfish && cd badfish
python setup.py build
python setup.py install --prefix ~/.local
NOTE:
- This will allow Badfish to be called from the terminal via the
badfishcommand - This requires
python3-develif you see errors about missingPython.h. - This is ideal for a non-root user, otherwise you'll get badfish in
/root/.local/bin/badfishfor example. - If you have problems running as root you will need to add whatever you set in
--prefix=to your$PATHby adding something like the following to the end of your~/.bashrcfile.
if [ -d "$HOME/.local/bin" ] ; then
PATH="$PATH:$HOME/.local/bin"
fi
git clone https://github.com/redhat-performance/badfish && cd badfish
pip install -r requirements.txt
NOTE:
- This will allow the badfish script execution via
./src/badfish/badfish.py
git clone https://github.com/redhat-performance/badfish && cd badfish
virtualenv .badfish_venv
source .badfish_venv/bin/activate
NOTE:
- Both setup methods above can be used within a virtualenv
- After using badfish, the virtual environment can be deactivated running the
deactivatecommand
Badfish operates against a YAML configuration file to toggle between key:value pair sets of boot interface/device strings. You just need to create your own interface config that matches your needs to easily swap/save interface/device boot ordering or select one-time boot devices.
Badfish happily runs in a container image using podman, for this you need to first pull the Badfish image via:
podman pull quay.io/quads/badfish
You can then run badfish from inside the container:
podman run -it --rm --dns $DNS_IP quay.io/quads/badfish -H $HOST -u $USER -p $PASS --reboot-only
NOTE:
- If you are running quads against a host inside a VPN you must specify your VPN DNS server ip address with
--dns - If you would like to use a different file for
config/idrac_interfaces.ymlyou can map a volume to your modified config with-v idrac_interfaces.yml:config/idrac_interfaces.yml
In our performance/scale R&D environments TripleO-based OpenStack deployments require a specific 10/25/40GbE NIC to be the primary boot device for PXE, followed by disk, and then followed by the rest of the interfaces.
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass -i config/idrac_interfaces.yml -t director
Foreman and Red Hat Satellite (as of 6.x based on Foreman) require managed systems to first always PXE from the interface that is Foreman-managed (DHCP/PXE). If the system is not set to build it will simply boot to local disk. In our setup we utilize a specific NIC for this interface based on system type.
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass -i config/idrac_interfaces.yml -t foreman
Badfish allows you to supply your own interface order type in addition to director and foreman modes as defined in idrac_interfaces.yml
- Supply your own distinct string in the first part of the key value (split by
_) - Refer to it via the string name
- Consequently host type overrides can also be leveraged
We will use the custom interface order called ocp5beta as an example.
Example any system you want to boot with a certain custom interface order.
ocp5beta_fc640_interfaces: NIC.Slot.2-4,NIC.Slot.2-1,NIC.Slot.2-2,NIC.Slot.2-3
Example a rack of systems you want to boot with a certain custom interface order.
ocp5beta_f21_fc640_interfaces: NIC.Slot.2-4,NIC.Slot.2-1,NIC.Slot.2-2,NIC.Slot.2-3
Example a specific system you want to boot with a certain custom interface order
ocp5beta_f21_h23_fc640_interfaces: NIC.Slot.2-4,NIC.Slot.2-1,NIC.Slot.2-2,NIC.Slot.2-3
Now you can run Badfish against the custom interface order type you have defined, refer to the custom overrides on further usage examples.
src/badfish/badfish.py --host-list /tmp/hosts -u root -p password -i config/idrac_interfaces.yml -t ocp5beta
To force systems to perform a one-time boot to a specific device you can use the --boot-to option and pass as an argument the device you want the one-time boot to be set to. This will change the one time boot BIOS attributes OneTimeBootMode and OneTimeBootSeqDev and on the next reboot it will attempt to PXE boot or boot from that interface string. You can obtain the device list via the --check-boot directive below.
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass --boot-to NIC.Integrated.1-3-1
To force systems to perform a one-time boot to a specific mac address you can use the --boot-to-mac option and pass as an argument the device mac address for a specific NIC that you want the one-time boot to be set to. This will change the one time boot BIOS attributes OneTimeBootMode and OneTimeBootSeqDev and on the next reboot it will attempt to PXE boot or boot from that interface.
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass --boot-to-mac A9:BB:4B:50:CA:54
To force systems to perform a one-time boot to a specific type you can use the --boot-to-type option and pass as an argument the device type, as defined on the iDRAC interfaces yaml, that you want the one-time boot to be set to. For this action you must also include the path to your interfaces yaml. This will change the one time boot BIOS attributes OneTimeBootMode and OneTimeBootSeqDev and on the next reboot it will attempt to PXE boot or boot from the first interface defined for that host type on the interfaces yaml file.
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass -i config/idrac_interfaces.yml --boot-to-type foreman
To force systems to perform a one-time boot to PXE, simply pass the --pxe flag to any of the commands above, by default it will pxe off the first available device for PXE booting.
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass -i config/idrac_interfaces.yml -t foreman --pxe
In certain cases you might need to only reboot the host, for this case we included the --reboot-only flag which will force a GracefulRestart on the target host. Note that this option is not to be used with any other option.
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass --reboot-only
For a hard reset you can use --power-cycle flag which will run a ForceOff instruction on the target host. Note that this option is not to be used with any other option.
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass --power-cycle
You can also turn a server on or off by using options --power-on and --power-off respectively.
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass --power-on
For checking the current power state of a server you can run badfish with the --power-state option.
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass --power-state
Partial Output:
- INFO - Power state for mgmt-your-server.example.com: On
For the replacement of racadm racreset, the optional argument --racreset was added. When this argument is passed to badfish, a graceful restart is triggered on the iDRAC itself.
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass --racreset
You can restore BIOS default settings by calling Badfish with the option --factory-reset.
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass --factory-reset
NOTE:
- WARNING: Use this carefully, vendor defaults differ and may be disruptive. Do not use this in the Scale Lab or ALIAS.
To check the current boot order of a specific host you can use the --check-boot option which will return an ordered list of boot devices. Additionally you can pass the -i option which will in turn print on screen what type of host does the current boot order match as those defined on the iDRAC interfaces yaml.
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass -i config/idrac_interfaces.yml --check-boot
At certain points during the execution of badfish the program might come across a non responsive resources and will automatically retry to establish connection. We have included a default value of 15 retries after failed attempts but this can be customized via the --retries optional argument which takes as input an integer with the number of desired retries.
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass -i config/idrac_interfaces.yml -t foreman --retries 20
If you would like to get a detailed list of all the devices supported by iDRAC you can run badfish with the --firware-inventory option which will return a list of devices with additional device info.
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass --firmware-inventory
If you would like to clear all the jobs that are queued on the remote iDRAC you can run badfish with the --clear-jobs option which query for all active jobs in the iDRAC queue and will post a request to clear the queue.
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass --clear-jobs
You can also force the clearing of Dell iDRAC job queues by passing the --force option.
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass --clear-jobs --force
If you would like to list all active jobs that are queued on the remote iDRAC you can run badfish with the --ls-jobs option which query for all active jobs in the iDRAC queue and will return a list with all active items.
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass --ls-jobs
For getting a list of network interfaces with individual metadata for each you can run badfish with the --ls-interfaces option.
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass --ls-interfaces
For getting a detailed list of memory devices you can run badfish with the --ls-memory option.
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass --ls-memory
For getting a detailed list of processors you can run badfish with the --ls-processors option.
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass --ls-processors
If you would like to check for any active virtual media you can run badfish with the --check-virtual-media option which query for all active virtual devices.
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass --check-virtual-media
If you would like to unmount all active virtual media you can run badfish with the --unmount-virtual-media option which post a request for unmounting all active virtual devices.
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass --unmount-virtual-media
NOTE:
- This functionality is only available for SuperMicro devices.
For checking if the global SRIOV mode is enabled you can use --get-sriov
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass --get-sriovNOTE:
- This is only supported on DELL devices.
For changing the mode of the SRIOV glabal BIOS attribute, we have included 2 new arguments.
In case the setting is in disabled mode, you can enable it by passing --enable-sriov
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass --enable-sriovOn the contrary, if you would like to disable the SRIOV mode, you can now pass --disable-sriov
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass --disable-sriovNOTE:
- This is only supported on DELL devices.
In the case you would like to execute a common badfish action on a list of hosts, you can pass the optional argument --host-list in place of -H with the path to a text file with the hosts you would like to action upon and any addtional arguments defining a common action for all these hosts.
./src/badfish/badfish.py --host-list /tmp/bad-hosts -u root -p yourpass --clear-jobs
If you would like to see a more detailed output on console you can use the --verbose option and get a additional debug logs. Note: this is the default log level for the --log argument.
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass -i config/idrac_interfaces.yml -t foreman --verbose
If you would like to log the output of badfish you can use the --log option and pass the path to where you want badfish to log it's output to.
./src/badfish/badfish.py -H mgmt-your-server.example.com -u root -p yourpass -i config/idrac_interfaces.yml -t foreman --log /tmp/bad.log
Your usage may vary, this is what our configuration looks like via config/idrac_interfaces.yml
| Machine Type | Network Interface |
|---|---|
| Dell fc640 | NIC.Integrated.1-1-1 |
| Dell r620 | NIC.Integrated.1-3-1 |
| Dell r630 | NIC.Slot.2-1-1 |
| Dell r930 | NIC.Integrated.1-3-1 |
| Dell r720xd | NIC.Integrated.1-3-1 |
| Dell r730xd | NIC.Integrated.1-3-1 |
| Dell r740xd | NIC.Integrated.1-3-1 |
| Dell r640 | NIC.Integrated.1-1-1 |
Every other method that requires passing the -i argument, is going to parse the key strings from this and look for the most adequate candidate for the given FQDN.
We format the key strings with the following criteria:
{host_type}_[{rack}_[{ULocation}_[{blade}_]]]{model}_interfaces
With rack, ULocation and blade being optional in a hierarchical fashion otherwise mandatory, ergo you can't define blade without ULocation and so forth. host_type and model values are always mandatory.
| Keys defined on interfaces yaml | FQDN | Use boot order |
|---|---|---|
| director_r620_interfaces | mgmt-f21-h17-000-r620.domain.com | NO |
| director_f21_r620_interfaces | mgmt-f21-h17-000-r620.domain.com | NO |
| director_f21_h17_r620_interfaces | mgmt-f21-h17-000-r620.domain.com | YES |
| Keys defined on interfaces yaml | FQDN | Use boot order |
|---|---|---|
| director_r620_interfaces | mgmt-f21-h18-000-r620.domain.com | NO |
| director_f21_r620_interfaces | mgmt-f21-h18-000-r620.domain.com | YES |
| director_f21_h17_r620_interfaces | mgmt-f21-h18-000-r620.domain.com | NO |
| Keys defined on interfaces yaml | FQDN | Use boot order |
|---|---|---|
| director_r620_interfaces | mgmt-f22-h17-000-r620.domain.com | YES |
| director_f21_r620_interfaces | mgmt-f22-h17-000-r620.domain.com | NO |
| director_f21_h17_r620_interfaces | mgmt-f22-h17-000-r620.domain.com | NO |
We love pull requests and welcome contributions from everyone! Please use the development branch to send pull requests. Here are the general steps you'd want to follow.
- Fork the Badfish Github repository
- Clone the forked repository
- Push your changes to your forked clone
- Open a pull request against our
developmentbranch.
- Here is some useful documentation
- You can find us on IRC in
#badfish(or#quads) onirc.libera.chatif you have questions or need help. Click here to join in your browser.