Complaints about calling the upper() function on a NoneType object

[bieryAtFnal]

I've noticed that when a daq process crashes, drunc seems to encounter problems with calling the upper() function on a NoneType object.

Here is a sample of the error message that I see:

ERROR    "controller_driver": Command 'execute_fsm_command' failed on 'df-01' (response flag                 shell_utils.py:138
                    'UNHANDLED_EXCEPTION_THROWN')                                                                                         
           ERROR    "controller_driver": Exception thrown from child: Stacktrace [bold red]on remote server![/bold red] shell_utils.py:123
                    Traceback (most recent call last):                                                                                    
                      File                                                                                                                
                    "/home/nfs/biery/dunedaq/20Aug20AugNightlyFDv5Test_1332/.venv/lib/python3.10/site-packages/drunc/co                   
                    ntroller/controller.py", line 290, in propagate_to_child                                                              
                        response = child.propagate_command(command, command_data, token)                                                  
                      File                                                                                                                
                    "/home/nfs/biery/dunedaq/20Aug20AugNightlyFDv5Test_1332/.venv/lib/python3.10/site-packages/drunc/co                   
                    ntroller/children_interface/rest_api_child.py", line 522, in propagate_command                                        
                        exit_state = exit_state.upper(),                                                                                  
                    AttributeError: 'NoneType' object has no attribute 'upper'                                                            
                                                                                                                                          
                                                                                                                                          
                    AttributeError: 'NoneType' object has no attribute 'upper'                                                            
                     df-01 -> 6                                                                                                           

I'll post sample instructions to demonstrate this behavior next.

[bieryAtFnal]

Here are sample steps for demonstrating this behavior:

DATE_PREFIX=`date '+%d%b'`
TIME_SUFFIX=`date '+%H%M'`

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt latest_v5
dbt-create -n NFD_DEV_240820_A9 ${DATE_PREFIX}20AugNightlyFDv5Test_${TIME_SUFFIX}
cd ${DATE_PREFIX}20AugNightlyFDv5Test_${TIME_SUFFIX}/sourcecode

git clone https://github.com/DUNE-DAQ/appmodel.git -b develop
cd appmodel ; git checkout 4e34d68adc4; cd ..
git clone https://github.com/DUNE-DAQ/fdreadoutlibs.git -b develop
cd fdreadoutlibs; git checkout ab0d71cc4f9; cd ..
git clone https://github.com/DUNE-DAQ/fdreadoutmodules.git -b develop
cd fdreadoutmodules; git checkout 105a58f82cc2; cd ..
cd ..

sed -i 's/unpack_one_register(second_half)/unpack_one_register(first_half)/' sourcecode/fdreadoutlibs/include/fdreadoutlibs/wibeth/tpg/FrameExpand.hpp 

dbt-workarea-env
dbt-build -j 12
dbt-workarea-env

mkdir rundir
cd rundir

# Execute the following commands by hand:

killall drunc-controller
drunc-unified-shell ssh-standalone

# within drunc

boot test/config/test-session.data.xml test-session
fsm conf
fsm start run_number 101

# outside of drunc:
# kill the df-01 application

# back inside drunc:

fsm drain_dataflow
fsm stop_trigger_sources
fsm stop
fsm scrap

# note the error messages printed out as part of these last several commands
exit