keep record stack trace of failed relvals/unit tests in opensearch

[smuzaffar]

For each relval, we keep trace of all %MSG-e ... messages and cms exceptions. We should also keep trace of stack trace (if available). For example a failed relvals with output like

#0  0x000040003def6960 in poll () from /lib64/libc.so.6
#1  0x0000400042fb5100 in edm::service::InitRootHandlers::stacktraceFromThread() () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02872/el9_aarch64_gcc12/cms/cmssw/CMSSW_15_0_X_2025-01-13-2300/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000400042fb5334 in sig_dostack_then_abort () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02872/el9_aarch64_gcc12/cms/cmssw/CMSSW_15_0_X_2025-01-13-2300/lib/el9_aarch64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000400131ffe038 in ?? ()
#5  0x00004000514463b0 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02872/el9_aarch64_gcc12/cms/cmssw/CMSSW_15_0_X_2025-01-13-2300/external/el9_aarch64_gcc12/lib/libCling.so
#6  0x00004000514463b0 in cling::Interpreter::RunFunction(clang::FunctionDecl const*, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02872/el9_aarch64_gcc12/cms/cmssw/CMSSW_15_0_X_2025-01-13-2300/external/el9_aarch64_gcc12/lib/libCling.so
#7  0x000040005144751c in cling::Interpreter::EvaluateInternal(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cling::CompilationOptions, cling::Value*, cling::Transaction**, unsigned long) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02872/el9_aarch64_gcc12/cms/cmssw/CMSSW_15_0_X_2025-01-13-2300/external/el9_aarch64_gcc12/lib/libCling.so
#8  0x000040005152b384 in cling::MetaSema::actOnxCommand(llvm::StringRef, llvm::StringRef, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02872/el9_aarch64_gcc12/cms/cmssw/CMSSW_15_0_X_2025-01-13-2300/external/el9_aarch64_gcc12/lib/libCling.so
#9  0x000040005153aa8c in cling::MetaParser::isXCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02872/el9_aarch64_gcc12/cms/cmssw/CMSSW_15_0_X_2025-01-13-2300/external/el9_aarch64_gcc12/lib/libCling.so
#10 0x000040005153b090 in cling::MetaParser::isCommand(cling::MetaSema::ActionResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02872/el9_aarch64_gcc12/cms/cmssw/CMSSW_15_0_X_2025-01-13-2300/external/el9_aarch64_gcc12/lib/libCling.so
#11 0x000040005152487c in cling::MetaProcessor::process(llvm::StringRef, cling::Interpreter::CompilationResult&, cling::Value*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02872/el9_aarch64_gcc12/cms/cmssw/CMSSW_15_0_X_2025-01-13-2300/external/el9_aarch64_gcc12/lib/libCling.so
#12 0x000040005135f37c in HandleInterpreterException(cling::MetaProcessor*, char const*, cling::Interpreter::CompilationResult&, cling::Value*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02872/el9_aarch64_gcc12/cms/cmssw/CMSSW_15_0_X_2025-01-13-2300/external/el9_aarch64_gcc12/lib/libCling.so
#13 0x00004000513752b8 in TCling::ProcessLine(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02872/el9_aarch64_gcc12/cms/cmssw/CMSSW_15_0_X_2025-01-13-2300/external/el9_aarch64_gcc12/lib/libCling.so
#14 0x00004000513757f4 in TCling::ProcessLineSynch(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02872/el9_aarch64_gcc12/cms/cmssw/CMSSW_15_0_X_2025-01-13-2300/external/el9_aarch64_gcc12/lib/libCling.so
#15 0x000040003d7f8fa8 in TApplication::ExecuteFile(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02872/el9_aarch64_gcc12/cms/cmssw/CMSSW_15_0_X_2025-01-13-2300/external/el9_aarch64_gcc12/lib/libCore.so
#16 0x0000400051368edc in TCling::ExecuteMacro(char const*, TInterpreter::EErrorCode*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02872/el9_aarch64_gcc12/cms/cmssw/CMSSW_15_0_X_2025-01-13-2300/external/el9_aarch64_gcc12/lib/libCling.so
#17 0x000040003d850d9c in TROOT::Macro(char const*, int*, bool) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02872/el9_aarch64_gcc12/cms/cmssw/CMSSW_15_0_X_2025-01-13-2300/external/el9_aarch64_gcc12/lib/libCore.so
#18 0x000040003d8346a0 in TPluginManager::LoadHandlerMacros(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02872/el9_aarch64_gcc12/cms/cmssw/CMSSW_15_0_X_2025-01-13-2300/external/el9_aarch64_gcc12/lib/libCore.so
#19 0x000040003d834cfc in TPluginManager::LoadHandlersFromPluginDirs(char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02872/el9_aarch64_gcc12/cms/cmssw/CMSSW_15_0_X_2025-01-13-2300/external/el9_aarch64_gcc12/lib/libCore.so
#20 0x000040003d835d84 in TPluginManager::FindHandler(char const*, char const*) () from /cvmfs/cms-ib.cern.ch/sw/aarch64/nweek-02872/el9_aarch64_gcc12/cms/cmssw/CMSSW_15_0_X_2025-01-13-2300/external/el9_aarch64_gcc12/lib/libCore.so

we should store this stack trace ( starting from sig_dostack_then_abort ) and keep at least some reasonable number of lines (may be 20?)

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @smuzaffar.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here