"HYPHYMPI" failing but "hyphy" working

[lrschaff]

Hello HyPhy team,

When using RELAX on 2.5.68, the "hyphy" (non MPI) version works perfectly fine on my dataset. However when using the MPI version, "HYPHYMPI" it seems to fail at "fitting the general descriptive model."

Running with:

$ srun HYPHYMPI RELAX --alignment ./gene_600.fa --tree ./536mammals_ML_tree.nwk --test Foreground --models All --srv Branch-site --multiple-hit "Double+Triple" --keepident true

And getting the error:

[...]
### Fitting the general descriptive (separate k per branch) model
Grid result 7 = -213986.1435143871 (best = -192163.4429078236)Error:

Master node received an error:Operation '^' is not implemented/defined for a AssociativeList
    While computing: utility.Extend(^(mpi.gJHXkMzL.arguments[2]),mpi.gJHXkMzL.result)

Function call stack
1 :  [namespace = mpi.gJHXkMzL] utility.Extend(^(arguments[2]),result);

    Keyword arguments:
        {
         "keepident":"true",
         "multiple-hit":"Double+Triple"
        }
-------
2 :  [namespace = mpi.aux.EaU_Gajr] Call((queue[from])[utility.getGlobalValue("terms.mpi.callback")],from,Eval(result),(queue[from])[utility.getGlobalValue("terms.mpi.arguments")]);

    Keyword arguments:
        {
         "keepident":"true",
         "multiple-hit":"Double+Triple"
        }
-------
3 :  [namespace = mpi.uWUXdajl] node=aux._handle_receieve(queue);

    Keyword arguments:
        {
         "keepident":"true",
         "multiple-hit":"Double+Triple"
        }
-------
4 :  [namespace = mpi.jfvuBvnJ] mpi.QueueComplete(queue);

    Keyword arguments:
        {
         "keepident":"true",
         "multiple-hit":"Double+Triple"
        }
-------
5 :  [namespace = DhYdAqBM] grid_results=mpi.ComputeOnGridSetValues(&likelihoodFunction,run_options[utility.getGlobalValue("terms.search_grid")],{0:model_objects,1:initial_values,2:run_options[utility.getGlobalValue("terms.run_options.proportional_branch_length_scaler")]},"mpi.ComputeOnGrid.SimpleEvaluatorWithValues","mpi.ComputeOnGrid.ResultHandler");

    Keyword arguments:
        {
         "keepident":"true",
         "multiple-hit":"Double+Triple"
        }
-------
6 :  relax.grid_search.results=estimators.FitLF(relax.filter_names,relax.trees,relax.ge_model_map,relax.final_partitioned_mg_results,relax.model_object_map,{"retain-lf-object":TRUE,terms.run_options.apply_user_constraints:"relax.init.k",terms.run_options.proportional_branch_length_scaler:relax.nm.bl_scalers,terms.run_options.optimization_settings:{"OPTIMIZATION_METHOD":"nedler-mead","MAXIMUM_OPTIMIZATION_ITERATIONS":500,"OPTIMIZATION_PRECISION":relax.nm.precision},terms.search_grid:relax.initial_grid,terms.search_restarts:relax.N.initial_guesses,terms.run_options.optimization_log:relax.optimization_log_file(".GE-1-log.json")});

    Keyword arguments:
        {
         "keepident":"true",
         "multiple-hit":"Double+Triple"
        }
-------
[...]

Any insight on how to fix this error will be greatly appreciated.

Thank you!

Lillie

[spond]

Dear @lrschaff,

Looks like a bug with something not being properly exported to compute nodes by MPI. I'll mark it as such and investigate. Thanks for reporting!

Best,
Sergei

[lrschaff]

Thank you for looking into it!

Best,

Lillie

[lrschaff]

Hello Sergei,

I noticed that this bug was listed as a milestone for version 2.5.69, but the latest release does not appear to include a fix. When testing the latest code from the master branch, it seems the issue still persists. Additionally, it seems the development branch is currently behind the master, with no unique commits.

Best,

Lillie