RELAX internal errors

[00-kelvin]

Hi there Sergei,

I have run RELAX on a set of 4,754 alignments, of which the vast majority were successful. However, 13 of the genes have produced a couple of unfamiliar "Internal errors" that I hoped you might be able to help me resolve.

The majority of the errors look something like this:

Error:
Internal error 
 _LikelihoodFunction::GradientLocateTheBump: in the Brent loop iteration 1. -54766.27880548131 / -54766.27880546547.
 Optimization direction: 
{
{0} 
{0} 
{1.715105996973671e-08} 
{1.716390808058301e-08} 
{0} 
{0} 
{0} 
{0} 
{0} 
{0} 
{0} 
{0} 
{0} 
{0} 
{0} 
{0} 
{0} 
{0} 
{0} 
{0} 
{0} 
{0} 

... [more zeroes corresponding to the number of branches] ...

{0} 
{0} 
}

Function call stack
1 :  [namespace = eUTMjqjz] Optimize(mles, ^lf_id);
-------
2 :  relax.null_model.fit=estimators.FitExistingLF(relax.alternative_model.fit[terms.likelihood_function],relax.model_object_map);
-------
3 :  relax.FitMainTestPair(relax.loop_passes==1);

One or 2 have run into a different error, which looks like:

Error:
Internal error 
Internal error in _SimplexMethod; final point log-likelihood does not match the best recorded log-L: -56113.76672277355 vs -55969.71332045213

Function call stack
1 :  [namespace = ZSwdPZJK] Optimize(mles, likelihoodFunction, run_options[utility.getGlobalValue("terms.run_options.optimization_settings")>
-------
2 :  relax.general_descriptive.fit=estimators.FitLF(relax.filter_names,relax.trees,relax.model_map,relax.general_descriptive.fit,relax.model>
-------
3 :  relax.FitMainTestPair(relax.loop_passes==1);
-------

On one occasion, a gene that initially errored out with the latter "SimplexMethod" error later errored out instead with the "GradientLocateTheBump" error upon re-running relax with more threads.

I am attaching a few of the alignment/tree files which gave these errors, and the script I used to run RELAX is below.

Thank you,
Calvin

#!/bin/bash

#SBATCH --job-name=250129_relax
#SBATCH --partition=parallel
#SBATCH --account=agordus1
#SBATCH --time=3-00:00:00
#SBATCH [email protected]
#SBATCH --mail-type=ALL
#SBATCH --array=1-2355
#SBATCH -n 3
#SBATCH --output=/data/agordus1/crunnel2/reports/%x/%A_%a.out

#make directory to store slurm reports
mkdir -p /data/agordus1/crunnel2/reports/$SBATCH_JOB_NAME/

module load anaconda
conda activate /home/crunnel2/anaconda3/envs/selection

WD=/data/agordus1/crunnel2/060125_N5_o75
HOG_LIST=${WD}/HOG_CDS/relax_rerun_list.txt
FG_NAME=ow_fg_cons

CURRENT_HOG=$(sed "${SLURM_ARRAY_TASK_ID}q;d" $HOG_LIST)

#check if RELAX has already completed for this HOG 
RELAX_OUT=${WD}/${CURRENT_HOG}/${CURRENT_HOG}_RELAX.json

if grep -q "p-value" ${RELAX_OUT}; then
	echo "RELAX already complete."
else
	#run relax
	hyphy relax \
	 CPU=${SLURM_NTASKS} \
	 --alignment ${WD}/${CURRENT_HOG}/${CURRENT_HOG}.fltrd.fasta \
	 --tree ${WD}/${CURRENT_HOG}/${CURRENT_HOG}.${FG_NAME}.tree \
	 --test "Unlabeled branches" \
	 --multiple-hits Double+Triple \
	 --models Minimal \
	 --srv Yes \
 	 ENV="TOLERATE_NUMERICAL_ERRORS=1;" \
	 --output ${RELAX_OUT}
fi

conda deactivate

relax_errors.zip

[spond]

Dear @00-kelvin,

Could you check the version of HyPhy you have? I was able to run a couple files without issues with the current version.

N5.HOG0022712.fltrd.fasta.RELAX.json.zip

N5.HOG0063733.fltrd.fasta.RELAX.json.zip

Best,
Sergei

[00-kelvin]

Interesting... I don't think it's a version issue, as I've made sure to keep updated and it has run successfully on many other genes in the dataset.

hyphy --version
HYPHY 2.5.64(MP) for Linux on x86_64 x86 SSE4 SIMD zlib (v1.3.1)

Was your command approximately the same as mine? And can I ask how many cores you used? I've been running these tests on a computing cluster and RELAX has often taken multiple days to run - it seems like you were able to get them done much faster than that.

[00-kelvin]

Oops, I just realized that there actually is a 2.5.65 now... I will update to that and see if that resolves the issue.

[spond]

Dear @00-kelvin,

Yes, exactly the same command as what you head in your script. I ran it on my M4 MBPro using "all available cores", which is 10.

The current version is actually 2.5.68.

Best,
Sergei

[00-kelvin]

Ah, ok. 2.5.65 seems to be the most recent version available through Anaconda -- I'll try that first and if I still have issues, maybe I will try building from source instead. Thank you! I'll let you know what I find out.