FileIO error

[biona001]

As reported by @GuJQ5, there is some problem with reading .h5 formatted LD files.

$ app_linux_x86/bin/GhostKnockoffGWAS --zfile UCB_GWAS.txt.gz --LD-files EUR --N 44161 --genome-build 19 --out UCB_GWAS_results

┌ Warning: Error requiring `UnPack` from `JLD2`
│ exception = 
│ IOError: stat("/home/groups/sabatti/.julia/packages/JLD2/ryhNR/ext/UnPackExt.jl"): permission denied (EACCES) 
│ Stacktrace: 
│ [1] uv_error 
│ @ ./libuv.jl:100 [inlined] 
│ [2] stat(path::String) 
│ @ Base.Filesystem ./stat.jl:152 
│ [3] isfile(path::String) 
│ @ Base.Filesystem ./stat.jl:461 
│ [4] macro expansion 
│ @ /home/groups/sabatti/.julia/packages/Requires/Z8rfN/src/Requires.jl:37 [inlined] 
│ [5] top-level scope 
│ @ /home/groups/sabatti/.julia/packages/JLD2/ryhNR/src/JLD2.jl:607 
│ [6] eval 
│ @ ./boot.jl:370 [inlined] 
│ [7] eval 
│ @ /home/groups/sabatti/.julia/packages/JLD2/ryhNR/src/JLD2.jl:1 [inlined] 
 [8] (::JLD2.var"#131#134")() 
│ @ JLD2 /home/groups/sabatti/.julia/packages/Requires/Z8rfN/src/require.jl:101 
│ [9] macro expansion 
│ @ ./timing.jl:393 [inlined] 
│ [10] err(f::Any, listener::Module, modname::String, file::String, line::Any) 
│ @ Requires /home/groups/sabatti/.julia/packages/Requires/Z8rfN/src/require.jl:47 
│ [11] (::JLD2.var"#130#133")() 
│ @ JLD2 /home/groups/sabatti/.julia/packages/Requires/Z8rfN/src/require.jl:100 
│ [12] withpath(f::Any, path::String) 
│ @ Requires /home/groups/sabatti/.julia/packages/Requires/Z8rfN/src/require.jl:37 
│ [13] (::JLD2.var"#129#132")() 
│ @ JLD2 /home/groups/sabatti/.julia/packages/Requires/Z8rfN/src/require.jl:99 
│ [14] listenpkg(f::Any, pkg::Base.PkgId) 
│ @ Requires /home/groups/sabatti/.julia/packages/Requires/Z8rfN/src/require.jl:20 
│ [15] macro expansion 
│ @ /home/groups/sabatti/.julia/packages/Requires/Z8rfN/src/require.jl:98 [inlined] 
│ [16] __init__() 
│ @ JLD2 /home/groups/sabatti/.julia/packages/JLD2/ryhNR/src/JLD2.jl:606 
└ @ Requires /home/groups/sabatti/.julia/packages/Requires/Z8rfN/src/require.jl:51 

┌ Warning: Error requiring `FileIO` from `HDF5` 
│ exception = 
│ IOError: stat("/home/groups/sabatti/.julia/packages/HDF5/HtnQZ/src/fileio.jl"): permission denied (EACCES) 
│ Stacktrace: 
│ [1] uv_error 
│ @ ./libuv.jl:100 [inlined] 
│ [2] stat(path::String) 
│ @ Base.Filesystem ./stat.jl:152 
│ [3] isfile(path::String) 
│ @ Base.Filesystem ./stat.jl:461 
│ [4] top-level scope 
│ @ /home/groups/sabatti/.julia/packages/Requires/Z8rfN/src/Requires.jl:37 
│ [5] eval 
│ @ ./boot.jl:370 [inlined] 
│ [6] eval 
│ @ /home/groups/sabatti/.julia/packages/HDF5/HtnQZ/src/HDF5.jl:1 [inlined] 
│ [7] (::HDF5.var"#115#121")() 
│ @ HDF5 /home/groups/sabatti/.julia/packages/Requires/Z8rfN/src/require.jl:101 
│ [8] macro expansion 
│ @ ./timing.jl:393 [inlined] 
│ [9] err(f::Any, listener::Module, modname::String, file::String, line::Any) 
│ @ Requires /home/groups/sabatti/.julia/packages/Requires/Z8rfN/src/require.jl:47 
│ [10] (::HDF5.var"#114#120")() 
│ @ HDF5 /home/groups/sabatti/.julia/packages/Requires/Z8rfN/src/require.jl:100 
│ [11] withpath(f::Any, path::String) 
│ @ Requires /home/groups/sabatti/.julia/packages/Requires/Z8rfN/src/require.jl:37 
│ [12] (::HDF5.var"#113#119")() 
│ @ HDF5 /home/groups/sabatti/.julia/packages/Requires/Z8rfN/src/require.jl:99 
│ [13] listenpkg 
│ @ /home/groups/sabatti/.julia/packages/Requires/Z8rfN/src/require.jl:20 [inlined] 
│ [14] macro expansion 
│ @ /home/groups/sabatti/.julia/packages/Requires/Z8rfN/src/require.jl:98 [inlined] 
│ [15] __init__() 
│ @ HDF5 /home/groups/sabatti/.julia/packages/HDF5/HtnQZ/src/HDF5.jl:119 
└ @ Requires /home/groups/sabatti/.julia/packages/Requires/Z8rfN/src/require.jl:51 

Welcome to GhostKnockoffGWAS analysis! 
You have specified the following options: 
zfile = /oak/stanford/groups/zihuai/UCB data/Mar_13_James/UCB_GWAS.txt.gz 
LD_files = /oak/stanford/groups/zihuai/UCB data/Mar_13_James/EUR 
N (sample size) = 44161 
hg_build = 19 
outdir = /oak/stanford/groups/zihuai/UCB data/Mar_13_James/ 
outfile = /oak/stanford/groups/zihuai/UCB data/Mar_13_James/UCB_GWAS_results seed = 2023 
verbose = true 
random_shuffle = true 
skip_shrinkage_check = false 

count_matchable_snps processed chr 1, cumulative SNPs = 46145 
count_matchable_snps processed chr 2, cumulative SNPs = 94450 
count_matchable_snps processed chr 3, cumulative SNPs = 135010 
count_matchable_snps processed chr 4, cumulative SNPs = 173294 
count_matchable_snps processed chr 5, cumulative SNPs = 209902 
count_matchable_snps processed chr 6, cumulative SNPs = 253171 
count_matchable_snps processed chr 7, cumulative SNPs = 285925 
count_matchable_snps processed chr 8, cumulative SNPs = 316982 
count_matchable_snps processed chr 9, cumulative SNPs = 343328 
count_matchable_snps processed chr 10, cumulative SNPs = 373313 
count_matchable_snps processed chr 11, cumulative SNPs = 403207 
count_matchable_snps processed chr 12, cumulative SNPs = 431788 
count_matchable_snps processed chr 13, cumulative SNPs = 452614 
count_matchable_snps processed chr 14, cumulative SNPs = 471730 
count_matchable_snps processed chr 15, cumulative SNPs = 489924 
count_matchable_snps processed chr 16, cumulative SNPs = 510232 
count_matchable_snps processed chr 17, cumulative SNPs = 528617 
count_matchable_snps processed chr 18, cumulative SNPs = 546334 
count_matchable_snps processed chr 19, cumulative SNPs = 561544 
count_matchable_snps processed chr 20, cumulative SNPs = 576703 
count_matchable_snps processed chr 21, cumulative SNPs = 585317 
count_matchable_snps processed chr 22, cumulative SNPs = 594385 
Error encountered while load FileIO.File{FileIO.DataFormat{:HDF5}, String}("EUR/chr1/LD_start100826405_end102041015.h5"). 

Fatal error: 
ERROR: HDF5 load error: neither load nor fileio_load is defined 
due to FileIO.SpecError(HDF5, :load) 
Will try next loader. 

Stacktrace: 
[1] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::FileIO.Formatted; options::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) 
@ FileIO /home/groups/sabatti/.julia/packages/FileIO/BE7iZ/src/loadsave.jl:209 
[2] action 
@ /home/groups/sabatti/.julia/packages/FileIO/BE7iZ/src/loadsave.jl:196 [inlined] 
[3] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::Symbol, ::String; options::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) 
@ FileIO /home/groups/sabatti/.julia/packages/FileIO/BE7iZ/src/loadsave.jl:185 
[4] action 
@ /home/groups/sabatti/.julia/packages/FileIO/BE7iZ/src/loadsave.jl:185 [inlined] 
[5] load(::String; options::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) 
@ FileIO /home/groups/sabatti/.julia/packages/FileIO/BE7iZ/src/loadsave.jl:113 
[6] load 
@ /home/groups/sabatti/.julia/packages/FileIO/BE7iZ/src/loadsave.jl:109 [inlined] 
[7] macro expansion 
@ /home/groups/sabatti/.julia/dev/GhostKnockoffGWAS/src/ghostbasil_parallel.jl:131 [inlined] 
[8] macro expansion 
@ ./timing.jl:393 [inlined] 
[9] ghostknockoffgwas(LD_files::String, z::Vector{Float64}, chr::Vector{Int64}, pos::Vector{Int64}, effect_allele::Vector{String}, non_effect_allele::Vector{String}, N::Int64, hg_build::Int64, outdir::String; outname::String, seed::Int64, target_chrs::Vector{Int64}, A_scaling_factor::Float64, kappa_lasso::Float64, LD_shrinkage::Bool, target_fdrs::Vector{Float64}, verbose::Bool, skip_shrinkage_check::Bool, random_shuffle::Bool) 
@ GhostKnockoffGWAS /home/groups/sabatti/.julia/dev/GhostKnockoffGWAS/src/ghostbasil_parallel.jl:130 
[10] macro expansion 
@ /home/groups/sabatti/.julia/dev/GhostKnockoffGWAS/src/app.jl:43 [inlined] 
[11] macro expansion 
@ ./timing.jl:393 [inlined] 
[12] julia_main() 
@ GhostKnockoffGWAS /home/groups/sabatti/.julia/dev/GhostKnockoffGWAS/src/app.jl:42 
[13] top-level scope 
@ none:1

Stacktrace: 
[1] handle_error(e::FileIO.LoaderError, q::Base.PkgId, bt::Vector{Union{Ptr{Nothing}, Base.InterpreterIP}}) 
@ FileIO /home/groups/sabatti/.julia/packages/FileIO/BE7iZ/src/error_handling.jl:61 
[2] handle_exceptions(exceptions::Vector{Tuple{Any, Union{Base.PkgId, Module}, Vector}}, action::String) 
@ FileIO /home/groups/sabatti/.julia/packages/FileIO/BE7iZ/src/error_handling.jl:56
[3] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::FileIO.Formatted; options::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) 
@ FileIO /home/groups/sabatti/.julia/packages/FileIO/BE7iZ/src/loadsave.jl:228 
[4] action 
@ /home/groups/sabatti/.julia/packages/FileIO/BE7iZ/src/loadsave.jl:196 [inlined] 
[5] action(::Symbol, ::Vector{Union{Base.PkgId, Module}}, ::Symbol, ::String; options::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) 
@ FileIO /home/groups/sabatti/.julia/packages/FileIO/BE7iZ/src/loadsave.jl:185 
[6] action 
@ /home/groups/sabatti/.julia/packages/FileIO/BE7iZ/src/loadsave.jl:185 [inlined] 
[7] load(::String; options::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) 
@ FileIO /home/groups/sabatti/.julia/packages/FileIO/BE7iZ/src/loadsave.jl:113 
[8] load 
@ /home/groups/sabatti/.julia/packages/FileIO/BE7iZ/src/loadsave.jl:109 [inlined] 
[9] macro expansion 
@ /home/groups/sabatti/.julia/dev/GhostKnockoffGWAS/src/ghostbasil_parallel.jl:131 [inlined] 
[10] macro expansion 
@ ./timing.jl:393 [inlined] 
[11] ghostknockoffgwas(LD_files::String, z::Vector{Float64}, chr::Vector{Int64}, pos::Vector{Int64}, effect_allele::Vector{String}, non_effect_allele::Vector{String}, N::Int64, hg_build::Int64, outdir::String; outname::String, seed::Int64, target_chrs::Vector{Int64}, A_scaling_factor::Float64, kappa_lasso::Float64, LD_shrinkage::Bool, target_fdrs::Vector{Float64}, verbose::Bool, skip_shrinkage_check::Bool, random_shuffle::Bool) 
@ GhostKnockoffGWAS /home/groups/sabatti/.julia/dev/GhostKnockoffGWAS/src/ghostbasil_parallel.jl:130 
[12] macro expansion 
@ /home/groups/sabatti/.julia/dev/GhostKnockoffGWAS/src/app.jl:43 [inlined] 
[13] macro expansion 
@ ./timing.jl:393 [inlined] 
[14] julia_main() 
@ GhostKnockoffGWAS /home/groups/sabatti/.julia/dev/GhostKnockoffGWAS/src/app.jl:42 
[15] top-level scope 
@ none:1

[biona001]

Slight update: on a fresh AWS instance, I tried running the current (v0.1.1) software with the current LD data on zenodo (.h5 format), and there were no issues. Prior to this, I also tried running the example data on UCLA's Hoffman2 cluster, and there were no issues there as well.

Thus, I'm fairly certain this problem is unique to Stanford's Sherlock cluster. To move forward, I think there are 2 possibilities:

Option1:

One option is to try removing HDF5.jl dependency, adding UnPack dependency, and update all LD files to using .jld2 extension. I'm not sure if this can resolve the issue on Sherlock, however.

Option2 (preferred):

Another possibility is that HDF5.jl library, while included in Project.toml, is not being compiled into the final app because it was not invoked anywhere within GhostKnockoffGWAS (in ghostbasil_parallel.jl, reading is accomplished via JLD2.read()). Thus, we can try explicitly calling HDF5.read() and adding a precompile statement in precompile.jl to ensure HDF5.jl is embedded within the app. Finally, we should also add UnPack.jl as a dependency.

[biona001]

6ef7be7 employs option 2, which allowed GhostKnockoffGWAS to run the example on @Jinzhou-Li 's Sherlock account. However the warning messages persisted, which is a known issue that should have been fixed. According to this post, all packages that depends on Requires.jl is likely not going to work well with PackageCompiler.jl.

Thus, I am now trying:

Option 2.5

This is the broadly similar to option 2 above, but I remove dependency on JLD2.jl, and support for HDF5.jl relies on my personal HDF5.jl fork (which has all the Requires.@require statements commented out). This is not super ideal but it's acceptable if this issue goes away.

75f2bbf confirmed that option 2.5 indeed works.