Trouble installing dev branch

[ollieeknight]

Heya,

Love the package and the amount of support you put into it - was hoping you could help me troubleshoot why I'm having issues installing a dev branch from this issue...

conda create -y -n cellbender python=3.7
conda activate cellbender
pip install git+https://github.com/broadinstitute/CellBender@sf_dev_0.3.0_postreg_posterior_format_h5

after a successful install, when I call the command I get the following error

(cellbender) oknight@raven01:~/bin/miniconda3/envs> cellbender
Traceback (most recent call last):
  File "/u/oknight/bin/miniconda3/envs/cellbender/bin/cellbender", line 8, in <module>
    sys.exit(main())
  File "/u/oknight/bin/miniconda3/envs/cellbender/lib/python3.7/site-packages/cellbender/base_cli.py", line 103, in main
    parser = get_populated_argparser()
  File "/u/oknight/bin/miniconda3/envs/cellbender/lib/python3.7/site-packages/cellbender/base_cli.py", line 79, in get_populated_argparser
    parser.add_argument('-v', '--version', action='version', version=get_version())
  File "/u/oknight/bin/miniconda3/envs/cellbender/lib/python3.7/site-packages/cellbender/base_cli.py", line 20, in get_version
    with open(os.path.join(os.path.dirname(__file__), '..', 'VERSION')) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/u/oknight/bin/miniconda3/envs/cellbender/lib/python3.7/site-packages/cellbender/../VERSION'

Am I making a rookie mistake somewhere?

[sjfleming]

Hi @ollieeknight , I don't think you are making a mistake, I think there's something wrong...

I have tried to implement a fix for this. (I always pip install in editable -e mode, and the current code works when doing that, but you're right... when you try to pip install straight from GitHub, I see the same error.)

The branch sf_dev_0.3.0_postreg now has all the changes from sf_dev_0.3.0_postreg_posterior_format_h5, plus a change that I think will address this issue.

If I try

pip install git+https://github.com/broadinstitute/CellBender@sf_dev_0.3.0_postreg

in a clean conda env, I now see

cellbender -v

gives 0.3.0

[ollieeknight]

nice! this works perfectly now. thanks so much!

[ollieeknight]

just to update you, if you weren't aware of this bug and wanted to know...

After running one sample, with otherwise successful outputs:

cp: cannot stat '/u/oknight/bin/miniconda3/envs/cellbender/lib/python3.7/site-packages/cellbender/remove_background/report.ipynb': No such file or directory
[NbConvertApp] WARNING | pattern 'tmp.report.ipynb' matched no files
rm: cannot remove 'tmp.report.ipynb': No such file or directory
[NbConvertApp] WARNING | pattern 'tmp.report.nbconvert.ipynb' matched no files
mv: cannot stat 'tmp.report.nbconvert.html': No such file or directory
rm: cannot remove 'tmp.report.nbconvert.ipynb': No such file or directory
cellbender:remove-background: Unable to create report.
cellbender:remove-background: Traceback (most recent call last):
  File "/u/oknight/bin/miniconda3/envs/cellbender/lib/python3.7/site-packages/cellbender/remove_background/run.py", line 414, in compute_output_denoised_counts_re>    output=html_report_file,
  File "/u/oknight/bin/miniconda3/envs/cellbender/lib/python3.7/site-packages/cellbender/remove_background/report.py", line 79, in run_notebook_make_html
    title=('CellBender: ' + os.path.basename(output).replace('_report.html', '')),
  File "/u/oknight/bin/miniconda3/envs/cellbender/lib/python3.7/site-packages/cellbender/remove_background/report.py", line 56, in _postprocess_html
    with open(file, mode='r') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'INT1_DOGMA/outs/cellbender_output_report.html'

cellbender:remove-background: Completed remove-background.
cellbender:remove-background: 2023-04-28 18:37:07

[sjfleming]

Oh shoot, it looks like, without the editable install, the Jupyter notebook I use as the report template is not being included in the package...

[sjfleming]

@ollieeknight can you try it one more time? :)

[ollieeknight]

yep, everything's run without error now. nice one!

[ollieeknight]

Further to this, I wanted to write my experience using the --exclude-feature-types Peaks:

I read the filtered .h5 file into R, however... it includes the Peaks features when importing. This is even though I don't think they're being used for filtering features: cellbender_output_report.pdf

> alldata.rna$INT2a
An object of class Seurat 
162756 features across 21586 samples within 1 assay 
Active assay: RNA (162756 features, 0 variable features)

and the overlap is here:

> rownames(alldata.rna$INT2a)[36370:36400]
 [1] "RBMY1D"             "RBMY1E"             "PRY2"               "AC007359.1"        
 [5] "RBMY1F"             "TTTY5"              "TTTY6"              "DAZ1"              
 [9] "PRYP3"              "TTTY3B"             "CDY1B"              "DAZ3"              
[13] "BPY2C"              "TTTY4C"             "TTTY17C"            "LINC00266-4P"      
[17] "chr1:9746-10684"    "chr1:15785-16616"   "chr1:17079-17939"   "chr1:28903-29782"  
[21] "chr1:181041-181769" "chr1:183853-184632" "chr1:186539-187280" "chr1:191137-192005"
[25] "chr1:267585-268468" "chr1:585744-586642" "chr1:605081-605980" "chr1:778319-779226"
[29] "chr1:812456-813400" "chr1:814858-815723" "chr1:816864-817735"

which matches exactly with what the cellbender log has

cellbender:remove-background: Loading data from INT1_DOGMA/outs/raw_feature_bc_matrix.h5
cellbender:remove-background: CellRanger v3 format
cellbender:remove-background: Features in dataset: 36385 Gene Expression, 87500 Peaks
cellbender:remove-background: Excluding Peaks features (output will equal input).
cellbender:remove-background:     - This results in the exclusion of 87500 features.

[sjfleming]

Hi @ollieeknight , yes, you are actually seeing the intended behavior. In some sense, I guess it's strange to see Peak information in the output file from CellBender when you tell it to "exclude peaks", but on the other hand, what it's really doing is excluding them from the analysis, as you said. CellBender acts like the Peak features are not present during the analysis, but afterward, it appends the Peak features, unaltered, to the output data (that's the (output will equal input) comment in the log).

The behavior recapitulates what you would get if you manually divided your file in two: RNA and ATAC, and then fed the RNA part through CellBender, and then combined that RNA CellBender output with the raw ATAC data.

You should see that the RNA data has been modified in the CellBender output, but that the ATAC data is untouched.

[sjfleming]

Closed by #238