Discrepancy with genes count in toy dataset

[rvernica]

When running the Scanorama algorithm on the toy dataset used in the README the output looks like this:

Loaded data/293t_jurkat/293t with 32738 genes and 2885 cells
...
After data/293t_jurkat/293t: 32639 genes
...
Found 32639 genes among all datasets

This leads the user to believe that there are 32,738 genes in the 293t dataset. On closer examination, the dataset contains the names of 32,738 names, but the matrix only contains data for 18,760 genes. So, the number of genes used in the algorithm is significantly smaller that the number reported in the output:

>>> X, genes = load_mtx('data/293t_jurkat/293t')

>>> X
<2885x32738 sparse matrix of type '<class 'numpy.float64'>'
	with 9854100 stored elements in Compressed Sparse Row format>
# The sparse matrix has 32,738 columns
>>> len(genes)
32738
# The gene list has 32,738 names

>>> len(set(X.nonzero()[1]))
18760
# But there is data for only 18,760 genes in the matrix
>>> X.nonzero()[1]
array([   40,    57,    67, ..., 32706, 32707, 32708], dtype=int32)
# Looking at the columns that have data, confirms that there are less than 32,738 genes in the data

[brianhie]

Hi @rvernica,

From the look of the log file, the matrix does have dimension 32738 along the gene axis. The discrepancy is just explained by many genes having zero entries across all cells (e.g., in the corresponding gene assembly, no reads mapped to those genes). I think that this logging behavior doesn't need to be changed -- Scanorama will simply report the size of the gene dimension, and not do anything "smarter" like counting genes with zero reads across all cells. In any case, these genes can easily be filtered out before data analysis.