Fit Gamma-Poisson Generalized Linear Models Reliably.
The core design aims of gmlGamPoi
are:
DESeq2
or edgeR
You can install the release version of glmGamPoi from BioConductor:
if (!requireNamespace("BiocManager", quietly = TRUE))
install.packages("BiocManager")
BiocManager::install("glmGamPoi")
For the latest developments, see the GitHub repo.
To fit a single Gamma-Poisson GLM do:
# overdispersion = 1/size
counts <- rnbinom(n = 10, mu = 5, size = 1/0.7)
# size_factors = FALSE, because only a single GLM is fitted
fit <- glmGamPoi::glm_gp(counts, design = ~ 1, size_factors = FALSE)
fit
#> glmGamPoiFit object:
#> The data had 1 rows and 10 columns.
#> A model with 1 coefficient was fitted.
# Internally fit is just a list:
as.list(fit)
#> $Beta
#> Intercept
#> [1,] 1.504077
#>
#> $overdispersions
#> [1] 0.3792855
#>
#> $Mu
#> [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
#> [1,] 4.5 4.5 4.5 4.5 4.5 4.5 4.5 4.5 4.5 4.5
#>
#> $size_factors
#> [1] 1 1 1 1 1 1 1 1 1 1
#>
#> $model_matrix
#> Intercept
#> [1,] 1
#> [2,] 1
#> [3,] 1
#> [4,] 1
#> [5,] 1
#> [6,] 1
#> [7,] 1
#> [8,] 1
#> [9,] 1
#> [10,] 1
#> attr(,"assign")
#> [1] 0
#>
#> $design_formula
#> ~1
The glm_gp()
function returns a list with the results of the fit. Most importantly, it contains the estimates for the coefficients β and the overdispersion.
Fitting repeated Gamma-Poisson GLMs for each gene of a single cell dataset is just as easy:
I will first load an example dataset using the TENxPBMCData
package. The dataset has 33,000 genes and 4340 cells. It takes roughly 1.5 minutes to fit the Gamma-Poisson model on the full dataset. For demonstration purposes, I will subset the dataset to 300 genes, but keep the 4340 cells:
library(SummarizedExperiment)
library(DelayedMatrixStats)
# The full dataset with 33,000 genes and 4340 cells
# The first time this is run, it will download the data
pbmcs <- TENxPBMCData::TENxPBMCData("pbmc4k")
#> snapshotDate(): 2020-04-27
#> see ?TENxPBMCData and browseVignettes('TENxPBMCData') for documentation
#> loading from cache
# I want genes where at least some counts are non-zero
non_empty_rows <- which(rowSums2(assay(pbmcs)) > 0)
pbmcs_subset <- pbmcs[sample(non_empty_rows, 300), ]
pbmcs_subset
#> class: SingleCellExperiment
#> dim: 300 4340
#> metadata(0):
#> assays(1): counts
#> rownames(300): ENSG00000126457 ENSG00000109832 ... ENSG00000143819
#> ENSG00000188243
#> rowData names(3): ENSEMBL_ID Symbol_TENx Symbol
#> colnames: NULL
#> colData names(11): Sample Barcode ... Individual Date_published
#> reducedDimNames(0):
#> altExpNames(0):
I call glm_gp()
to fit one GLM model for each gene and force the calculation to happen in memory.
fit <- glmGamPoi::glm_gp(pbmcs_subset, on_disk = FALSE)
summary(fit)
#> glmGamPoiFit object:
#> The data had 300 rows and 4340 columns.
#> A model with 1 coefficient was fitted.
#> The design formula is: Y~1
#>
#> Beta:
#> Min 1st Qu. Median 3rd Qu. Max
#> Intercept -8.38 -6.43 -3.77 -2.46 1.01
#>
#> overdispersion:
#> Min 1st Qu. Median 3rd Qu. Max
#> 0 0 0.406 1.37 16102
#>
#> size_factors:
#> Min 1st Qu. Median 3rd Qu. Max
#> 0.402 0.969 1 1.05 1.75
#>
#> Mu:
#> Min 1st Qu. Median 3rd Qu. Max
#> 9.24e-05 0.00158 0.0229 0.087 4.81
I compare my method (in-memory and on-disk) with DESeq2 and edgeR. Both are classical methods for analyzing RNA-Seq datasets and have been around for almost 10 years. Note that both tools can do a lot more than just fitting the Gamma-Poisson model, so this benchmark only serves to give a general impression of the performance.
# Explicitly realize count matrix in memory
pbmcs_subset <- as.matrix(assay(pbmcs_subset))
model_matrix <- matrix(1, nrow = ncol(pbmcs_subset))
bench::mark(
glmGamPoi_in_memory = {
glmGamPoi::glm_gp(pbmcs_subset, design = model_matrix, on_disk = FALSE)
}, glmGamPoi_on_disk = {
glmGamPoi::glm_gp(pbmcs_subset, design = model_matrix, on_disk = TRUE)
}, DESeq2 = suppressMessages({
dds <- DESeq2::DESeqDataSetFromMatrix(pbmcs_subset,
colData = data.frame(name = seq_len(4340)),
design = ~ 1)
dds <- DESeq2::estimateSizeFactors(dds, "poscounts")
dds <- DESeq2::estimateDispersions(dds, quiet = TRUE)
dds <- DESeq2::nbinomWaldTest(dds, minmu = 1e-6)
}), edgeR = {
edgeR_data <- edgeR::DGEList(pbmcs_subset)
edgeR_data <- edgeR::calcNormFactors(edgeR_data)
edgeR_data <- edgeR::estimateDisp(edgeR_data, model_matrix)
edgeR_fit <- edgeR::glmFit(edgeR_data, design = model_matrix)
}, check = FALSE
)
#> # A tibble: 4 x 6
#> expression min median `itr/sec` mem_alloc `gc/sec`
#> <bch:expr> <bch:tm> <bch:tm> <dbl> <bch:byt> <dbl>
#> 1 glmGamPoi_in_memory 1.33s 1.33s 0.751 NA 0.751
#> 2 glmGamPoi_on_disk 5.38s 5.38s 0.186 NA 0.744
#> 3 DESeq2 30.38s 30.38s 0.0329 NA 0.230
#> 4 edgeR 12.92s 12.92s 0.0774 NA 0.542
On this dataset, glmGamPoi
is more than 6 times faster than edgeR
and more than 20 times faster than DESeq2
. glmGamPoi
does not use approximations to achieve this performance increase. The performance comes from an optimized algorithm for inferring the overdispersion for each gene. It is tuned for datasets typically encountered in single RNA-seq with many samples and many small counts, by avoiding duplicate calculations.
To demonstrate that the method is not sacrificing accuracy, I compare the parameters that each method estimates. I find that means and β coefficients are identical, but that the estimates of the overdispersion estimates from glmGamPoi
are more reliable:
# Results with my method
fit <- glmGamPoi::glm_gp(pbmcs_subset, design = model_matrix, on_disk = FALSE)
# DESeq2
dds <- DESeq2::DESeqDataSetFromMatrix(pbmcs_subset,
colData = data.frame(name = seq_len(4340)),
design = ~ 1)
dds <- DESeq2::estimateSizeFactors(dds, "poscounts")
dds <- DESeq2::estimateDispersions(dds, quiet = TRUE)
dds <- DESeq2::nbinomWaldTest(dds, minmu = 1e-6)
#edgeR
edgeR_data <- edgeR::DGEList(pbmcs_subset)
edgeR_data <- edgeR::calcNormFactors(edgeR_data)
edgeR_data <- edgeR::estimateDisp(edgeR_data, model_matrix)
edgeR_fit <- edgeR::glmFit(edgeR_data, design = model_matrix)
I am comparing the gene-wise estimates of the coefficients from all three methods. Points on the diagonal line are identical. The inferred Beta coefficients and gene means agree well between the methods, however the overdispersion differs quite a bit. DESeq2
has problems estimating most of the overdispersions and sets them to 1e-8
. edgeR
only approximates the overdispersions which explains the variation around the overdispersions calculated with glmGamPoi
.
The method scales linearly, with the number of rows and columns in the dataset. For example: fitting the full pbmc4k
dataset with subsampling on a modern MacBook Pro in-memory takes ~1 minute and on-disk a little over 4 minutes. Fitting the pbmc68k
(17x the size) takes ~73 minutes (17x the time) on-disk. Fitting that dataset in-memory is not possible because it is just too big: the maximum in-memory matrix size is 2^31-1 ≈ 2.1e9
is elements, the pbmc68k
dataset however has nearly 100 million elements more than that.
sessionInfo()
#> R version 4.0.0 (2020-04-24)
#> Platform: x86_64-pc-linux-gnu (64-bit)
#> Running under: Ubuntu 18.04.4 LTS
#>
#> Matrix products: default
#> BLAS: /home/biocbuild/bbs-3.11-bioc/R/lib/libRblas.so
#> LAPACK: /home/biocbuild/bbs-3.11-bioc/R/lib/libRlapack.so
#>
#> locale:
#> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
#> [3] LC_TIME=en_US.UTF-8 LC_COLLATE=C
#> [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
#> [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
#> [9] LC_ADDRESS=C LC_TELEPHONE=C
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
#>
#> attached base packages:
#> [1] parallel stats4 stats graphics grDevices utils datasets
#> [8] methods base
#>
#> other attached packages:
#> [1] TENxPBMCData_1.5.0 HDF5Array_1.16.0
#> [3] rhdf5_2.32.0 SingleCellExperiment_1.10.0
#> [5] DelayedMatrixStats_1.10.0 SummarizedExperiment_1.18.0
#> [7] DelayedArray_0.14.0 matrixStats_0.56.0
#> [9] Biobase_2.48.0 GenomicRanges_1.40.0
#> [11] GenomeInfoDb_1.24.0 IRanges_2.22.0
#> [13] S4Vectors_0.26.0 BiocGenerics_0.34.0
#> [15] BiocStyle_2.16.0
#>
#> loaded via a namespace (and not attached):
#> [1] bitops_1.0-6 bit64_0.9-7
#> [3] RColorBrewer_1.1-2 httr_1.4.1
#> [5] tools_4.0.0 utf8_1.1.4
#> [7] R6_2.4.1 DBI_1.1.0
#> [9] colorspace_1.4-1 tidyselect_1.0.0
#> [11] DESeq2_1.28.0 bit_1.1-15.2
#> [13] curl_4.3 compiler_4.0.0
#> [15] cli_2.0.2 bookdown_0.18
#> [17] scales_1.1.0 bench_1.1.1
#> [19] genefilter_1.70.0 rappdirs_0.3.1
#> [21] stringr_1.4.0 digest_0.6.25
#> [23] rmarkdown_2.1 XVector_0.28.0
#> [25] pkgconfig_2.0.3 htmltools_0.4.0
#> [27] limma_3.44.0 dbplyr_1.4.3
#> [29] fastmap_1.0.1 rlang_0.4.5
#> [31] RSQLite_2.2.0 shiny_1.4.0.2
#> [33] BiocParallel_1.22.0 dplyr_0.8.5
#> [35] RCurl_1.98-1.2 magrittr_1.5
#> [37] GenomeInfoDbData_1.2.3 Matrix_1.2-18
#> [39] fansi_0.4.1 Rcpp_1.0.4.6
#> [41] munsell_0.5.0 Rhdf5lib_1.10.0
#> [43] lifecycle_0.2.0 edgeR_3.30.0
#> [45] stringi_1.4.6 yaml_2.2.1
#> [47] zlibbioc_1.34.0 glmGamPoi_1.0.0
#> [49] BiocFileCache_1.12.0 AnnotationHub_2.20.0
#> [51] grid_4.0.0 blob_1.2.1
#> [53] promises_1.1.0 ExperimentHub_1.14.0
#> [55] crayon_1.3.4 lattice_0.20-41
#> [57] beachmat_2.4.0 splines_4.0.0
#> [59] annotate_1.66.0 magick_2.3
#> [61] locfit_1.5-9.4 knitr_1.28
#> [63] pillar_1.4.3 geneplotter_1.66.0
#> [65] XML_3.99-0.3 glue_1.4.0
#> [67] BiocVersion_3.11.1 evaluate_0.14
#> [69] BiocManager_1.30.10 vctrs_0.2.4
#> [71] httpuv_1.5.2 gtable_0.3.0
#> [73] purrr_0.3.4 assertthat_0.2.1
#> [75] ggplot2_3.3.0 xfun_0.13
#> [77] mime_0.9 xtable_1.8-4
#> [79] pracma_2.2.9 later_1.0.0
#> [81] survival_3.1-12 tibble_3.0.1
#> [83] AnnotationDbi_1.50.0 memoise_1.1.0
#> [85] ellipsis_0.3.0 interactiveDisplayBase_1.26.0