Introduction
You will probably be familiar with multiple testing procedures that take a set of p-values and then calculate adjusted p-values. Given a significance level \(\alpha\), one can then declare the rejected hypotheses. In R this is most commonly done with the p.adjust function in the stats package, and a popular choice is controlling the false discovery rate (FDR) with the method of (Benjamini and Hochberg 1995), provided by the choice method="BH" in p.adjust. A characteristic feature of this and other methods –responsible both for their versatility and limitations– is that they do not use anything else beyond the p-values: no other potential information that might set the tests apart, such as different signal quality, power, prior probability.
IHW (Independent Hypothesis Weighting) is also a multiple testing procedure, but in addition to the p-values it allows you to specify a covariate for each test. The covariate should be informative of the power or prior probability of each individual test, but is chosen such that the p-values for those hypotheses that are truly null do not depend on the covariate (Ignatiadis et al. 2016). Therefore the input of IHW is the following:
- a vector of p-values (of length \(m\)),
- a matching vector of covariates,
- the significance level \(\alpha \in (0,1)\) at which the FDR should be controlled.
IHW then calculates weights for each p-value (non-negative numbers \(w_i \geq 0\) such that they average to 1, \(\sum_{i=1}^m w_i = m\)). IHW also returns a vector of adjusted p-values by applying the procedure of Benjamini Hochberg (BH) to the weighted p-values \(P^\text{weighted}_i = \frac{P_i}{w_i}\).
The weights allow different prioritization of the individual hypotheses, based on their covariate. This means that the ranking of hypotheses with p-value weighting is in general different than without. Two hypotheses with the same p-value can have different weighted p-values: the one with the higher weight will then have a smaller value of \(P^\text{weighted}_i\), and consequently it can even happen that one but not the other gets rejected by the subsequent BH procedure.
As an example, let’s see how to use the IHW package in analysing for RNA-Seq differential gene expression. and then also look at some other examples where the method is applicable.
An example: RNA-Seq differential expression
We analyze the airway RNA-Seq dataset using DESeq2 (Love, Huber, and Anders 2014).
library("DESeq2")
library("dplyr")
data("airway", package = "airway")
dds <- DESeqDataSet(se = airway, design = ~ cell + dex) %>% DESeq
deRes <- as.data.frame(results(dds))
The output is a dataframe with the following columns, and one row for each tested hypothesis (i.e., for each gene):
colnames(deRes)
## [1] "baseMean" "log2FoldChange" "lfcSE" "stat"
## [5] "pvalue" "padj"
In particular, we have p-values and baseMean (i.e., the mean of normalized counts) for each gene. As argued in the DESeq2 paper, these two statistics are approximately independent under the null hypothesis. Thus we have all the ingredient necessary for a IHW analysis (p-values and covariates), which we will apply at a significance level 0.1.
FDR control
First load IHW:
library("IHW")
ihwRes <- ihw(pvalue ~ baseMean, data = deRes, alpha = 0.1)
This returns an object of the class ihwResult. We can get, e.g., the total number of rejections.
rejections(ihwRes)
## [1] 4873
And we can also extract the adjusted p-values:
head(adj_pvalues(ihwRes))
## [1] 0.001130504 NA 0.160424028 0.880323612 1.000000000 1.000000000
sum(adj_pvalues(ihwRes) <= 0.1, na.rm = TRUE) == rejections(ihwRes)
## [1] TRUE
We can compare this to the result of applying the method of Benjamini and Hochberg to the p-values only:
padjBH <- p.adjust(deRes$pvalue, method = "BH")
sum(padjBH <= 0.1, na.rm = TRUE)
## [1] 4081
IHW produced quite a bit more rejections than that. How did we get this power? Essentially it was possible by assigning appropriate weights to each hypothesis. We can retrieve the weights as follows:
head(weights(ihwRes))
## [1] 2.116238 NA 2.429560 2.292776 1.502119 0.000000
Internally, what happened was the following: We split the hypotheses into \(n\) different strata (here \(n=22\)) based on increasing value of baseMean and we also randomly split them into \(k\) folds (here \(k=5\)). Then, for each combination of fold and stratum, we learned the weights. The discretization into strata facilitates the estimation of the distribution function conditionally on the covariate and the optimization of the weights. The division into random folds helps us to avoid overfitting the data, something which could otherwise result in loss of control of the FDR (Ignatiadis et al. 2016).
The values of \(n\) and \(k\) can be accessed through
c(nbins(ihwRes), nfolds(ihwRes))
## [1] 22 5
In particular, each hypothesis test gets assigned a weight depending on the combination of its assigned fold and stratum.
We can also see this internal representation of the weights as a (\(n\) X \(k\)) matrix:
weights(ihwRes, levels_only = TRUE)
## [,1] [,2] [,3] [,4] [,5]
## [1,] 0.0000000 0.0000000 0.0000000 0.000000 0.0000000
## [2,] 0.0000000 0.0000000 0.0000000 0.000000 0.0000000
## [3,] 0.0000000 0.0000000 0.0000000 0.000000 0.0000000
## [4,] 0.0000000 0.0000000 0.0000000 0.000000 0.0000000
## [5,] 0.0000000 0.0000000 0.0000000 0.000000 0.0000000
## [6,] 0.0000000 0.0000000 0.0000000 0.000000 0.0000000
## [7,] 0.0000000 0.0000000 0.0000000 0.000000 0.0000000
## [8,] 0.0000000 0.0000000 0.0000000 0.000000 0.0000000
## [9,] 0.0000000 0.0000000 0.0000000 0.000000 0.0000000
## [10,] 0.0000000 0.0000000 0.0000000 0.000000 0.0000000
## [11,] 0.3012165 0.1929644 0.4853261 0.490779 0.2956142
## [12,] 0.3012165 0.5013695 0.4676253 0.490779 0.4961546
## [13,] 0.6227621 0.7859151 1.2021934 1.317340 0.7777406
## [14,] 1.2640912 1.5021192 1.2021934 1.383575 1.7530250
## [15,] 2.1978658 2.4505372 2.2772911 2.782278 2.3312196
## [16,] 2.1562363 2.4505372 2.2927765 2.782278 2.3312196
## [17,] 3.4526740 2.4505372 2.2927765 2.438397 2.3312196
## [18,] 2.9427856 2.5112086 2.2927765 2.438397 2.4295601
## [19,] 2.4829555 2.5112086 2.1908606 2.116238 2.4295601
## [20,] 2.4829555 2.5112086 2.3106615 2.511352 2.4295601
## [21,] 1.6923820 2.2724171 2.1105565 1.554436 2.1091202
## [22,] 1.8961571 2.2724171 2.1105565 2.254088 2.1091202
Diagnostic plots
Estimated weights
plot(ihwRes)
We see that the general trend is driven by the covariate (stratum) and is the same across the different folds. As expected, the weight functions calculated on different random subsets of the data behave similarly. For the data at hand, genes with very low baseMean count get assigned a weight of 0, while genes with high baseMean count get prioritized.
Decision boundary
plot(ihwRes, what = "decisionboundary")
The plot shows the implied decision boundaries for the unweighted p-values, as a function of the covariate.
Raw versus adjusted p-values
library("ggplot2")
gg <- ggplot(as.data.frame(ihwRes), aes(x = pvalue, y = adj_pvalue, col = group)) +
geom_point(size = 0.25) + scale_colour_hue(l = 70, c = 150, drop = FALSE)
gg
gg %+% subset(as.data.frame(ihwRes), adj_pvalue <= 0.2)
The ihwResult object ihwRes can be converted to a dataframe that contains the following columns:
ihwResDf <- as.data.frame(ihwRes)
colnames(ihwResDf)
## [1] "pvalue" "adj_pvalue" "weight" "weighted_pvalue"
## [5] "group" "covariate" "fold"
FWER control with IHW
The standard IHW method presented above controls the FDR by using a weighted Benjamini-Hochberg procedure with data-driven weights. The same principle can be applied for FWER control by using a weighted Bonferroni procedure. Everything works exactly as above, just use the argument adjustment_type. For example:
ihwBonferroni <- ihw(pvalue ~ baseMean, data = deRes, alpha = 0.1, adjustment_type = "bonferroni")
Advanced usage: Working with incomplete p-value lists
So far, we have assumed that a complete list of p-values is available, i.e. for each test that was performed we know the resulting p-value. However, this information is not always available or practical.
- Some software tools simply do not calculate or report all p-values. For example, as noted by (Ochoa et al. 2015), HMMER only returns the lowest p-values. Similarly, MatrixEQTL (Shabalin 2012) by default only returns p-values below a pre-specified threshold, for example all p-values below \(10^{-5}\). In the case of HMMER, this is done because higher p-values are not reliable, while for MatrixEQTL it reduces storage requirements.
- Generally for very large multiple testing problems, it can be practical to only explicitly compute and store small enough p-values, and for the larger ones only record that they they were large, but not the precise value. This can greatly reduce the needed computing resources (in particular, working memory).
Since rejections take place at the low end of the p-value distribution, we do not lose a lot of information by discarding the exact values of the higher p-values, as long as we keep track of how many of them there are. Thus, the above situations can be easily handled.
Before proceeding with the walkthrough for handling such cases with IHW, we quickly review how this is handled by p.adjust. We first simulate some data, where the power under the alternative depends on a covariate X. p-values are calculated by a simple one-sided z-test.
sim <- tibble(
X = runif(100000, min = 0, max = 2.5), # covariate
H = rbinom(length(X), size = 1, prob = 0.1), # hypothesis true or false
Z = rnorm(length(X), mean = H * X), # Z-score
p = 1 - pnorm(Z)) # pvalue
We can apply the Benjamini-Hochberg procedure to these p-values:
sim <- mutate(sim, padj = p.adjust(p, method="BH"))
sum(sim$padj <= 0.1)
## [1] 623
Now assume we only have access to the p-values \(\leq 0.1\):
reporting_threshold <- 0.1
sim <- mutate(sim, reported = (p <= reporting_threshold))
simSub <- filter(sim, reported)
Then we can still use p.adjust on simSub, as long as we inform it of how many hypotheses were originally. We specify this by setting the n function argument.
simSub = mutate(simSub, padj2 = p.adjust(p, method = "BH", n = nrow(sim)))
ggplot(simSub, aes(x = padj, y = padj2)) + geom_point(cex = 0.2)
The plot shows the BH-adjusted p-values computed from all p-values and then subset (x-axis) versus the BH-adjusted p-values computed from the subset of reported p-values only, using the n argument of p.adjust (y-axis).
stopifnot(with(simSub, max(abs(padj - padj2)) <= 0.001))
We see that the results agree. Now, the same approach can be used with IHW, but is slighly more complicated. In particular, we need to provide information about how many hypotheses were tested at each given value of the covariate. This means that there are two modifications to the standard IHW workflow:
- If a numeric covariate is provided, IHW internally discretizes it and in this way bins the hypotheses into groups (strata). For the advanced functionality, this discretization has to be done beforehand by the user. In other words, the covariate provided by the user has to be a factor. For this, the convenience function
groups_by_filter is provided, which returns a factor that stratifies a numeric covariate into a given number of groups with approximately the same number of hypotheses in each of the groups. This is a very simple function, largely equivalent to cut(., quantile(., probs = seq(0, 1, length.out = nbins)).
- For the algorithm to work correctly, it is necessary to know the total number of hypotheses in each of the bins. However, if only a subset of p-values are reported, IHW obviously cannot know the number of hypotheses per bin automatically. Therefore, the user has to specify this information manually via the
m_groups argument. (When there is only 1 bin, IHW reduces to BH and m_groups would be equivalent to the n argument of p.adjust.)
For example, if the whole grouping factor is available (e.g., when it was generated by using groups_by_filter on the full vector of covariates), then one can apply the table function to it to calculate the number of hypotheses per bin. This is then used as an input for the m_groups argument. More elaborate strategies are needed in more complicated cases, e.g., when the full vector of covariates does not fit into memory.
nbins <- 20
sim <- mutate(sim, group = groups_by_filter(X, nbins))
m_groups <- table(sim$group)
Now we can apply IHW to the data subset that results from only keeping low p-values (reported is TRUE), with the manually specified m_groups.
ihwS <- ihw(p ~ group, alpha = 0.1, data = filter(sim, reported), m_groups = m_groups)
ihwS
## ihwResult object with 13993 hypothesis tests
## Nominal FDR control level: 0.1
## Split into 20 bins, based on an ordinal covariate
For comparison, let’s also call IHW on the full dataset (in a real application, this would normally not be available).
ihwF <- ihw(p ~ group, alpha = 0.1, data = sim)
Now we can compare ihwS and ihwF. This is a little bit more subtle than above for the BH method, because
unlike BH, IHW also uses the information available in higher p-values to determine the weights (i.e. the Grenander estimator of the p-value distribution will be slighly different if you only use a subset of small p-values), and more importantly
because IHW involves “random” data splitting, which pans out differently when the set of hypotheses is different.
Modulo these two aspects, the results should be approximately the same, in terms of the weights curves and the final number of rejections.
gridExtra::grid.arrange(
plot(ihwS),
plot(ihwF),
ncol = 2)
c(rejections(ihwS),
rejections(ihwF))
## [1] 1059 1067
We can also plot the weighted, BH-adjusted p-values against each other:
qplot(adj_pvalues(ihwF)[sim$reported], adj_pvalues(ihwS), cex = I(0.2),
xlim = c(0, 0.1), ylim = c(0, 0.1)) + coord_fixed()
