Contents

The rhdf5 provides limited support for read-only access to HDF5 files stored in Amazon S3 buckets. This is implemented via the HDF5 S3 Virtual File Driver and allows access to HDF5 files hosted in both public and private S3 buckets.

Currently only the functions h5ls(), h5dump() and h5read() are supported.

library(rhdf5)

1 Public S3 Buckets

To access a file in a public Amazon S3 bucket you provide the fileโ€™s URL to the file argument. You also need to set the argument s3 = TRUE, otherwise h5ls() will treat the URL as a path on the local disk fail.

public_S3_url <- "https://rhdf5-public.s3.eu-central-1.amazonaws.com/h5ex_t_array.h5"
h5ls(file = public_S3_url,
     s3 = TRUE)
##   group name       otype dclass dim
## 0     /  DS1 H5I_DATASET  ARRAY   4

The same arguments are also valid for using h5dump() to retrieve the contents of a file.

public_S3_url <- "https://rhdf5-public.s3.eu-central-1.amazonaws.com/h5ex_t_cmpd.h5"
h5dump(file = public_S3_url,
     s3 = TRUE)
## $DS1
##   Serial number          Location Temperature (F) Pressure (inHg)
## 1          1153 Exterior (static)           53.23           24.57
## 2          1184            Intake           55.12           22.95
## 3          1027   Intake manifold          103.55           31.23
## 4          1313  Exhaust manifold         1252.89           84.11

In addition to examining and reading whole files, we can also extract just a subset, without needed to read or download the entire file. In the example below we use h5ls() to examine a file in an S3 bucket and identify the name of a dataset within it (a1) and the number of dimensions for that dataset (3). We can then use h5read() along with the name and index arguments to read only a subset of the dataset into our R session.

public_S3_url <- 'https://rhdf5-public.s3.eu-central-1.amazonaws.com/rhdf5ex_t_float_3d.h5'
h5ls(file = public_S3_url, s3 = TRUE)
##   group name       otype dclass        dim
## 0     /   a1 H5I_DATASET  FLOAT 5 x 10 x 2
h5read(public_S3_url, 
       name = "a1", 
       index = list(1:2, 3, NULL),
       s3 = TRUE)
## , , 1
## 
##           [,1]
## [1,] 0.2444485
## [2,] 0.3873723
## 
## , , 2
## 
##           [,1]
## [1,] 0.7906603
## [2,] 0.3274960

2 Private S3 Buckets

To access files in a private Amazon S3 bucket you will need to provide three additional details: The AWS region where the files are hosted, your AWS access key ID, and your AWS secret access key. More information on how to obtain AWS access keys can be found under AWS Security Credentials.

These three values need to be stored in a list like below. Important note: for now they must be in this specific order.

## these are example credentials and will not work
s3_cred <- list(
    aws_region = "eu-central-1",
    access_key_id = "AKIAIOSFODNN7EXAMPLE",
    secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
)

Finally we pass this list to h5ls() via the s3credentials argument.

public_S3_url <- "https://rhdf5-private.s3.eu-central-1.amazonaws.com/h5ex_t_array.h5"
h5ls(file = public_S3_url,
     s3 = TRUE,
     s3credentials = s3_cred)

The s3credentials arguments is used in exactly the same way for h5dump() and h5read().

3 Session Info

sessionInfo()
## R version 4.1.2 (2021-11-01)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 20.04.4 LTS
## 
## Matrix products: default
## BLAS:   /home/biocbuild/bbs-3.14-bioc/R/lib/libRblas.so
## LAPACK: /home/biocbuild/bbs-3.14-bioc/R/lib/libRlapack.so
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
##  [3] LC_TIME=en_GB              LC_COLLATE=C              
##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  methods   base     
## 
## other attached packages:
## [1] ggplot2_3.3.5    dplyr_1.0.8      rhdf5_2.38.1     BiocStyle_2.22.0
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_1.0.8           highr_0.9            bslib_0.3.1         
##  [4] compiler_4.1.2       pillar_1.7.0         BiocManager_1.30.16 
##  [7] jquerylib_0.1.4      rhdf5filters_1.6.0   tools_4.1.2         
## [10] bit_4.0.4            digest_0.6.29        gtable_0.3.0        
## [13] jsonlite_1.8.0       evaluate_0.15        lifecycle_1.0.1     
## [16] tibble_3.1.6         pkgconfig_2.0.3      rlang_1.0.2         
## [19] cli_3.2.0            DBI_1.1.2            magick_2.7.3        
## [22] microbenchmark_1.4.9 yaml_2.3.5           xfun_0.30           
## [25] fastmap_1.1.0        withr_2.5.0          stringr_1.4.0       
## [28] knitr_1.37           generics_0.1.2       vctrs_0.3.8         
## [31] sass_0.4.0           bit64_4.0.5          grid_4.1.2          
## [34] tidyselect_1.1.2     glue_1.6.2           R6_2.5.1            
## [37] fansi_1.0.2          rmarkdown_2.12       bookdown_0.24       
## [40] farver_2.1.0         Rhdf5lib_1.16.0      purrr_0.3.4         
## [43] magrittr_2.0.2       codetools_0.2-18     scales_1.1.1        
## [46] htmltools_0.5.2      ellipsis_0.3.2       assertthat_0.2.1    
## [49] colorspace_2.0-3     labeling_0.4.2       utf8_1.2.2          
## [52] stringi_1.7.6        munsell_0.5.0        crayon_1.5.0