ReUseData provides functionalities to construct workflow-based data
recipes for fully tracked and reproducible data processing. Evaluation
of data recipes generates curated data resources in their generic
formats (e.g., VCF, bed), as well as a YAML manifest file recording
the recipe parameters, data annotations, and data file paths for
subsequent reuse. The datasets are locally cached using a database
infrastructure, where updating and searching of specific data is made easy.
The data reusability is assured through cloud hosting and enhanced interoperability with downstream software tools or analysis workflows. The workflow strategy enables cross platform reproducibility of curated data resources.
if (!requireNamespace("BiocManager", quietly = TRUE))
install.packages("BiocManager")
BiocManager::install("ReUseData")Use the development version:
BiocManager::install("ReUseData", version = "devel")suppressPackageStartupMessages(library(Rcwl))
library(ReUseData)ReUseData core functions for data managementHere we introduce the core functions of ReUseData for data
management and reuse: getData for reproducible data generation,
dataUpdate for syncing and updating data cache, and dataSearch for
multi-keywords searching of dataset of interest.
First, we can construct data recipes by transforming shell or other ad
hoc data preprocessing scripts into workflow-based data recipes. Some
prebuilt data recipes for public data resources (e.g., downloading,
unzipping and indexing) are available for direct use through
recipeSearch and recipeLoad functions. Then we will assign values
to the input parameters and evaluate the recipe to generate data of
interest.
## set cache in tempdir for test
Sys.setenv(cachePath = file.path(tempdir(), "cache"))
recipeUpdate()
#> Updating recipes...
#> bowtie2_index.R added
#> echo_out.R added
#> ensembl_liftover.R added
#> gcp_broad_gatk_hg19.R added
#> gcp_broad_gatk_hg38.R added
#> gcp_gatk_mutect2_b37.R added
#> gcp_gatk_mutect2_hg38.R added
#> gencode_annotation.R added
#> gencode_genome_grch38.R added
#> gencode_transcripts.R added
#> hisat2_index.R added
#> reference_genome.R added
#> salmon_index.R added
#> STAR_index.R added
#> ucsc_database.R added
#>
#> recipeHub with 15 records
#> cache path: /tmp/RtmpDTBgrk/cache/ReUseDataRecipe
#> # recipeSearch() to query specific recipes using multipe keywords
#> # recipeUpdate() to update the local recipe cache
#>
#> name
#> BFC1 | bowtie2_index
#> BFC2 | echo_out
#> BFC3 | ensembl_liftover
#> BFC4 | gcp_broad_gatk_hg19
#> BFC5 | gcp_broad_gatk_hg38
#> ... ...
#> BFC11 | hisat2_index
#> BFC12 | reference_genome
#> BFC13 | salmon_index
#> BFC14 | STAR_index
#> BFC15 | ucsc_database
recipeSearch("echo")
#> recipeHub with 1 records
#> cache path: /tmp/RtmpDTBgrk/cache/ReUseDataRecipe
#> # recipeSearch() to query specific recipes using multipe keywords
#> # recipeUpdate() to update the local recipe cache
#>
#> name
#> BFC2 | echo_out
echo_out <- recipeLoad("echo_out")
#> Note: you need to assign a name for the recipe: rcpName <- recipeLoad('xx')
#> Data recipe loaded!
#> Use inputs() to check required input parameters before evaluation.
#> Check here: https://rcwl.org/dataRecipes/echo_out.html
#> for user instructions (e.g., eligible input values, data source, etc.)
inputs(echo_out)
#> inputs:
#> input (input) (string):
#> outfile (outfile) (string):Users can then assign values to the input parameters, and evaluate the
recipe (getData) to generate data of interest. Users need to specify
an output directory for all files (desired data file, intermediate
files that are internally generated as workflow scripts or annotation
files). Detailed notes for the data is encouraged which will be used
for keywords matching for later data search.
We can install cwltool first to make sure a cwl-runner is available.
invisible(Rcwl::install_cwltool())echo_out$input <- "Hello World!"
echo_out$outfile <- "outfile"
outdir <- file.path(tempdir(), "SharedData")
res <- getData(echo_out,
outdir = outdir,
notes = c("echo", "hello", "world", "txt"))
#> }[1;30mINFO[0m Final process status is successThe file path to newly generated dataset can be easily retrieved. It
can also be retrieved using dataSearch() functions with multiple
keywords. Before that, dataUpdate() needs to be done.
res$output
#> [1] "/tmp/RtmpDTBgrk/SharedData/outfile.txt"There are some automatically generated files to help track the data
recipe evaluation, including *.sh to record the original shell
script, *.cwl file as the official workflow script which was
internally submitted for data recipe evaluation, *.yml file as part
of CWL workflow evaluation, which also record data annotations, and
*.md5 checksum file to check/verify the integrity of generated data
file.
list.files(outdir, pattern = "echo")
#> [1] "echo_out_Hello_World!_outfile.cwl" "echo_out_Hello_World!_outfile.md5"
#> [3] "echo_out_Hello_World!_outfile.sh" "echo_out_Hello_World!_outfile.yml"The *.yml file contains information about recipe input parameters,
the file path to output file, the notes for the dataset, and
auto-added date for data generation time. A later data search using
dataSearch() will refer to this file for keywords match.
readLines(res$yml)
#> [1] "input: Hello World!"
#> [2] "outfile: outfile"
#> [3] "# output: /tmp/RtmpDTBgrk/SharedData/outfile.txt"
#> [4] "# notes: echo hello world txt"
#> [5] "# date: 2024-05-01 01:35:51.665942"dataUpdate() creates (if first time use), syncs and updates the local
cache for curated datasets. It finds and reads all the .yml files
recursively in the provided data folder, creates a cache record for
each dataset that is associated (including newly generated ones with
getData()), and updates the local cache for later data searching and
reuse.
IMPORTANT: It is recommended that users create a specified folder for
data archival (e.g., file/path/to/SharedData) that other group
members have access to, and use sub-folders for different kinds of
datasets (e.g., those generated from same recipe).
(dh <- dataUpdate(dir = outdir))
#>
#> Updating data record...
#> outfile.txt added
#> dataHub with 1 records
#> cache path: /tmp/RtmpDTBgrk/cache/ReUseData
#> # dataUpdate() to update the local data cache
#> # dataSearch() to query a specific dataset
#> # Additional information can be retrieved using:
#> # dataNames(), dataParams(), dataNotes(), dataPaths(), dataTag() or mcols()
#>
#> name Path
#> BFC1 | outfile.txt /tmp/RtmpDTBgrk/SharedData/outfile.txtdataUpdate and dataSearch return a dataHub object with a list of
all available or matching datasets.
One can subset the list with [ and use getter functions to retrieve
the annotation information about the data, e.g., data names,
parameters values to the recipe, notes, tags, and the corresponding
yaml file.
dh[1]
#> dataHub with 1 records
#> cache path: /tmp/RtmpDTBgrk/cache/ReUseData
#> # dataUpdate() to update the local data cache
#> # dataSearch() to query a specific dataset
#> # Additional information can be retrieved using:
#> # dataNames(), dataParams(), dataNotes(), dataPaths(), dataTag() or mcols()
#>
#> name Path
#> BFC1 | outfile.txt /tmp/RtmpDTBgrk/SharedData/outfile.txt
## dh["BFC1"]
dh[dataNames(dh) == "outfile.txt"]
#> dataHub with 1 records
#> cache path: /tmp/RtmpDTBgrk/cache/ReUseData
#> # dataUpdate() to update the local data cache
#> # dataSearch() to query a specific dataset
#> # Additional information can be retrieved using:
#> # dataNames(), dataParams(), dataNotes(), dataPaths(), dataTag() or mcols()
#>
#> name Path
#> BFC1 | outfile.txt /tmp/RtmpDTBgrk/SharedData/outfile.txt
dataNames(dh)
#> [1] "outfile.txt"
dataParams(dh)
#> [1] "input: Hello World!; outfile: outfile"
dataNotes(dh)
#> [1] "echo hello world txt"
dataTags(dh)
#> [1] ""
dataYml(dh)
#> [1] "/tmp/RtmpDTBgrk/SharedData/echo_out_Hello_World!_outfile.yml"ReUseData, as the name suggests, commits to promoting the data reuse.
Data can be prepared in standard input formats (toList), e.g.,
YAML and JSON, to be easily integrated in workflow methods that are
locally or cloud-hosted.
(dh1 <- dataSearch(c("echo", "hello", "world")))
#> dataHub with 1 records
#> cache path: /tmp/RtmpDTBgrk/cache/ReUseData
#> # dataUpdate() to update the local data cache
#> # dataSearch() to query a specific dataset
#> # Additional information can be retrieved using:
#> # dataNames(), dataParams(), dataNotes(), dataPaths(), dataTag() or mcols()
#>
#> name Path
#> BFC1 | outfile.txt /tmp/RtmpDTBgrk/SharedData/outfile.txt
toList(dh1, listNames = c("input_file"))
#> $input_file
#> [1] "/tmp/RtmpDTBgrk/SharedData/outfile.txt"
toList(dh1, format = "yaml", listNames = c("input_file"))
#> [1] "input_file: /tmp/RtmpDTBgrk/SharedData/outfile.txt"
toList(dh1, format = "json", file = file.path(tempdir(), "data.json"))
#> File is saved as: "/tmp/RtmpDTBgrk/data.json"
#> {
#> "outfile.txt": "/tmp/RtmpDTBgrk/SharedData/outfile.txt"
#> }Data can also be aggregated from different resources by tagging with specific software tools.
dataSearch()
#> dataHub with 1 records
#> cache path: /tmp/RtmpDTBgrk/cache/ReUseData
#> # dataUpdate() to update the local data cache
#> # dataSearch() to query a specific dataset
#> # Additional information can be retrieved using:
#> # dataNames(), dataParams(), dataNotes(), dataPaths(), dataTag() or mcols()
#>
#> name Path
#> BFC1 | outfile.txt /tmp/RtmpDTBgrk/SharedData/outfile.txt
dataTags(dh[1]) <- "#gatk"
dataSearch("#gatk")
#> dataHub with 1 records
#> cache path: /tmp/RtmpDTBgrk/cache/ReUseData
#> # dataUpdate() to update the local data cache
#> # dataSearch() to query a specific dataset
#> # Additional information can be retrieved using:
#> # dataNames(), dataParams(), dataNotes(), dataPaths(), dataTag() or mcols()
#>
#> name Path
#> BFC1 | outfile.txt /tmp/RtmpDTBgrk/SharedData/outfile.txtThe package can also be used to add annotation and notes to existing data resources or experiment data for management. Here we add exisiting “exp_data” to local data repository.
exp_data <- file.path(tempdir(), "exp_data")
dir.create(exp_data)We first add notes to the data, and then update data repository with information from the new dataset.
annData(exp_data, notes = c("experiment data"))
#> meta.yml added
#> [1] "/tmp/RtmpDTBgrk/exp_data/meta.yml"
dataUpdate(exp_data)
#>
#> Updating data record...
#> exp_data added
#> dataHub with 2 records
#> cache path: /tmp/RtmpDTBgrk/cache/ReUseData
#> # dataUpdate() to update the local data cache
#> # dataSearch() to query a specific dataset
#> # Additional information can be retrieved using:
#> # dataNames(), dataParams(), dataNotes(), dataPaths(), dataTag() or mcols()
#>
#> name Path
#> BFC1 | outfile.txt /tmp/RtmpDTBgrk/SharedData/outfile.txt
#> BFC2 | exp_data /tmp/RtmpDTBgrk/exp_dataNow our data hub cached meta information from two different directories, one from data recipe and one from exisiting data. Data can be retrieved by keywords.
dataSearch("experiment")
#> dataHub with 1 records
#> cache path: /tmp/RtmpDTBgrk/cache/ReUseData
#> # dataUpdate() to update the local data cache
#> # dataSearch() to query a specific dataset
#> # Additional information can be retrieved using:
#> # dataNames(), dataParams(), dataNotes(), dataPaths(), dataTag() or mcols()
#>
#> name Path
#> BFC2 | exp_data /tmp/RtmpDTBgrk/exp_dataNOTE: if the argument cloud=TRUE is enabled, dataUpdate() will
also cache the pregenerated data sets (from evaluation of public
ReUseData recipes) that are available on ReUseData google bucket and
return in the dataHub object that are fully searchable. Please see
the following section for details.
With the prebuilt data recipes for curation (e.g., downloading, unzipping, indexing) of commonly used public data resources we have pregenerated some data sets and put them on the cloud space for direct use.
Before searching, one need to use dataUpdate(cloud=TRUE) to sync the
existing data sets on cloud, then dataSearch() can be used to search
any available data set either in local cache and on the cloud.
gcpdir <- file.path(tempdir(), "gcpData")
dataUpdate(gcpdir, cloud=TRUE)
#>
#> Updating data record...
#> 3fe4666a2ec055_GRCh38.primary_assembly.genome.fa.1.bt2 added
#> 3fe4661fa31ac0_GRCh38.primary_assembly.genome.fa.2.bt2 added
#> 3fe466579f29_GRCh38.primary_assembly.genome.fa.3.bt2 added
#> 3fe4666b7b4f31_GRCh38.primary_assembly.genome.fa.4.bt2 added
#> 3fe4666c249846_GRCh38.primary_assembly.genome.fa.rev.1.bt2 added
#> 3fe4661a495220_GRCh38.primary_assembly.genome.fa.rev.2.bt2 added
#> 3fe466697eda72_outfile.txt added
#> 3fe466766c96a9_GRCh37_to_GRCh38.chain added
#> 3fe4661e9295a8_GRCh37_to_NCBI34.chain added
#> 3fe4661d11062_GRCh37_to_NCBI35.chain added
#> 3fe46636943e5b_GRCh37_to_NCBI36.chain added
#> 3fe4664dfe8a92_GRCh38_to_GRCh37.chain added
#> 3fe46635f75b8a_GRCh38_to_NCBI34.chain added
#> 3fe4663a240d2b_GRCh38_to_NCBI35.chain added
#> 3fe46613a68d88_GRCh38_to_NCBI36.chain added
#> 3fe4665fd61674_NCBI34_to_GRCh37.chain added
#> 3fe4664a491bbf_NCBI34_to_GRCh38.chain added
#> 3fe4662c14978c_NCBI35_to_GRCh37.chain added
#> 3fe46611040baf_NCBI35_to_GRCh38.chain added
#> 3fe466334ebb3b_NCBI36_to_GRCh37.chain added
#> 3fe46653328aa0_NCBI36_to_GRCh38.chain added
#> 3fe4666ceac13f_GRCm38_to_NCBIM36.chain added
#> 3fe46625582dcd_GRCm38_to_NCBIM37.chain added
#> 3fe4665771bf6b_NCBIM36_to_GRCm38.chain added
#> 3fe4663d0c3190_NCBIM37_to_GRCm38.chain added
#> 3fe4666d67c8ad_1000G_omni2.5.b37.vcf.gz added
#> 3fe46652e7c93a_1000G_omni2.5.b37.vcf.gz.tbi added
#> 3fe4667230dc05_Mills_and_1000G_gold_standard.indels.b37.vcf.gz added
#> 3fe4664e8fa121_Mills_and_1000G_gold_standard.indels.b37.vcf.gz.tbi added
#> 3fe4667cc9be0b_1000G_omni2.5.hg38.vcf.gz added
#> 3fe4664db7ee30_1000G_omni2.5.hg38.vcf.gz.tbi added
#> 3fe46638be6177_Mills_and_1000G_gold_standard.indels.hg38.vcf.gz added
#> 3fe4661c6cd8cc_Mills_and_1000G_gold_standard.indels.hg38.vcf.gz.tbi added
#> 3fe4664e0f8d5a_af-only-gnomad.raw.sites.vcf added
#> 3fe4662439b0a8_af-only-gnomad.raw.sites.vcf.idx added
#> 3fe4668917112_Mutect2-exome-panel.vcf.idx added
#> 3fe4666858df7a_Mutect2-WGS-panel-b37.vcf added
#> 3fe466db88b1a_Mutect2-WGS-panel-b37.vcf.idx added
#> 3fe4667efe07bb_small_exac_common_3.vcf added
#> 3fe4666eb7523_small_exac_common_3.vcf.idx added
#> 3fe466f899b7c_1000g_pon.hg38.vcf.gz added
#> 3fe46635924617_1000g_pon.hg38.vcf.gz.tbi added
#> 3fe46654e9ffb5_af-only-gnomad.hg38.vcf.gz added
#> 3fe4664580f706_af-only-gnomad.hg38.vcf.gz.tbi added
#> 3fe4666fb65342_small_exac_common_3.hg38.vcf.gz added
#> 3fe46668908d3e_small_exac_common_3.hg38.vcf.gz.tbi added
#> 3fe46625570d7a_gencode.v41.annotation.gtf added
#> 3fe46639ff6f01_gencode.v42.annotation.gtf added
#> 3fe46614a524ca_gencode.vM30.annotation.gtf added
#> 3fe466365b1929_gencode.vM31.annotation.gtf added
#> 3fe4666d4e2a3c_gencode.v41.transcripts.fa added
#> 3fe46667d7af6b_gencode.v41.transcripts.fa.fai added
#> 3fe4662345da68_gencode.v42.transcripts.fa added
#> 3fe46612a65809_gencode.v42.transcripts.fa.fai added
#> 3fe4663f496ed6_gencode.vM30.pc_transcripts.fa added
#> 3fe46660520bf9_gencode.vM30.pc_transcripts.fa.fai added
#> 3fe466e20b7_gencode.vM31.pc_transcripts.fa added
#> 3fe46612313811_gencode.vM31.pc_transcripts.fa.fai added
#> 3fe4665282e7fe_GRCh38.primary_assembly.genome.fa.1.ht2 added
#> 3fe4664e9dc1d8_GRCh38.primary_assembly.genome.fa.2.ht2 added
#> 3fe466efaf61c_GRCh38.primary_assembly.genome.fa.3.ht2 added
#> 3fe466203ad62f_GRCh38.primary_assembly.genome.fa.4.ht2 added
#> 3fe46675c234f_GRCh38.primary_assembly.genome.fa.5.ht2 added
#> 3fe4662b67cee8_GRCh38.primary_assembly.genome.fa.6.ht2 added
#> 3fe4666e4a6389_GRCh38.primary_assembly.genome.fa.7.ht2 added
#> 3fe4662b95d3f7_GRCh38.primary_assembly.genome.fa.8.ht2 added
#> 3fe46633f93ffb_GRCh38_full_analysis_set_plus_decoy_hla.fa.fai added
#> 3fe46656a34303_GRCh38_full_analysis_set_plus_decoy_hla.fa.amb added
#> 3fe466394e5f11_GRCh38_full_analysis_set_plus_decoy_hla.fa.ann added
#> 3fe46632f747b6_GRCh38_full_analysis_set_plus_decoy_hla.fa.bwt added
#> 3fe4665d8eb826_GRCh38_full_analysis_set_plus_decoy_hla.fa.pac added
#> 3fe46648d7fa8d_GRCh38_full_analysis_set_plus_decoy_hla.fa.sa added
#> 3fe46668898dcd_GRCh38_full_analysis_set_plus_decoy_hla.fa added
#> 3fe4663278b7dc_GRCh38.primary_assembly.genome.fa.fai added
#> 3fe466e58f194_GRCh38.primary_assembly.genome.fa.amb added
#> 3fe466583fe10f_GRCh38.primary_assembly.genome.fa.ann added
#> 3fe4661b09451a_GRCh38.primary_assembly.genome.fa.bwt added
#> 3fe46633afff0e_GRCh38.primary_assembly.genome.fa.pac added
#> 3fe466123f5010_GRCh38.primary_assembly.genome.fa.sa added
#> 3fe4662fae69e4_GRCh38.primary_assembly.genome.fa added
#> 3fe4666a0b1838_hs37d5.fa.fai added
#> 3fe4667f8d7a4c_hs37d5.fa.amb added
#> 3fe4661786194f_hs37d5.fa.ann added
#> 3fe466d50f2a0_hs37d5.fa.bwt added
#> 3fe4661233d256_hs37d5.fa.pac added
#> 3fe46656cf8826_hs37d5.fa.sa added
#> 3fe4666da2fe99_hs37d5.fa added
#> 3fe4661241f30d_complete_ref_lens.bin added
#> 3fe4666900c037_ctable.bin added
#> 3fe4664025e698_ctg_offsets.bin added
#> 3fe46660dfb4e5_duplicate_clusters.tsv added
#> 3fe46677fbb653_info.json added
#> 3fe4666060bcc7_mphf.bin added
#> 3fe466683bd835_pos.bin added
#> 3fe4662363853c_pre_indexing.log added
#> 3fe4664eab2050_rank.bin added
#> 3fe46613d1ac2c_ref_indexing.log added
#> 3fe466575cc537_refAccumLengths.bin added
#> 3fe466254e6353_reflengths.bin added
#> 3fe4664d200b3e_refseq.bin added
#> 3fe466a540ced_seq.bin added
#> 3fe4662dd1b7a_versionInfo.json added
#> 3fe46615f805cb_salmon_index added
#> 3fe46672dd9abb_chrLength.txt added
#> 3fe4663555d356_chrName.txt added
#> 3fe4662450f75f_chrNameLength.txt added
#> 3fe4664b1d7bca_chrStart.txt added
#> 3fe466505f1870_exonGeTrInfo.tab added
#> 3fe4665800f66e_exonInfo.tab added
#> 3fe4665d5ccbdb_geneInfo.tab added
#> 3fe466d8254_Genome added
#> 3fe466420c0ea6_genomeParameters.txt added
#> 3fe4665cea4627_Log.out added
#> 3fe46617939ba4_SA added
#> 3fe4664f5d0146_SAindex added
#> 3fe4666f1e187d_sjdbInfo.txt added
#> 3fe4666e6323ca_sjdbList.fromGTF.out.tab added
#> 3fe4663cffffe0_sjdbList.out.tab added
#> 3fe4661600b8a_transcriptInfo.tab added
#> 3fe4665763e401_GRCh38.GENCODE.v42_100 added
#> 3fe4667d25e678_knownGene_hg38.sql added
#> 3fe466623fc070_knownGene_hg38.txt added
#> 3fe4664f5f9a54_refGene_hg38.sql added
#> 3fe4665d86a33f_refGene_hg38.txt added
#> 3fe4664a7b98a5_knownGene_mm39.sql added
#> 3fe46672c31f90_knownGene_mm39.txt added
#> 3fe4662c31c38f_refGene_mm39.sql added
#> 3fe4665e4d44d1_refGene_mm39.txt added
#> dataHub with 130 records
#> cache path: /tmp/RtmpDTBgrk/cache/ReUseData
#> # dataUpdate() to update the local data cache
#> # dataSearch() to query a specific dataset
#> # Additional information can be retrieved using:
#> # dataNames(), dataParams(), dataNotes(), dataPaths(), dataTag() or mcols()
#>
#> name
#> BFC1 | outfile.txt
#> BFC2 | exp_data
#> BFC3 | GRCh38.primary_assembly.genome.fa.1.bt2
#> BFC4 | GRCh38.primary_assembly.genome.fa.2.bt2
#> BFC5 | GRCh38.primary_assembly.genome.fa.3.bt2
#> ... ...
#> BFC126 | refGene_hg38.txt
#> BFC127 | knownGene_mm39.sql
#> BFC128 | knownGene_mm39.txt
#> BFC129 | refGene_mm39.sql
#> BFC130 | refGene_mm39.txt
#> Path
#> BFC1 /tmp/RtmpDTBgrk/SharedData/outfile.txt
#> BFC2 /tmp/RtmpDTBgrk/exp_data
#> BFC3 https://storage.googleapis.com/reusedata/bowtie2_index/GRCh38.pr...
#> BFC4 https://storage.googleapis.com/reusedata/bowtie2_index/GRCh38.pr...
#> BFC5 https://storage.googleapis.com/reusedata/bowtie2_index/GRCh38.pr...
#> ... ...
#> BFC126 https://storage.googleapis.com/reusedata/ucsc_database/refGene_h...
#> BFC127 https://storage.googleapis.com/reusedata/ucsc_database/knownGene...
#> BFC128 https://storage.googleapis.com/reusedata/ucsc_database/knownGene...
#> BFC129 https://storage.googleapis.com/reusedata/ucsc_database/refGene_m...
#> BFC130 https://storage.googleapis.com/reusedata/ucsc_database/refGene_m...If the data of interest already exist on the cloud, then
getCloudData will directly download the data to your computer. Add
it to the local caching system using dataUpdate() for later use.
(dh <- dataSearch(c("ensembl", "GRCh38")))
#> dataHub with 8 records
#> cache path: /tmp/RtmpDTBgrk/cache/ReUseData
#> # dataUpdate() to update the local data cache
#> # dataSearch() to query a specific dataset
#> # Additional information can be retrieved using:
#> # dataNames(), dataParams(), dataNotes(), dataPaths(), dataTag() or mcols()
#>
#> name
#> BFC10 | GRCh37_to_GRCh38.chain
#> BFC14 | GRCh38_to_GRCh37.chain
#> BFC15 | GRCh38_to_NCBI34.chain
#> BFC16 | GRCh38_to_NCBI35.chain
#> BFC17 | GRCh38_to_NCBI36.chain
#> BFC19 | NCBI34_to_GRCh38.chain
#> BFC21 | NCBI35_to_GRCh38.chain
#> BFC23 | NCBI36_to_GRCh38.chain
#> Path
#> BFC10 https://storage.googleapis.com/reusedata/ensembl_liftover/GRCh37...
#> BFC14 https://storage.googleapis.com/reusedata/ensembl_liftover/GRCh38...
#> BFC15 https://storage.googleapis.com/reusedata/ensembl_liftover/GRCh38...
#> BFC16 https://storage.googleapis.com/reusedata/ensembl_liftover/GRCh38...
#> BFC17 https://storage.googleapis.com/reusedata/ensembl_liftover/GRCh38...
#> BFC19 https://storage.googleapis.com/reusedata/ensembl_liftover/NCBI34...
#> BFC21 https://storage.googleapis.com/reusedata/ensembl_liftover/NCBI35...
#> BFC23 https://storage.googleapis.com/reusedata/ensembl_liftover/NCBI36...
getCloudData(dh[1], outdir = gcpdir)
#> Data is downloaded:
#> /tmp/RtmpDTBgrk/gcpData/GRCh37_to_GRCh38.chainNow we create the data cache with only local data files, and we can see that the downloaded data is available.
dataUpdate(gcpdir) ## Update local data cache (without cloud data)
#>
#> Updating data record...
#> GRCh37_to_GRCh38.chain added
#> dataHub with 131 records
#> cache path: /tmp/RtmpDTBgrk/cache/ReUseData
#> # dataUpdate() to update the local data cache
#> # dataSearch() to query a specific dataset
#> # Additional information can be retrieved using:
#> # dataNames(), dataParams(), dataNotes(), dataPaths(), dataTag() or mcols()
#>
#> name
#> BFC1 | outfile.txt
#> BFC2 | exp_data
#> BFC3 | GRCh38.primary_assembly.genome.fa.1.bt2
#> BFC4 | GRCh38.primary_assembly.genome.fa.2.bt2
#> BFC5 | GRCh38.primary_assembly.genome.fa.3.bt2
#> ... ...
#> BFC127 | knownGene_mm39.sql
#> BFC128 | knownGene_mm39.txt
#> BFC129 | refGene_mm39.sql
#> BFC130 | refGene_mm39.txt
#> BFC131 | GRCh37_to_GRCh38.chain
#> Path
#> BFC1 /tmp/RtmpDTBgrk/SharedData/outfile.txt
#> BFC2 /tmp/RtmpDTBgrk/exp_data
#> BFC3 https://storage.googleapis.com/reusedata/bowtie2_index/GRCh38.pr...
#> BFC4 https://storage.googleapis.com/reusedata/bowtie2_index/GRCh38.pr...
#> BFC5 https://storage.googleapis.com/reusedata/bowtie2_index/GRCh38.pr...
#> ... ...
#> BFC127 https://storage.googleapis.com/reusedata/ucsc_database/knownGene...
#> BFC128 https://storage.googleapis.com/reusedata/ucsc_database/knownGene...
#> BFC129 https://storage.googleapis.com/reusedata/ucsc_database/refGene_m...
#> BFC130 https://storage.googleapis.com/reusedata/ucsc_database/refGene_m...
#> BFC131 /tmp/RtmpDTBgrk/gcpData/GRCh37_to_GRCh38.chain
dataSearch() ## data is available locally!!!
#> dataHub with 131 records
#> cache path: /tmp/RtmpDTBgrk/cache/ReUseData
#> # dataUpdate() to update the local data cache
#> # dataSearch() to query a specific dataset
#> # Additional information can be retrieved using:
#> # dataNames(), dataParams(), dataNotes(), dataPaths(), dataTag() or mcols()
#>
#> name
#> BFC1 | outfile.txt
#> BFC2 | exp_data
#> BFC3 | GRCh38.primary_assembly.genome.fa.1.bt2
#> BFC4 | GRCh38.primary_assembly.genome.fa.2.bt2
#> BFC5 | GRCh38.primary_assembly.genome.fa.3.bt2
#> ... ...
#> BFC127 | knownGene_mm39.sql
#> BFC128 | knownGene_mm39.txt
#> BFC129 | refGene_mm39.sql
#> BFC130 | refGene_mm39.txt
#> BFC131 | GRCh37_to_GRCh38.chain
#> Path
#> BFC1 /tmp/RtmpDTBgrk/SharedData/outfile.txt
#> BFC2 /tmp/RtmpDTBgrk/exp_data
#> BFC3 https://storage.googleapis.com/reusedata/bowtie2_index/GRCh38.pr...
#> BFC4 https://storage.googleapis.com/reusedata/bowtie2_index/GRCh38.pr...
#> BFC5 https://storage.googleapis.com/reusedata/bowtie2_index/GRCh38.pr...
#> ... ...
#> BFC127 https://storage.googleapis.com/reusedata/ucsc_database/knownGene...
#> BFC128 https://storage.googleapis.com/reusedata/ucsc_database/knownGene...
#> BFC129 https://storage.googleapis.com/reusedata/ucsc_database/refGene_m...
#> BFC130 https://storage.googleapis.com/reusedata/ucsc_database/refGene_m...
#> BFC131 /tmp/RtmpDTBgrk/gcpData/GRCh37_to_GRCh38.chainThe data supports user-friendly discovery and access through the
ReUseData portal, where detailed instructions are provided for
straight-forward incorporation into data analysis pipelines run on
local computing nodes, web resources, and cloud computing platforms
(e.g., Terra, CGC).
Here we provide a function meta_data() to create a data frame that
contains all information about the data sets in the specified file
path (recursively), including the annotation file ($yml column),
parameter values for the recipe ($params column), data file path
($output column), keywords for data file (notes columns), date of
data generation (date column), and any tag if available (tag
column).
Use cleanup = TRUE to cleanup any invalid or expired/older
intermediate files.
mt <- meta_data(outdir)
head(mt)
#> yml
#> 1 /tmp/RtmpDTBgrk/SharedData/echo_out_Hello_World!_outfile.yml
#> params output
#> 1 input: Hello World!; outfile: outfile /tmp/RtmpDTBgrk/SharedData/outfile.txt
#> notes date
#> 1 echo hello world txt 2024-05-01 01:35:51.665942sessionInfo()
#> R version 4.4.0 beta (2024-04-15 r86425)
#> Platform: x86_64-pc-linux-gnu
#> Running under: Ubuntu 22.04.4 LTS
#>
#> Matrix products: default
#> BLAS: /home/biocbuild/bbs-3.19-bioc/R/lib/libRblas.so
#> LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.10.0
#>
#> locale:
#> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
#> [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
#> [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
#> [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
#> [9] LC_ADDRESS=C LC_TELEPHONE=C
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
#>
#> time zone: America/New_York
#> tzcode source: system (glibc)
#>
#> attached base packages:
#> [1] stats4 stats graphics grDevices utils datasets methods
#> [8] base
#>
#> other attached packages:
#> [1] ReUseData_1.4.0 Rcwl_1.20.0 S4Vectors_0.42.0
#> [4] BiocGenerics_0.50.0 yaml_2.3.8 BiocStyle_2.32.0
#>
#> loaded via a namespace (and not attached):
#> [1] tidyselect_1.2.1 dplyr_1.1.4 blob_1.2.4
#> [4] filelock_1.0.3 R.utils_2.12.3 fastmap_1.1.1
#> [7] BiocFileCache_2.12.0 promises_1.3.0 digest_0.6.35
#> [10] base64url_1.4 mime_0.12 lifecycle_1.0.4
#> [13] RSQLite_2.3.6 magrittr_2.0.3 compiler_4.4.0
#> [16] rlang_1.1.3 sass_0.4.9 progress_1.2.3
#> [19] tools_4.4.0 utf8_1.2.4 data.table_1.15.4
#> [22] knitr_1.46 prettyunits_1.2.0 brew_1.0-10
#> [25] htmlwidgets_1.6.4 bit_4.0.5 curl_5.2.1
#> [28] reticulate_1.36.1 RColorBrewer_1.1-3 batchtools_0.9.17
#> [31] BiocParallel_1.38.0 purrr_1.0.2 withr_3.0.0
#> [34] R.oo_1.26.0 grid_4.4.0 fansi_1.0.6
#> [37] git2r_0.33.0 xtable_1.8-4 debugme_1.2.0
#> [40] cli_3.6.2 rmarkdown_2.26 DiagrammeR_1.0.11
#> [43] crayon_1.5.2 generics_0.1.3 httr_1.4.7
#> [46] visNetwork_2.1.2 DBI_1.2.2 cachem_1.0.8
#> [49] parallel_4.4.0 BiocManager_1.30.22 basilisk_1.16.0
#> [52] vctrs_0.6.5 Matrix_1.7-0 jsonlite_1.8.8
#> [55] dir.expiry_1.12.0 bookdown_0.39 hms_1.1.3
#> [58] bit64_4.0.5 jquerylib_0.1.4 RcwlPipelines_1.20.0
#> [61] glue_1.7.0 codetools_0.2-20 stringi_1.8.3
#> [64] later_1.3.2 tibble_3.2.1 pillar_1.9.0
#> [67] basilisk.utils_1.16.0 rappdirs_0.3.3 htmltools_0.5.8.1
#> [70] R6_2.5.1 dbplyr_2.5.0 evaluate_0.23
#> [73] shiny_1.8.1.1 lattice_0.22-6 R.methodsS3_1.8.2
#> [76] png_0.1-8 backports_1.4.1 memoise_2.0.1
#> [79] httpuv_1.6.15 bslib_0.7.0 Rcpp_1.0.12
#> [82] checkmate_2.3.1 xfun_0.43 pkgconfig_2.0.3