ISAnalytics 1.0.11
ISAnalytics import functions familyIn this vignette we’re going to explain more in detail how functions of the import family should be used, the most common workflows to follow and more.
To install the package run the following code:
## For release version
if (!requireNamespace("BiocManager", quietly = TRUE)) {
install.packages("BiocManager")
}
BiocManager::install("ISAnalytics")
## For devel version
if (!requireNamespace("BiocManager", quietly = TRUE)) {
install.packages("BiocManager")
}
# The following initializes usage of Bioc devel
BiocManager::install(version = "devel")
BiocManager::install("ISAnalytics")To install from GitHub:
# For release version
if (!require(devtools)) {
install.packages("devtools")
}
devtools::install_github("calabrialab/ISAnalytics",
ref = "RELEASE_3_12",
dependencies = TRUE,
build_vignettes = TRUE
)
## Safer option for vignette building issue
devtools::install_github("calabrialab/ISAnalytics",
ref = "RELEASE_3_12"
)
# For devel version
if (!require(devtools)) {
install.packages("devtools")
}
devtools::install_github("calabrialab/ISAnalytics",
ref = "master",
dependencies = TRUE,
build_vignettes = TRUE
)
## Safer option for vignette building issue
devtools::install_github("calabrialab/ISAnalytics",
ref = "master"
)library(ISAnalytics)ISAnalytics has a verbose option that allows some functions to print
additional information to the console while they’re executing.
To disable this feature do:
# DISABLE
options("ISAnalytics.verbose" = FALSE)
# ENABLE
options("ISAnalytics.verbose" = TRUE)Some functions also produce report in a user-friendly HTML format, to set this feature:
# DISABLE HTML REPORTS
options("ISAnalytics.widgets" = FALSE)
# ENABLE HTML REPORTS
options("ISAnalytics.widgets" = TRUE)The vast majority of the functions included in this package is designed to work in combination with Vispa2 pipeline. If you don’t know what it is, we strongly recommend you to take a look at these links:
Vispa2 produces a standard file system structure starting from a folder you specify as your workbench or root. The structure always follows this schema:
We’ve included 2 examples of this structure in our package, one correct and the
other one including errors or potential problems. They are both in .zip format,
so you might want to unzip them if you plan to experiment with them.
An example on how to access them:
root_correct <- system.file("extdata", "fs.zip", package = "ISAnalytics")
root_correct <- unzip_file_system(root_correct, "fs")
fs::dir_tree(root_correct)
#> /tmp/RtmpHKhRpe/fs
#> ├── PROJECT1100
#> │ ├── bam
#> │ ├── bcmuxall
#> │ ├── bed
#> │ ├── iss
#> │ │ ├── ABX-LR-PL5-POOL14-1
#> │ │ │ └── stats.sequence_PROJECT1100.ABX-LR-PL5-POOL14-1.tsv
#> │ │ └── ABX-LR-PL6-POOL15-1
#> │ │ └── stats.sequence_PROJECT1100.ABX-LR-PL6-POOL15-1.tsv
#> │ ├── quality
#> │ ├── quantification
#> │ │ ├── ABX-LR-PL5-POOL14-1
#> │ │ │ ├── PROJECT1100_ABX-LR-PL5-POOL14-1_fragmentEstimate_matrix.no0.annotated.tsv.xz
#> │ │ │ ├── PROJECT1100_ABX-LR-PL5-POOL14-1_fragmentEstimate_matrix.tsv.xz
#> │ │ │ ├── PROJECT1100_ABX-LR-PL5-POOL14-1_seqCount_matrix.no0.annotated.tsv.xz
#> │ │ │ └── PROJECT1100_ABX-LR-PL5-POOL14-1_seqCount_matrix.tsv.xz
#> │ │ └── ABX-LR-PL6-POOL15-1
#> │ │ ├── PROJECT1100_ABX-LR-PL6-POOL15-1_fragmentEstimate_matrix.no0.annotated.tsv.xz
#> │ │ ├── PROJECT1100_ABX-LR-PL6-POOL15-1_fragmentEstimate_matrix.tsv.xz
#> │ │ ├── PROJECT1100_ABX-LR-PL6-POOL15-1_seqCount_matrix.no0.annotated.tsv.xz
#> │ │ └── PROJECT1100_ABX-LR-PL6-POOL15-1_seqCount_matrix.tsv.xz
#> │ └── report
#> ├── PROJECT1101
#> │ ├── bam
#> │ ├── bcmuxall
#> │ ├── bed
#> │ ├── iss
#> │ │ └── ABY-LR-PL4-POOL54-2
#> │ │ └── stats.sequence_PROJECT1101.ABY-LR-PL4-POOL54-2.tsv
#> │ ├── quality
#> │ ├── quantification
#> │ │ └── ABY-LR-PL4-POOL54-2
#> │ │ ├── PROJECT1101_ABY-LR-PL4-POOL54-2_fragmentEstimate_matrix.no0.annotated.tsv.xz
#> │ │ ├── PROJECT1101_ABY-LR-PL4-POOL54-2_fragmentEstimate_matrix.tsv.xz
#> │ │ ├── PROJECT1101_ABY-LR-PL4-POOL54-2_seqCount_matrix.no0.annotated.tsv.xz
#> │ │ └── PROJECT1101_ABY-LR-PL4-POOL54-2_seqCount_matrix.tsv.xz
#> │ └── report
#> └── VA
#> └── CLOEXP
#> ├── bam
#> ├── bcmuxall
#> ├── bed
#> ├── iss
#> │ └── POOL6-1
#> │ └── stats.sequence_VA.POOL6-1.tsv
#> ├── quality
#> ├── quantification
#> │ └── POOL6-1
#> │ ├── VA-CLOEXP-POOL6-1_fragmentEstimate_matrix.no0.annotated.tsv.xz
#> │ ├── VA-CLOEXP-POOL6-1_fragmentEstimate_matrix.tsv.xz
#> │ ├── VA-CLOEXP-POOL6-1_seqCount_matrix.no0.annotated.tsv.xz
#> │ └── VA-CLOEXP-POOL6-1_seqCount_matrix.tsv.xz
#> └── reportIf you want to import a single integration matrix you can do so by using the
import_single_Vispa2Matrix function.
This function reads the file and converts it into a tidy structure: several
different formats can be read, since you can specify the column separator.
If you’re not familiar with the “tidy” concept, we recommend to take a look at
this link to get the basics:
This package is in fact based on the tidyverse and tries to follow its
philosophy and guidelines as close as possible.
Vispa2 pipeline and the associated Create Matrix tool produce matrices with a
standard structure which we’ll refer to as “messy”, because different
experimental data is divided in different columns and there are a lot of
NA values.
| chr | integration_locus | strand | GeneName | GeneStrand | exp_1 | exp_2 | exp_3 | exp_4 | exp_5 |
|---|---|---|---|---|---|---|---|---|---|
| 20 | 39085641 | + | MAFB | - | NA | 94925.98 | NA | NA | 61143.20 |
| 7 | 155446768 | + | RBM33 | + | NA | NA | 68981.02 | NA | NA |
| 19 | 12278742 | - | ZNF136 | + | NA | NA | 60410.18 | NA | 68463.19 |
| 6 | 1865825 | - | GMDS | - | NA | NA | NA | 46666.04 | NA |
| 7 | 21508806 | + | SP4 | + | NA | NA | NA | NA | NA |
| 17 | 5187666 | + | RABEP1 | + | NA | NA | NA | NA | NA |
example_matrix_path <- system.file("extdata", "ex_annotated_ISMatrix.tsv.xz",
package = "ISAnalytics"
)
imported_im <- import_single_Vispa2Matrix(
path = example_matrix_path,
to_exclude = NULL,
separator = "\t"
)
#> Warning: compression format not supported by fread
#> ℹ File will be read using readr
#> Reading file...
#> ℹ Mode: classic
#> *** File info ***
#> * --- Matrix type: NEW
#> * --- Annotated: TRUE
#> * --- Dimensions: 50 x 10
#> * --- Read mode: classic
#> Reshaping...
#>
|
| | 0%
|
|===== | 5%
|
|========== | 9%
|
|=============== | 14%
|
|==================== | 18%
|
|========================= | 23%
|
|============================== | 27%
|
|=================================== | 32%
|
|======================================== | 36%
|
|============================================= | 41%
|
|================================================== | 45%
|
|======================================================= | 50%
|
|============================================================ | 55%
|
|================================================================= | 59%
|
|====================================================================== | 64%
|
|=========================================================================== | 68%
|
|================================================================================ | 73%
|
|===================================================================================== | 77%
|
|========================================================================================== | 82%
|
|=============================================================================================== | 86%
|
|==================================================================================================== | 91%
|
|========================================================================================================= | 95%
|
|==============================================================================================================| 100%
#> Done!| chr | integration_locus | strand | GeneName | GeneStrand | CompleteAmplificationID | Value |
|---|---|---|---|---|---|---|
| 20 | 39085641 | + | MAFB | - | exp_2 | 94925.98 |
| 20 | 39085641 | + | MAFB | - | exp_5 | 61143.20 |
| 7 | 155446768 | + | RBM33 | + | exp_3 | 68981.02 |
| 19 | 4049601 | + | ZBTB7A | - | exp_1 | 99324.68 |
| 19 | 4049601 | + | ZBTB7A | - | exp_2 | 83873.38 |
| 19 | 12278742 | - | ZNF136 | + | exp_3 | 60410.18 |
We will refer to the structure generated by import_single_Vispa2Matrix as
“integration matrix” for convenience.
To be considered an integration matrix the data frame must contain the mandatory
variables, which are “chr” (chromosome), “integration_locus” and “strand”.
It might also contain annotation variables if the matrix was annotated during
the Vispa2 pipeline run.
You can access these names by using two functions:
# Displays the mandatory vars, can be called also for manipulation purposes
# on tibble instead of calling individual variables
mandatory_IS_vars()
#> [1] "chr" "integration_locus" "strand"
# Displays the annotation variables
annotation_IS_vars()
#> [1] "GeneName" "GeneStrand"You can of course operate on the integration matrices as you would on any other data frame, but some functions will check the presence of specific columns because they’re needed in that context.
While you can import single matrices for brief analysis, what you would like to
do most of the times is import multiple matrices at once, based on certain
parameters. To do that you must first import the association file, which is the
file that holds all associated metadata and information about every project,
pool and single experiment.
The function that imports this file does not simply read it into your R
environment, but performs an alignment check with your file system, so you
have to specify the path to the root folder where your Vispa2 runs produce
output (see the previous section).
To import the association file do:
path_as_file <- system.file("extdata", "ex_association_file.tsv",
package = "ISAnalytics"
)
withr::with_options(list(ISAnalytics.widgets = FALSE), {
association_file <- import_association_file(
path = path_as_file,
root = root_correct,
tp_padding = 4,
dates_format = "dmy",
separator = "\t"
)
})
#> *** Association file import summary ***
#> ℹ For detailed report please set option 'ISAnalytics.widgets' to TRUE
#> * Parsing problems detected: TRUE
#> * Date parsing problems: TRUE
#> * Column problems detected: TRUE
#> * NAs found in important columns: TRUE
#> * File system alignment: no problems detected
association_file
#> # A tibble: 49 x 67
#> ProjectID FUSIONID PoolID TagSequence SubjectID VectorType VectorID ExperimentID Tissue TimePoint DNAFragmentation
#> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr>
#> 1 CLOEXP LR-Clonal… POOL6 LTR10LC10 VA2020-m… lenti 737.pCCL… PLATE1 <NA> 0000 SONIC
#> 2 CLOEXP LR-Clonal… POOL6 LTR11LC11 VA2020-m… lenti 737.pCCL… PLATE1 <NA> 0000 SONIC
#> 3 CLOEXP LR-Clonal… POOL6 LTR12LC12 VA2020-m… lenti 737.pCCL… PLATE1 <NA> 0000 SONIC
#> 4 CLOEXP LR-Clonal… POOL6 LTR13LC13 VA2020-m… lenti 737.pCCL… PLATE1 <NA> 0000 SONIC
#> 5 CLOEXP LR-Clonal… POOL6 LTR29LC29 VA2020-m… lenti 737.pCCL… PLATE1 <NA> 0000 SONIC
#> 6 CLOEXP LR-Clonal… POOL6 LTR30LC30 VA2020-m… lenti 737.pCCL… PLATE1 <NA> 0000 SONIC
#> 7 CLOEXP LR-Clonal… POOL6 LTR31LC31 VA2020-m… lenti 737.pCCL… PLATE1 <NA> 0000 SONIC
#> 8 CLOEXP LR-Clonal… POOL6 LTR32LC32 VA2020-m… lenti 737.pCCL… PLATE1 <NA> 0000 SONIC
#> 9 CLOEXP LR-Clonal… POOL6 LTR48LC48 VA2020-m… lenti 737.pCCL… PLATE1 <NA> 0000 SONIC
#> 10 CLOEXP LR-Clonal… POOL6 LTR49LC49 VA2020-m… lenti 737.pCCL… PLATE1 <NA> 0000 SONIC
#> # … with 39 more rows, and 56 more variables: PCRMethod <chr>, TagIDextended <chr>, Keywords <chr>, CellMarker <chr>,
#> # TagID <chr>, NGSProvider <chr>, NGSTechnology <chr>, ConverrtedFilesDir <chr>, ConverrtedFilesName <chr>,
#> # SourceFileFolder <chr>, SourceFileNameR1 <chr>, SourceFileNameR2 <chr>, DNAnumber <chr>, ReplicateNumber <int>,
#> # DNAextractionDate <dttm>, DNAngUsed <dbl>, LinearPCRID <chr>, LinearPCRDate <dttm>, SonicationDate <dttm>,
#> # LigationDate <dttm>, 1stExpoPCRID <chr>, 1stExpoPCRDate <dttm>, 2ndExpoID <chr>, 2ndExpoDate <dttm>,
#> # FusionPrimerPCRID <chr>, FusionPrimerPCRDate <dttm>, PoolDate <dttm>, SequencingDate <dttm>, VCN <dbl>,
#> # Genome <chr>, SequencingRound <int>, Genotype <chr>, TestGroup <chr>, MOI <chr>, Engraftment <dbl>,
#> # Transduction <dbl>, Notes <chr>, AddedField1 <chr>, AddedField2 <chr>, AddedField3 <chr>, AddedField4 <chr>,
#> # concatenatePoolIDSeqRun <chr>, AddedField6_RelativeBloodPercentage <chr>, AddedField7_PurityTestFeasibility <dbl>,
#> # AddedField8_FacsSeparationPurity <dbl>, Kapa <dbl>, ulForPool <dbl>, CompleteAmplificationID <chr>, UniqueID <chr>,
#> # StudyTestID <chr>, StudyTestGroup <int>, MouseID <int>, Tigroup <chr>, Tisource <chr>, PathToFolderProjectID <chr>,
#> # Path <chr>If you have the “widgets” option active, this will produce a visual HTML report
of the results of the alignment check, either in Rstudio or in your browser.
If projects or pools are missing you will be notified: until you fix those
problems, those elements will be ignored until you re-import the association
file.
If you’re not interested in scanning the file system you can set the ‘root’ parameter to NULL and this step will be skipped.
The function can read multiple file formats including excel files, however since metadata are crucial for a correct workflow, we recommend using .tsv or .csv format to avoid potential parsing problems. Additionally you can also specify a filter to obtain a pre-filtered association file for your needs:
withr::with_options(list(ISAnalytics.widgets = FALSE), {
association_file_filtered <- import_association_file(
path = path_as_file,
root = root_correct,
tp_padding = 4,
dates_format = "dmy",
separator = "\t",
filter_for = list(ProjectID = "CLOEXP")
)
})
#> *** Association file import summary ***
#> ℹ For detailed report please set option 'ISAnalytics.widgets' to TRUE
#> * Parsing problems detected: TRUE
#> * Date parsing problems: TRUE
#> * Column problems detected: TRUE
#> * NAs found in important columns: TRUE
#> * File system alignment: no problems detected
association_file_filtered
#> # A tibble: 12 x 67
#> ProjectID FUSIONID PoolID TagSequence SubjectID VectorType VectorID ExperimentID Tissue TimePoint DNAFragmentation
#> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr> <chr>
#> 1 CLOEXP LR-Clonal… POOL6 LTR10LC10 VA2020-m… lenti 737.pCCL… PLATE1 <NA> 0000 SONIC
#> 2 CLOEXP LR-Clonal… POOL6 LTR11LC11 VA2020-m… lenti 737.pCCL… PLATE1 <NA> 0000 SONIC
#> 3 CLOEXP LR-Clonal… POOL6 LTR12LC12 VA2020-m… lenti 737.pCCL… PLATE1 <NA> 0000 SONIC
#> 4 CLOEXP LR-Clonal… POOL6 LTR13LC13 VA2020-m… lenti 737.pCCL… PLATE1 <NA> 0000 SONIC
#> 5 CLOEXP LR-Clonal… POOL6 LTR29LC29 VA2020-m… lenti 737.pCCL… PLATE1 <NA> 0000 SONIC
#> 6 CLOEXP LR-Clonal… POOL6 LTR30LC30 VA2020-m… lenti 737.pCCL… PLATE1 <NA> 0000 SONIC
#> 7 CLOEXP LR-Clonal… POOL6 LTR31LC31 VA2020-m… lenti 737.pCCL… PLATE1 <NA> 0000 SONIC
#> 8 CLOEXP LR-Clonal… POOL6 LTR32LC32 VA2020-m… lenti 737.pCCL… PLATE1 <NA> 0000 SONIC
#> 9 CLOEXP LR-Clonal… POOL6 LTR48LC48 VA2020-m… lenti 737.pCCL… PLATE1 <NA> 0000 SONIC
#> 10 CLOEXP LR-Clonal… POOL6 LTR49LC49 VA2020-m… lenti 737.pCCL… PLATE1 <NA> 0000 SONIC
#> 11 CLOEXP LR-Clonal… POOL6 LTR50LC50 VA2020-m… lenti 737.pCCL… PLATE1 <NA> 0000 SONIC
#> 12 CLOEXP LR-Clonal… POOL6 LTR51LC51 VA2020-m… lenti 737.pCCL… PLATE1 <NA> 0000 SONIC
#> # … with 56 more variables: PCRMethod <chr>, TagIDextended <chr>, Keywords <chr>, CellMarker <chr>, TagID <chr>,
#> # NGSProvider <chr>, NGSTechnology <chr>, ConverrtedFilesDir <chr>, ConverrtedFilesName <chr>,
#> # SourceFileFolder <chr>, SourceFileNameR1 <chr>, SourceFileNameR2 <chr>, DNAnumber <chr>, ReplicateNumber <int>,
#> # DNAextractionDate <dttm>, DNAngUsed <dbl>, LinearPCRID <chr>, LinearPCRDate <dttm>, SonicationDate <dttm>,
#> # LigationDate <dttm>, 1stExpoPCRID <chr>, 1stExpoPCRDate <dttm>, 2ndExpoID <chr>, 2ndExpoDate <dttm>,
#> # FusionPrimerPCRID <chr>, FusionPrimerPCRDate <dttm>, PoolDate <dttm>, SequencingDate <dttm>, VCN <dbl>,
#> # Genome <chr>, SequencingRound <int>, Genotype <chr>, TestGroup <chr>, MOI <chr>, Engraftment <dbl>,
#> # Transduction <dbl>, Notes <chr>, AddedField1 <chr>, AddedField2 <chr>, AddedField3 <chr>, AddedField4 <chr>,
#> # concatenatePoolIDSeqRun <chr>, AddedField6_RelativeBloodPercentage <chr>, AddedField7_PurityTestFeasibility <dbl>,
#> # AddedField8_FacsSeparationPurity <dbl>, Kapa <dbl>, ulForPool <dbl>, CompleteAmplificationID <chr>, UniqueID <chr>,
#> # StudyTestID <chr>, StudyTestGroup <int>, MouseID <int>, Tigroup <chr>, Tisource <chr>, PathToFolderProjectID <chr>,
#> # Path <chr>There are 2 different functions for importing multiple matrices in parallel:
import_parallel_Vispa2Matrices_interactiveimport_parallel_Vispa2Matrices_autoThe interactive version will ask you to input your choices directly into the
console, the automatic version will not, but has some limitations.
Both functions rely on the association file and some basic parameters, most
notably:
quantification_type: this is a string or a vector of characters indicating
which quantification types you want the function to look for. The possible
values are fragmentEstimate, seqCount, barcodeCount, cellCount, ShsCountmatrix_type: tells the function if it should consider annotated or not
annotated matrices. The only possible options are “annotated” and
“not_annotated”workers: indicates the number of parallel workers to instantiate when
importing. Keep in mind that the higher is the number, the faster the process
is, but also higher is the RAM peak, so you should be aware of this especially
if you’re dealing with really big matrices. Set this parameter according to
your needs and according to your hardware specifications.Both the versions will produce an HTML report as a summary of the importing process. The report includes:
Both the functions, by default, return a multi-quantification matrix (see ).
As stated before, with the interactive version you have more control and you can directly choose:
If you haven’t imported the association file yet, you can directly pass the path to the association file and the path to the root folder into the function: in this way the association file will automatically be imported.
Example:
withr::with_options(list(ISAnalytics.widgets = FALSE), {
matrices <- import_parallel_Vispa2Matrices_interactive(
association_file = path_as_file,
root = root_correct,
quantification_type = c("fragmentEstimate", "seqCount"),
matrix_type = "annotated",
workers = 2
)
})If you’ve already imported the association file you can instead call the function like this:
matrices <- import_parallel_Vispa2Matrices_interactive(
association_file = association_file,
root = NULL,
quantification_type = c("fragmentEstimate", "seqCount"),
matrix_type = "annotated",
workers = 2
)You can simply access the data frames by doing:
matrices$fragmentEstimate
matrices$seqCountIf you choose to opt for the automatic version you should keep in mind that the function automatically considers everything included in the association file, so if you want to import only a subset of projects and/or pools you should filter the association file according to your criteria before calling the function:
library(magrittr)
refined_af <- association_file %>% dplyr::filter(.data$ProjectID == "CLOEXP")In automatic version there is no way of discriminating duplicates, so there is the possibility to specify additional patterns to look for in file names to mitigate this problem. However, if after matching of the additional patterns duplicates are still found they’re simply discarded.
There are 2 additional parameters to set:
patterns: a string or a character vector containing regular expressions to
be matched on file names. If you’re not familiar with regular expressions,
I suggest you to start
from here stringr cheatsheetmatching_opt: a single string that tells the function how to match the
patterns. The possible values for this parameter are ANY, ALL, OPTIONAL:
patternspatternsYou can call the function with patterns set to NULL if you don’t wish to
match anything:
withr::with_options(list(ISAnalytics.widgets = FALSE), {
matrices_auto <- import_parallel_Vispa2Matrices_auto(
association_file = refined_af,
root = NULL,
quantification_type = c("fragmentEstimate", "seqCount"),
matrix_type = "annotated",
workers = 2,
patterns = NULL,
matching_opt = "ANY" # Same if you choose "ALL" or "OPTIONAL"
)
})
#> [1] "--- REPORT: FILES IMPORTED ---"
#> # A tibble: 2 x 5
#> ProjectID concatenatePoolIDSeqRun Quantification_type
#> <chr> <chr> <chr>
#> 1 CLOEXP POOL6-1 fragmentEstimate
#> 2 CLOEXP POOL6-1 seqCount
#> Files_chosen
#> <fs::path>
#> 1 /tmp/RtmpHKhRpe/fs/VA/CLOEXP/quantification/POOL6-1/VA-CLOEXP-POOL6-1_fragmentEstimate_matrix.no0.annotated.tsv.xz
#> 2 /tmp/RtmpHKhRpe/fs/VA/CLOEXP/quantification/POOL6-1/VA-CLOEXP-POOL6-1_seqCount_matrix.no0.annotated.tsv.xz
#> Imported
#> <lgl>
#> 1 TRUE
#> 2 TRUE
#> [1] "--- SUMMARY OF FILES CHOSEN FOR IMPORT ---"
#> # A tibble: 2 x 4
#> ProjectID concatenatePoolIDSeqRun Quantification_type
#> <chr> <chr> <chr>
#> 1 CLOEXP POOL6-1 fragmentEstimate
#> 2 CLOEXP POOL6-1 seqCount
#> Files_chosen
#> <fs::path>
#> 1 /tmp/RtmpHKhRpe/fs/VA/CLOEXP/quantification/POOL6-1/VA-CLOEXP-POOL6-1_fragmentEstimate_matrix.no0.annotated.tsv.xz
#> 2 /tmp/RtmpHKhRpe/fs/VA/CLOEXP/quantification/POOL6-1/VA-CLOEXP-POOL6-1_seqCount_matrix.no0.annotated.tsv.xz
#> [1] "--- INTEGRATION MATRICES FOUND REPORT ---"
#> # A tibble: 2 x 5
#> ProjectID concatenatePoolIDSeqRun Anomalies Quantification_type
#> <chr> <chr> <lgl> <chr>
#> 1 CLOEXP POOL6-1 FALSE fragmentEstimate
#> 2 CLOEXP POOL6-1 FALSE seqCount
#> Files_found
#> <fs::path>
#> 1 /tmp/RtmpHKhRpe/fs/VA/CLOEXP/quantification/POOL6-1/VA-CLOEXP-POOL6-1_fragmentEstimate_matrix.no0.annotated.tsv.xz
#> 2 /tmp/RtmpHKhRpe/fs/VA/CLOEXP/quantification/POOL6-1/VA-CLOEXP-POOL6-1_seqCount_matrix.no0.annotated.tsv.xz
matrices_auto
#> # A tibble: 1,476 x 8
#> chr integration_locus strand GeneName GeneStrand CompleteAmplificationID fragmentEstimate seqCount
#> <chr> <int> <chr> <chr> <chr> <fct> <dbl> <dbl>
#> 1 10 102326217 + HIF1AN + CLOEXP_POOL6_LTR10LC10_VA2020-mix10_VA… 1.00 1
#> 2 10 20775144 + MIR4675 + CLOEXP_POOL6_LTR10LC10_VA2020-mix10_VA… 1.00 1
#> 3 10 24622789 + KIAA1217 + CLOEXP_POOL6_LTR10LC10_VA2020-mix10_VA… 1.00 1
#> 4 10 30105047 + SVIL - CLOEXP_POOL6_LTR10LC10_VA2020-mix10_VA… 1.00 1
#> 5 10 32656933 - EPC1 - CLOEXP_POOL6_LTR10LC10_VA2020-mix10_VA… 1.00 1
#> 6 10 66717706 + LOC10192… - CLOEXP_POOL6_LTR10LC10_VA2020-mix10_VA… 1.00 1
#> 7 10 76391232 - ADK + CLOEXP_POOL6_LTR10LC10_VA2020-mix10_VA… 1.00 1
#> 8 10 85334019 - LOC10537… - CLOEXP_POOL6_LTR10LC10_VA2020-mix10_VA… 2.00 6
#> 9 11 107591208 + SLN - CLOEXP_POOL6_LTR10LC10_VA2020-mix10_VA… 1.00 1
#> 10 11 111744171 + FDXACB1 - CLOEXP_POOL6_LTR10LC10_VA2020-mix10_VA… 1.00 1
#> # … with 1,466 more rowsLet’s do an example with a file system where there are issues, such as duplicates:
root_err <- system.file("extdata", "fserr.zip", package = "ISAnalytics")
root_err <- unzip_file_system(root_err, "fserr")
fs::dir_tree(root_err)
#> /tmp/RtmpHKhRpe/fserr
#> ├── PROJECT1100
#> │ ├── bam
#> │ ├── bcmuxall
#> │ ├── bed
#> │ ├── iss
#> │ │ └── ABX-LR-PL5-POOL14-1
#> │ │ └── stats.sequence_PROJECT1100.ABX-LR-PL5-POOL14-1.tsv
#> │ ├── quality
#> │ ├── quantification
#> │ │ ├── ABX-LR-PL5-POOL14-1
#> │ │ │ ├── PROJECT1100_ABX-LR-PL5-POOL14-1_fragmentEstimate_matrix-NoMate.no0.annotated.tsv.xz
#> │ │ │ ├── PROJECT1100_ABX-LR-PL5-POOL14-1_fragmentEstimate_matrix-NoMate2.no0.annotated.tsv.xz
#> │ │ │ ├── PROJECT1100_ABX-LR-PL5-POOL14-1_fragmentEstimate_matrix.no0.annotated.tsv.xz
#> │ │ │ └── PROJECT1100_ABX-LR-PL5-POOL14-1_seqCount_matrix.no0.annotated.tsv.xz
#> │ │ └── ABX-LR-PL6-POOL15-1
#> │ │ ├── PROJECT1100_ABX-LR-PL6-POOL15-1_fragmentEstimate_matrix.no0.annotated.tsv.xz
#> │ │ └── PROJECT1100_ABX-LR-PL6-POOL15-1_seqCount_matrix.no0.annotated.tsv.xz
#> │ └── report
#> └── VA
#> └── CLOEXP
#> ├── bam
#> ├── bcmuxall
#> ├── bed
#> ├── iss
#> │ └── POOL6-1
#> │ └── stats.sequence_VA.POOL6-1.tsv
#> ├── quality
#> ├── quantification
#> │ └── POOL6-1
#> │ ├── VA-CLOEXP-POOL6-1_fragmentEstimate_matrix-NoMate.no0.annotated.tsv.xz
#> │ ├── VA-CLOEXP-POOL6-1_fragmentEstimate_matrix.no0.annotated.tsv.xz
#> │ ├── VA-CLOEXP-POOL6-1_fragmentEstimate_matrix.tsv.xz
#> │ ├── VA-CLOEXP-POOL6-1_seqCount_matrix-NoMate.no0.annotated.tsv.xz
#> │ ├── VA-CLOEXP-POOL6-1_seqCount_matrix.no0.annotated.tsv.xz
#> │ └── VA-CLOEXP-POOL6-1_seqCount_matrix.tsv.xz
#> └── report
withr::with_options(list(ISAnalytics.widgets = FALSE), {
association_file_fserr <- import_association_file(path_as_file, root_err)
refined_af_err <- association_file_fserr %>%
dplyr::filter(.data$ProjectID == "CLOEXP")
matrices_auto2 <- import_parallel_Vispa2Matrices_auto(
association_file = refined_af_err,
root = NULL,
quantification_type = c("fragmentEstimate", "seqCount"),
matrix_type = "annotated",
workers = 2,
patterns = "NoMate",
matching_opt = "ANY" # Same if you choose "ALL" or "OPTIONAL"
)
})
#> *** Association file import summary ***
#> ℹ For detailed report please set option 'ISAnalytics.widgets' to TRUE
#> * Parsing problems detected: TRUE
#> * Date parsing problems: TRUE
#> * Column problems detected: TRUE
#> * NAs found in important columns: TRUE
#> * File system alignment: problems detected
#> [1] "--- REPORT: FILES IMPORTED ---"
#> # A tibble: 2 x 5
#> ProjectID concatenatePoolIDSeqRun Quantification_type
#> <chr> <chr> <chr>
#> 1 CLOEXP POOL6-1 fragmentEstimate
#> 2 CLOEXP POOL6-1 seqCount
#> Files_chosen
#> <fs::path>
#> 1 /tmp/RtmpHKhRpe/fserr/VA/CLOEXP/quantification/POOL6-1/VA-CLOEXP-POOL6-1_fragmentEstimate_matrix-NoMate.no0.annotated…
#> 2 /tmp/RtmpHKhRpe/fserr/VA/CLOEXP/quantification/POOL6-1/VA-CLOEXP-POOL6-1_seqCount_matrix-NoMate.no0.annotated.tsv.xz
#> Imported
#> <lgl>
#> 1 TRUE
#> 2 TRUE
#> [1] "--- SUMMARY OF FILES CHOSEN FOR IMPORT ---"
#> # A tibble: 2 x 4
#> ProjectID concatenatePoolIDSeqRun Quantification_type
#> <chr> <chr> <chr>
#> 1 CLOEXP POOL6-1 fragmentEstimate
#> 2 CLOEXP POOL6-1 seqCount
#> Files_chosen
#> <fs::path>
#> 1 /tmp/RtmpHKhRpe/fserr/VA/CLOEXP/quantification/POOL6-1/VA-CLOEXP-POOL6-1_fragmentEstimate_matrix-NoMate.no0.annotated…
#> 2 /tmp/RtmpHKhRpe/fserr/VA/CLOEXP/quantification/POOL6-1/VA-CLOEXP-POOL6-1_seqCount_matrix-NoMate.no0.annotated.tsv.xz
#> [1] "--- INTEGRATION MATRICES FOUND REPORT ---"
#> # A tibble: 2 x 5
#> ProjectID concatenatePoolIDSeqRun Anomalies Quantification_type
#> <chr> <chr> <lgl> <chr>
#> 1 CLOEXP POOL6-1 FALSE fragmentEstimate
#> 2 CLOEXP POOL6-1 FALSE seqCount
#> Files_found
#> <fs::path>
#> 1 /tmp/RtmpHKhRpe/fserr/VA/CLOEXP/quantification/POOL6-1/VA-CLOEXP-POOL6-1_fragmentEstimate_matrix-NoMate.no0.annotated…
#> 2 /tmp/RtmpHKhRpe/fserr/VA/CLOEXP/quantification/POOL6-1/VA-CLOEXP-POOL6-1_seqCount_matrix-NoMate.no0.annotated.tsv.xz
matrices_auto
#> # A tibble: 1,476 x 8
#> chr integration_locus strand GeneName GeneStrand CompleteAmplificationID fragmentEstimate seqCount
#> <chr> <int> <chr> <chr> <chr> <fct> <dbl> <dbl>
#> 1 10 102326217 + HIF1AN + CLOEXP_POOL6_LTR10LC10_VA2020-mix10_VA… 1.00 1
#> 2 10 20775144 + MIR4675 + CLOEXP_POOL6_LTR10LC10_VA2020-mix10_VA… 1.00 1
#> 3 10 24622789 + KIAA1217 + CLOEXP_POOL6_LTR10LC10_VA2020-mix10_VA… 1.00 1
#> 4 10 30105047 + SVIL - CLOEXP_POOL6_LTR10LC10_VA2020-mix10_VA… 1.00 1
#> 5 10 32656933 - EPC1 - CLOEXP_POOL6_LTR10LC10_VA2020-mix10_VA… 1.00 1
#> 6 10 66717706 + LOC10192… - CLOEXP_POOL6_LTR10LC10_VA2020-mix10_VA… 1.00 1
#> 7 10 76391232 - ADK + CLOEXP_POOL6_LTR10LC10_VA2020-mix10_VA… 1.00 1
#> 8 10 85334019 - LOC10537… - CLOEXP_POOL6_LTR10LC10_VA2020-mix10_VA… 2.00 6
#> 9 11 107591208 + SLN - CLOEXP_POOL6_LTR10LC10_VA2020-mix10_VA… 1.00 1
#> 10 11 111744171 + FDXACB1 - CLOEXP_POOL6_LTR10LC10_VA2020-mix10_VA… 1.00 1
#> # … with 1,466 more rowsAs you can see, in the file system with issues we have more than one file for quantification type, duplicates have “NoMate” suffix in their file name. By specifying this pattern in the function, we’re only going to import those files.
As for the interactive version, you can call the function with path to the association file and root if you want to simply import everything without filtering.
The ISAnalytics package (calabrialab, 2021) was made possible thanks to:
This package was developed using biocthis.
R session information.
#> ─ Session info ───────────────────────────────────────────────────────────────────────────────────────────────────────
#> setting value
#> version R version 4.0.5 (2021-03-31)
#> os Ubuntu 18.04.5 LTS
#> system x86_64, linux-gnu
#> ui X11
#> language (EN)
#> collate C
#> ctype en_US.UTF-8
#> tz America/New_York
#> date 2021-04-08
#>
#> ─ Packages ───────────────────────────────────────────────────────────────────────────────────────────────────────────
#> package * version date lib source
#> assertthat 0.2.1 2019-03-21 [2] CRAN (R 4.0.5)
#> BiocManager 1.30.12 2021-03-28 [2] CRAN (R 4.0.5)
#> BiocParallel 1.24.1 2021-04-08 [2] Bioconductor
#> BiocStyle * 2.18.1 2021-04-08 [2] Bioconductor
#> bookdown 0.21 2020-10-13 [2] CRAN (R 4.0.5)
#> bslib 0.2.4 2021-01-25 [2] CRAN (R 4.0.5)
#> cellranger 1.1.0 2016-07-27 [2] CRAN (R 4.0.5)
#> cli 2.4.0 2021-04-05 [2] CRAN (R 4.0.5)
#> colorspace 2.0-0 2020-11-11 [2] CRAN (R 4.0.5)
#> crayon 1.4.1 2021-02-08 [2] CRAN (R 4.0.5)
#> data.table 1.14.0 2021-02-21 [2] CRAN (R 4.0.5)
#> DBI 1.1.1 2021-01-15 [2] CRAN (R 4.0.5)
#> debugme 1.1.0 2017-10-22 [2] CRAN (R 4.0.5)
#> digest 0.6.27 2020-10-24 [2] CRAN (R 4.0.5)
#> dplyr 1.0.5 2021-03-05 [2] CRAN (R 4.0.5)
#> ellipsis 0.3.1 2020-05-15 [2] CRAN (R 4.0.5)
#> evaluate 0.14 2019-05-28 [2] CRAN (R 4.0.5)
#> fansi 0.4.2 2021-01-15 [2] CRAN (R 4.0.5)
#> forcats 0.5.1 2021-01-27 [2] CRAN (R 4.0.5)
#> fs 1.5.0 2020-07-31 [2] CRAN (R 4.0.5)
#> generics 0.1.0 2020-10-31 [2] CRAN (R 4.0.5)
#> ggplot2 3.3.3 2020-12-30 [2] CRAN (R 4.0.5)
#> ggrepel 0.9.1 2021-01-15 [2] CRAN (R 4.0.5)
#> glue 1.4.2 2020-08-27 [2] CRAN (R 4.0.5)
#> gtable 0.3.0 2019-03-25 [2] CRAN (R 4.0.5)
#> highr 0.8 2019-03-20 [2] CRAN (R 4.0.5)
#> hms 1.0.0 2021-01-13 [2] CRAN (R 4.0.5)
#> htmltools 0.5.1.1 2021-01-22 [2] CRAN (R 4.0.5)
#> htmlwidgets 1.5.3 2020-12-10 [2] CRAN (R 4.0.5)
#> httr 1.4.2 2020-07-20 [2] CRAN (R 4.0.5)
#> ISAnalytics * 1.0.11 2021-04-08 [1] Bioconductor
#> jquerylib 0.1.3 2020-12-17 [2] CRAN (R 4.0.5)
#> jsonlite 1.7.2 2020-12-09 [2] CRAN (R 4.0.5)
#> knitcitations * 1.0.12 2021-01-10 [2] CRAN (R 4.0.5)
#> knitr 1.31 2021-01-27 [2] CRAN (R 4.0.5)
#> lattice 0.20-41 2020-04-02 [2] CRAN (R 4.0.5)
#> lifecycle 1.0.0 2021-02-15 [2] CRAN (R 4.0.5)
#> lubridate 1.7.10 2021-02-26 [2] CRAN (R 4.0.5)
#> magrittr * 2.0.1 2020-11-17 [2] CRAN (R 4.0.5)
#> mnormt 2.0.2 2020-09-01 [2] CRAN (R 4.0.5)
#> munsell 0.5.0 2018-06-12 [2] CRAN (R 4.0.5)
#> nlme 3.1-152 2021-02-04 [2] CRAN (R 4.0.5)
#> pillar 1.5.1 2021-03-05 [2] CRAN (R 4.0.5)
#> pkgconfig 2.0.3 2019-09-22 [2] CRAN (R 4.0.5)
#> plyr 1.8.6 2020-03-03 [2] CRAN (R 4.0.5)
#> ps 1.6.0 2021-02-28 [2] CRAN (R 4.0.5)
#> psych 2.1.3 2021-03-27 [2] CRAN (R 4.0.5)
#> purrr 0.3.4 2020-04-17 [2] CRAN (R 4.0.5)
#> R6 2.5.0 2020-10-28 [2] CRAN (R 4.0.5)
#> Rcpp 1.0.6 2021-01-15 [2] CRAN (R 4.0.5)
#> reactable 0.2.3 2020-10-04 [2] CRAN (R 4.0.5)
#> readr 1.4.0 2020-10-05 [2] CRAN (R 4.0.5)
#> readxl 1.3.1 2019-03-13 [2] CRAN (R 4.0.5)
#> RefManageR 1.3.0 2020-11-13 [2] CRAN (R 4.0.5)
#> rlang 0.4.10 2020-12-30 [2] CRAN (R 4.0.5)
#> rmarkdown 2.7 2021-02-19 [2] CRAN (R 4.0.5)
#> rstudioapi 0.13 2020-11-12 [2] CRAN (R 4.0.5)
#> sass 0.3.1 2021-01-24 [2] CRAN (R 4.0.5)
#> scales 1.1.1 2020-05-11 [2] CRAN (R 4.0.5)
#> sessioninfo * 1.1.1 2018-11-05 [2] CRAN (R 4.0.5)
#> stringi 1.5.3 2020-09-09 [2] CRAN (R 4.0.5)
#> stringr 1.4.0 2019-02-10 [2] CRAN (R 4.0.5)
#> tibble 3.1.0 2021-02-25 [2] CRAN (R 4.0.5)
#> tidyr 1.1.3 2021-03-03 [2] CRAN (R 4.0.5)
#> tidyselect 1.1.0 2020-05-11 [2] CRAN (R 4.0.5)
#> tmvnsim 1.0-2 2016-12-15 [2] CRAN (R 4.0.5)
#> upsetjs 1.9.0 2021-02-15 [2] CRAN (R 4.0.5)
#> utf8 1.2.1 2021-03-12 [2] CRAN (R 4.0.5)
#> vctrs 0.3.7 2021-03-29 [2] CRAN (R 4.0.5)
#> withr 2.4.1 2021-01-26 [2] CRAN (R 4.0.5)
#> xfun 0.22 2021-03-11 [2] CRAN (R 4.0.5)
#> xml2 1.3.2 2020-04-23 [2] CRAN (R 4.0.5)
#> yaml 2.2.1 2020-02-01 [2] CRAN (R 4.0.5)
#> zip 2.1.1 2020-08-27 [2] CRAN (R 4.0.5)
#>
#> [1] /tmp/RtmpuXsGTR/Rinst7cb42c7ac9b7
#> [2] /home/biocbuild/bbs-3.12-bioc/R/libraryThis vignette was generated using BiocStyle (Oleś, Morgan, and Huber, 2021) with knitr (Xie, 2021) and rmarkdown (Allaire, Xie, McPherson, Luraschi, et al., 2021) running behind the scenes.
Citations made with knitcitations (Boettiger, 2021).
[1] J. Allaire, Y. Xie, J. McPherson, J. Luraschi, et al. rmarkdown: Dynamic Documents for R. R package version 2.7. 2021. <URL: https://github.com/rstudio/rmarkdown>.
[2] C. Boettiger. knitcitations: Citations for ‘Knitr’ Markdown Files. R package version 1.0.12. 2021. <URL: https://CRAN.R-project.org/package=knitcitations>.
[3] G. Csárdi, R. core, H. Wickham, W. Chang, et al. sessioninfo: R Session Information. R package version 1.1.1. 2018. <URL: https://CRAN.R-project.org/package=sessioninfo>.
[4] A. Oleś, M. Morgan, and W. Huber. BiocStyle: Standard styles for vignettes and other Bioconductor documents. R package version 2.18.1. 2021. <URL: https://github.com/Bioconductor/BiocStyle>.
[5] R Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing. Vienna, Austria, 2021. <URL: https://www.R-project.org/>.
[6] H. Wickham. “testthat: Get Started with Testing”. In: The R Journal 3 (2011), pp. 5-10. <URL: https://journal.r-project.org/archive/2011-1/RJournal_2011-1_Wickham.pdf>.
[7] Y. Xie. knitr: A General-Purpose Package for Dynamic Report Generation in R. R package version 1.31. 2021. <URL: https://yihui.org/knitr/>.
[8] calabrialab. Analyze gene therapy vector insertion sites data identified from genomics next generation sequencing reads for clonal tracking studies. https://github.com/calabrialab/ISAnalytics - R package version 1.0.11. 2021. DOI: 10.18129/B9.bioc.ISAnalytics. <URL: http://www.bioconductor.org/packages/ISAnalytics>.