Note: the most recent version of this tutorial can be found here.

Note: if you use systemPipeR in published research, please cite: Backman, T.W.H and Girke, T. (2016). systemPipeR: NGS Workflow and Report Generation Environment. BMC Bioinformatics, 17: 388. 10.1186/s12859-016-1241-0.

1 Workflow templates

The intended way of running systemPipeR workflows is via *.Rmd files, which can be executed either line-wise in interactive mode or with a single command from R or the command-line. This way comprehensive and reproducible analysis reports can be generated in PDF or HTML format in a fully automated manner by making use of the highly functional reporting utilities available for R.

Templates for setting up custom project reports are provided as *.Rmd files by the helper package systemPipeRdata and in the vignettes subdirectory of systemPipeR. The corresponding HTML of these report templates are available here: systemPipeRNAseq, systemPipeRIBOseq, systemPipeChIPseq and systemPipeVARseq. To work with *.Rmd files efficiently, basic knowledge of knitr and Latex or R Markdown v2 is required.

1.1 Directory Structure

Figure 1: systemPipeR’s preconfigured directory structure

The working environment of the sample data loaded in the previous step contains the following pre-configured directory structure. Directory names are indicated in green. Users can change this structure as needed, but need to adjust the code in their workflows accordingly.

workflow/ (e.g. rnaseq/)
- This is the root directory of the R session running the workflow.
- Run script ( *.Rmd) and sample annotation (targets.txt) files are located here.
- Note, this directory can have any name (e.g. rnaseq, varseq). Changing its name does not require any modifications in the run script(s).
- Important subdirectories:
  - param/
    - Stores non-CWL parameter files such as: *.param, *.tmpl and *.run.sh. These files are only required for backwards compatibility to run old workflows using the previous custom command-line interface.
    - param/cwl/: This subdirectory stores all the CWL parameter files. To organize workflows, each can have its own subdirectory, where all CWL param and input.yml files need to be in the same subdirectory.
  - data/
    - FASTQ files
    - FASTA file of reference (e.g. reference genome)
    - Annotation files
    - etc.
  - results/
    - Analysis results are usually written to this directory, including: alignment, variant and peak files (BAM, VCF, BED); tabular result files; and image/plot files
    - Note, the user has the option to organize results files for a given sample and analysis step in a separate subdirectory.

The following parameter files are included in each workflow template:

targets.txt: initial one provided by user; downstream targets_*.txt files are generated automatically
*.param/cwl: defines parameter for input/output file operations, e.g.:
- hisat2-se/hisat2-mapping-se.cwl
- hisat2-se/hisat2-mapping-se.yml
*_run.sh: optional bash scripts
Configuration files for computer cluster environments (skip on single machines):
- .batchtools.conf.R: defines the type of scheduler for batchtools pointing to template file of cluster, and located in user’s home directory
- *.tmpl: specifies parameters of scheduler used by a system, e.g. Torque, SGE, Slurm, etc.

2 RNA-Seq Workflow

This workflow demonstrates how to use various utilities for building and running automated end-to-end analysis workflows for RNA-Seq data.

The full workflow can be found here: HTML, .Rmd, and .R.

2.1 Loading package and workflow template

Load the RNA-Seq sample workflow into your current working directory.

library(systemPipeRdata)
genWorkenvir(workflow = "rnaseq")
setwd("rnaseq")

2.2 Create the workflow

This template provides some common steps for a RNAseq workflow. One can add, remove, modify workflow steps by operating on the sal object.

sal <- SPRproject()
sal <- importWF(sal, file_path = "systemPipeRNAseq.Rmd", verbose = FALSE)

Workflow includes following steps:

Read preprocessing
- Quality filtering (trimming)
- FASTQ quality report
Alignments: HISAT2 (or any other RNA-Seq aligner)
Alignment stats
Read counting
Sample-wise correlation analysis
Analysis of differentially expressed genes (DEGs)
GO term enrichment analysis
Gene-wise clustering

2.3 Run workflow

sal <- runWF(sal)

2.4 Workflow visualization

plotWF(sal)

2.5 Report generation

sal <- renderReport(sal)
sal <- renderLogs(sal)

3 ChIP-Seq Workflow

This workflow demonstrates how to use various utilities for building and running automated end-to-end analysis workflows for ChIP-Seq data.

The full workflow can be found here: HTML, .Rmd, and .R.

3.1 Loading package and workflow template

Load the ChIP-Seq sample workflow into your current working directory.

library(systemPipeRdata)
genWorkenvir(workflow = "chipseq")
setwd("chipseq")

Workflow includes following steps:

Read preprocessing
- Quality filtering (trimming)
- FASTQ quality report
Alignments: Bowtie2 or rsubread
Alignment stats
Peak calling: MACS2
Peak annotation with genomic context
Differential binding analysis
GO term enrichment analysis
Motif analysis

3.2 Create the workflow

This template provides some common steps for a ChIPseq workflow. One can add, remove, modify workflow steps by operating on the sal object.

sal <- SPRproject()
sal <- importWF(sal, file_path = "systemPipeChIPseq.Rmd", verbose = FALSE)

3.3 Run workflow

sal <- runWF(sal)

3.4 Workflow visualization

plotWF(sal)

3.5 Report generation

sal <- renderReport(sal)
sal <- renderLogs(sal)

4 VAR-Seq Workflow

This workflow demonstrates how to use various utilities for building and running automated end-to-end analysis workflows for VAR-Seq data.

The full workflow can be found here: HTML, .Rmd, and .R.

4.1 Loading package and workflow template

Load the VAR-Seq sample workflow into your current working directory.

library(systemPipeRdata)
genWorkenvir(workflow = "varseq")
setwd("varseq")

Workflow includes following steps:

Read preprocessing
- Quality filtering (trimming)
- FASTQ quality report
Alignments: gsnap, bwa
Variant calling: VariantTools, GATK, BCFtools
Variant filtering: VariantTools and VariantAnnotation
Variant annotation: VariantAnnotation
Combine results from many samples
Summary statistics of samples

4.2 Create the workflow

This template provides some common steps for a VARseq workflow. One can add, remove, modify workflow steps by operating on the sal object.

sal <- SPRproject()
sal <- importWF(sal, file_path = "systemPipeVARseq.Rmd", verbose = FALSE)

4.3 Run workflow

sal <- runWF(sal)

4.4 Workflow visualization

plotWF(sal)

4.5 Report generation

sal <- renderReport(sal)
sal <- renderLogs(sal)

5 Ribo-Seq Workflow

This workflow demonstrates how to use various utilities for building and running automated end-to-end analysis workflows for RIBO-Seq data.

The full workflow can be found here: HTML, .Rmd, and .R.

5.1 Loading package and workflow template

Load the RIBO-Seq sample workflow into your current working directory.

library(systemPipeRdata)
genWorkenvir(workflow = "riboseq")
setwd("riboseq")

Workflow includes following steps:

Read preprocessing
- Adaptor trimming and quality filtering
- FASTQ quality report
Alignments: HISAT2 (or any other RNA-Seq aligner)
Alignment stats
Compute read distribution across genomic features
Adding custom features to workflow (e.g. uORFs)
Genomic read coverage along transcripts
Read counting
Sample-wise correlation analysis
Analysis of differentially expressed genes (DEGs)
GO term enrichment analysis
Gene-wise clustering
Differential ribosome binding (translational efficiency)

This template provides some common steps for a RIBOseq workflow. One can add, remove, modify workflow steps by operating on the sal object.

sal <- SPRproject()
sal <- importWF(sal, file_path = "systemPipeRIBOseq.Rmd", verbose = FALSE)

5.2 Run workflow

sal <- runWF(sal)

5.3 Workflow visualization

plotWF(sal)

5.4 Report generation

sal <- renderReport(sal)
sal <- renderLogs(sal)

6 Version information

sessionInfo()

## R version 4.2.0 RC (2022-04-19 r82224)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Ubuntu 20.04.4 LTS
## 
## Matrix products: default
## BLAS:   /home/biocbuild/bbs-3.15-bioc/R/lib/libRblas.so
## LAPACK: /home/biocbuild/bbs-3.15-bioc/R/lib/libRlapack.so
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
##  [3] LC_TIME=en_GB              LC_COLLATE=C              
##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
## 
## attached base packages:
## [1] stats4    stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
##  [1] magrittr_2.0.3              systemPipeR_2.2.2          
##  [3] ShortRead_1.54.0            GenomicAlignments_1.32.0   
##  [5] SummarizedExperiment_1.26.1 Biobase_2.56.0             
##  [7] MatrixGenerics_1.8.0        matrixStats_0.62.0         
##  [9] BiocParallel_1.30.0         Rsamtools_2.12.0           
## [11] Biostrings_2.64.0           XVector_0.36.0             
## [13] GenomicRanges_1.48.0        GenomeInfoDb_1.32.1        
## [15] IRanges_2.30.0              S4Vectors_0.34.0           
## [17] BiocGenerics_0.42.0         BiocStyle_2.24.0           
## 
## loaded via a namespace (and not attached):
##  [1] bitops_1.0-7           webshot_0.5.3          RColorBrewer_1.1-3    
##  [4] httr_1.4.2             tools_4.2.0            bslib_0.3.1           
##  [7] utf8_1.2.2             R6_2.5.1               DT_0.22               
## [10] DBI_1.1.2              colorspace_2.0-3       tidyselect_1.1.2      
## [13] compiler_4.2.0         cli_3.3.0              rvest_1.0.2           
## [16] formatR_1.12           xml2_1.3.3             DelayedArray_0.22.0   
## [19] labeling_0.4.2         bookdown_0.26          sass_0.4.1            
## [22] scales_1.2.0           systemPipeRdata_2.0.1  systemfonts_1.0.4     
## [25] stringr_1.4.0          digest_0.6.29          rmarkdown_2.14        
## [28] svglite_2.1.0          jpeg_0.1-9             pkgconfig_2.0.3       
## [31] htmltools_0.5.2        fastmap_1.1.0          highr_0.9             
## [34] htmlwidgets_1.5.4      rlang_1.0.2            rstudioapi_0.13       
## [37] jquerylib_0.1.4        generics_0.1.2         farver_2.1.0          
## [40] hwriter_1.3.2.1        jsonlite_1.8.0         crosstalk_1.2.0       
## [43] dplyr_1.0.9            RCurl_1.98-1.6         kableExtra_1.3.4      
## [46] GenomeInfoDbData_1.2.8 Matrix_1.4-1           Rcpp_1.0.8.3          
## [49] munsell_0.5.0          fansi_1.0.3            lifecycle_1.0.1       
## [52] stringi_1.7.6          yaml_2.3.5             zlibbioc_1.42.0       
## [55] grid_4.2.0             parallel_4.2.0         crayon_1.5.1          
## [58] lattice_0.20-45        magick_2.7.3           knitr_1.39            
## [61] pillar_1.7.0           codetools_0.2-18       glue_1.6.2            
## [64] evaluate_0.15          latticeExtra_0.6-29    remotes_2.4.2         
## [67] BiocManager_1.30.17    png_0.1-7              vctrs_0.4.1           
## [70] gtable_0.3.0           purrr_0.3.4            assertthat_0.2.1      
## [73] ggplot2_3.3.5          xfun_0.30              viridisLite_0.4.0     
## [76] tibble_3.1.6           ellipsis_0.3.2

7 Funding

This project is funded by NSF award ABI-1661152.

systemPipeR: Workflows collection

Last update: 01 May, 2022

Package

1 Workflow templates

1.1 Directory Structure

2 RNA-Seq Workflow

2.1 Loading package and workflow template

2.2 Create the workflow

2.3 Run workflow

2.4 Workflow visualization

2.5 Report generation

3 ChIP-Seq Workflow

3.1 Loading package and workflow template

3.2 Create the workflow

3.3 Run workflow

3.4 Workflow visualization

3.5 Report generation

4 VAR-Seq Workflow

4.1 Loading package and workflow template

4.2 Create the workflow

4.3 Run workflow

4.4 Workflow visualization

4.5 Report generation

5 Ribo-Seq Workflow

5.1 Loading package and workflow template

5.2 Run workflow

5.3 Workflow visualization

5.4 Report generation

6 Version information

7 Funding

8 References