Skip to content

Customized Deployment

Deployments of the 'Nextflow on AWS Batch' solution are based on nested CloudFormation templates, and on artifacts comprising scripts, software packages, and configuration files. The templates and artifacts are stored in S3 buckets, and their S3 URLs are used when launching the top-level template and as parameters to that template's deployment.

VPC

The quick start link deploys the AWS VPC Quickstart, which creates a VPC with up to 4 Availability Zones, each with a public subnet and a private subnet with NAT Gateway access to the Internet.

Genomics Workflow Core

This quick start link deploys the CloudFormation template gwfcore-root.template.yaml for the Genomics Workflow Core (GWFCore) from the Genomics Workflows on AWS solution. This template launches a number of nested templates, as shown below:

  • Root Stack gwfcore-root - Top level template for Genomics Workflow Core
    • S3 Stack gwfcore-s3 - S3 bucket (new or existing) for storing analysis results
    • IAM Stack gwfcore-iam - Creates IAM roles to use with AWS Batch scalable genomics workflow environment
    • Code Stack gwfcore-code - Creates AWS CodeCommit repos and CodeBuild projects for Genomics Workflows Core assets and artifacts
    • Launch Template Stack gwfcore-launch-template - Creates an EC2 Launch Template for AWS Batch based genomics workflows
    • Batch Stack gwfcore-batch - Deploys resource for a AWS Batch environment that is suitable for genomics, including default and high-priority JobQueues

Root Stack

The quick start solution links to the CloudFormation console, where the 'Amazon S3 URL' field is prefilled with the S3 URL of a copy of the root stack template, hosted in the public S3 bucket aws-genomics-workflows.

custom-deploy-0

To use a customized root stack, upload your modified stack template to an S3 bucket (see Building a Custom Distribution), and specify that template's URL in 'Amazon S3 URL'.

Artifacts and Nested Stacks

The subsequent screen, 'Specify Stack Details', allows for customization of the deployed resources in the 'Distribution Configuration' section.

custom-deploy-1

  • Artifact S3 Bucket Name and Artifact S3 Prefix define the location of the artifacts uploaded prior to this deployment. By default, pre-prepared artifacts are stored in the aws-genomics-workflows bucket.
  • Template Root URL defines the bucket and prefix used to store nested templates, called by the root template.

To use your own modified artifacts or nested templates, build and upload as described in Building a Custom Distribution, and specify the bucket and prefix in the fields above.

Workflow Orchestrators

Nextflow

This quick start deploys the Nextflow template nextflow-resources.template.yaml, which launches one nested stack:

  • Root Stack nextflow-resources - Creates resources specific to running Nextflow on AWS
    • Container Build Stack container-build - Creates resources for building a Docker container image using CodeBuild, storing the image in ECR, and optionally creating a corresponding Batch Job Definition

The nextflow root stack is specified in the same way as the GWFCore root stack, above, and a location for a modified root stack may be specified as with the Core stack.

The subsequent 'Specify Stack Details' screen has fields allowing the customization of the Nextflow deployment.

nextflow-0

  • S3NextflowPrefix, S3LogsDirPrefix, and S3WorkDirPrefix specify the path within the GWFCore bucket in which to store per-run data and log files.
  • TemplateRootUrl specifies the path to the nested templates called by the Nextflow root template, as with the GWFCore root stack.