CSIRO Uses Serverless Architecture to Query 100x More Genomic Variants on AWS

CSIRO has developed Serverless Beacon (sBeacon), a serverless architecture for querying genomic variant data without copying large genomic datasets into a central database. The system uses Amazon S3, AWS Lambda, Amazon Athena, and Amazon DynamoDB to provide an implementation of the Global Alliance for Genomics and Health (GA4GH) Beacon standard, supporting genomic queries across large cohorts while keeping the underlying data in its original storage.

The architecture separates genomic data from the metadata needed to search it. Variant Call Format (VCF) files remain in Amazon S3, while metadata such as disease status, age, and location is indexed separately. During onboarding, sBeacon creates indexes and Athena tables for the metadata, while ontology terms are handled through CSIRO's Ontoserver to support queries combining genomic and phenotypic information.

sBeacon data onboarding architecture (Source: AWS Blog Post)

For variant queries, API Gateway routes requests to Lambda functions that identify relevant datasets and genomic regions. The Variant Querying Module uses bcftools and its htslib library to retrieve specific portions of VCF files through HTTP byte range requests. This allows the system to read the required genomic region without transferring the complete file. Results are aggregated and returned through the Beacon API.

Anuradha Wickramarachchi, Solutions Architect for Bioinformatics Products at CSIRO, highlighted data compression as an important consideration in the design,

One key consideration we have exercised carefully is having the data in the most compressed form. And querying while they are in the compressed form is greatly beneficial.

CSIRO evaluated the architecture using chromosome 1 from the 1000 Genomes Project, containing 2,504 samples in a 1.1 GB compressed VCF file. The dataset was onboarded in 18 seconds. A query covering 10,000 base pairs averaged 1.52 seconds at an estimated compute cost of USD 0.00013. Latency remained broadly consistent as matching variants increased from four to 400.

Lambda distributes genomic queries while S3 serves only the required compressed file ranges (Source: AWS Blog Post)

The design builds on earlier sBeacon research published in Nature Biotechnology, which reported support for up to 40 million individuals and 3 billion genomic locations. The original Beacon work described the federated model, while the GA4GH protocol was designed to operate across distributed data sources without prescribing an underlying storage technology. The original Beacon work described the federated model as a way for organizations to retain control of their genomic data while exposing standardized query interfaces. The 2026 implementation extends that model with cloud services that provide elastic query execution while retaining genomic files in object storage.