Australian Bureau of Statistics runs 2021 Census on the AWS Cloud

Earlier this year, the Australian Bureau of Statistics (ABS) ran the Australian Census, the agency’s most significant workload, on Amazon Web Services (AWS). This is the first time the Census has been run on the cloud.

What is the Australian Census?

Every five years, ABS counts every person and household in Australia. The Census is the most comprehensive snapshot of the country, and includes around 10 million households and over 25 million people. Census data provides information about the economic, social, and cultural make-up of the country.

Question topics include income and work, health, education, and cultural diversity. The statistics are used by governments, business, and community organisations to make important decisions about transport, schools, health care, roads, and buildings, and to help plan local services for individuals, families, and communities.

For the 2021 Census, ABS engaged PwC Australia (PwC) to create a robust Census Digital Service (CDS), hosted on AWS, that would be able to handle the web traffic generated by millions of users. This Census was different, with the COVID-19 pandemic causing lockdowns across the country. It was important to have a digital option that was accessible and reliable.

In preparation for the 2021 Census, the CDS was thoroughly tested for user demand at more than 2,000 times the expected peak workload. The Census form was open from July 28, 2021 until late September 2021. Over that period the ABS received over 9.6 million forms in total, with over 7.6 million submitted online.

The CDS scaled to serve forms to over 2.5 million people in a 24-hour period on Census Day. At peak time, over 142 forms were submitted per second.

Architectural overview

ABS wanted a secure, efficient, and scalable system built on a proven cloud provider. In 2017, they used a serverless architecture on AWS to deliver the online component of the Australian Marriage Law Postal Survey. This experience demonstrated to ABS the benefits of scalability, reliability, and cost effectiveness that serverless provides.

The Census website is hosted on Amazon Simple Storage Service (Amazon S3) and uses Amazon CloudFront as a content delivery network and AWS WAF and AWS Shield Advanced for security and distributed denial of service (DDoS) protection. The Census form is submitted via a call to Amazon API Gateway, which validates the message format before invoking an AWS Lambda function, which then inserts the data into an Amazon DynamoDB table. DynamoDB Streams and Amazon Simple Queue Service (Amazon SQS) are used to transport the data to ABS’s processing environment. These services, which are highly scalable and serverless, form the core of the CDS.

The ABS assessed the CDS to the highest level of security compliance possible for public cloud-based systems, achieving PROTECTED level certification under the Australian government’s Information Security Registered Assessors Program (IRAP) certification program.

The solution delivered by PwC used AWS services that had been assessed to PROTECTED level, and used encryption throughout the system to provide additional assurance that the privacy of Australians’ data remained secure.

Before the Census went live, the ABS wanted to verify that AWS could effectively protect the CDS against DDoS attacks. ABS, AWS, and PwC also conducted extensive DDoS tests, which were equivalent to being in the top 1% of DDoS attacks observed by AWS globally. The security measures put in place successfully blocked traffic from more than 130,000 malicious IP addresses during the Census period.

Steve Hamilton, chief information officer at ABS said, “The Census Digital Service achieved high levels of security, reliability, and scale thanks to the serverless architecture built on AWS. The most important benefit of working with AWS is that ABS doesn’t have to worry about building and operating the underlying infrastructure, and ABS can focus on delivering a simple and easy experience for the people of Australia.”

Census support systems

In addition to the CDS built by PwC, the ABS delivered three more systems on AWS to support Census operations.

Operational Insights, a serverless data lake built by AWS Professional Services, AWS Partner Shine Solutions, and ABS used services including Amazon S3, AWS Glue, AWS Database Migration Service (AWS DMS), and Amazon Redshift to provide insights to ABS staff on Census operations. This included data such as the number of submitted forms, the progress of Census field workers, and a breakdown of types of queries from the public. In the past, this information was updated every 24 hours, but with Operational Insights the data was available in near real-time.

The Paper Form Request Service, an automated call centre built by AWS Professional Services and ABS using Amazon Connect, allows people to request paper forms. People who called the hotline number were prompted by an automated agent to enter a few details like their Census ID number and their postcode to be verified. This meant close to 380,000 people did not have to speak to staff at the Census Contact Centre to request a paper form, and collectively saved thousands of hours of wait time in the Contact Centre.

The natural language processing system, built by AWS partner Shine Solutions, AWS Professional Services, and ABS, used Amazon Comprehend to analyse feedback from customer forms. The analysis included most frequent words (like “questions,” “good,” “easy,” “quick,” and “simple”) and the sentiment of feedback.

Future of ABS on AWS

“In this day and age, people expect a fast and simple online experience when using government services,” said Bindi Kindermann, program manager for 2021 Census Field Operations at ABS. “Using AWS, the ABS was able to scale and securely deliver services to people across the country, making it possible for them to quickly and easily participate in this nation-wide event.”

With the success of the 2021 Census, the ABS is continuing to expand its use of AWS into broader areas of its business, making use of the security, reliability, and scalability of the cloud. Get more information on the Australian Census and learn more about how Australian public sector customers are using AWS.

Subscribe to the AWS Public Sector Blog newsletter to get the latest in AWS tools, solutions, and innovations from the public sector delivered to your inbox, or contact us.

The AWS Public Sector Blog needs your help. Please take a few minutes to share insights regarding your experience with the AWS Public Sector Blog in this survey, and we’ll use feedback from the survey to create more content aligned with the preferences of our readers.