Amazon Chime SDK for Telemedicine Solution

Introduction

To meet the needs of the emerging wave of digital healthcare, our global healthcare customers have been building telehealth solutions that comply with healthcare regulations, such as HIPAA and GDPR, in order to deliver higher quality service and better patient outcomes. For example, telehealth visits have grown to over 30 times higher than before the COVID pandemic. In addition to ambulatory care visits, behavioral healthcare providers, such as Talkspace, can also innovate securely on  AWS in order to make online mental health consulting accessible from anywhere, thereby revolutionizing mental health care for the digital age.

Amazon Chime SDK has been a critical service powering these virtual patient care and communications by global healthcare companies. And now, Amazon Chime SDK has released two new features: media capture pipelines and live transcription, which  can be utilized to enhance Amazon Chime SDK meetings. We have previously demonstrated how to create a media capture pipeline in order to securely deliver meeting recordings to an Amazon S3 bucket. This post will further extend the Amazon Chime SDK chat application with live transcription and media capture functions.

Amazon Chime SDK chat messages are persistent and will be processed by Amazon Kinesis Data Stream. The chat serves as a waiting room scenario in this telemedicine solution. The chat messages can trigger a Lambda function that invokes the Amazon Lex bot to greet the attendee and conduct basic interactions before the healthcare provider arrives, such as self-check-in and future appointment scheduling. The live transcriptions of Amazon Chime meetings entail medical transcription analysis powered by Amazon Comprehend Medical. Besides the live transcription feature of Amazon Chime SDK, Amazon Transcribe and Transcribe Medical can be integrated with other telehealth platforms, such as automatic Zoom transcriptions.

Telemedicine Solution Deployment

Below is the overview diagram:

Here are step-by-step instructions for how to deploy this solution to your AWS account in the us-east-1 region.

1. Create an Amazon Lex V1 Bot by importing the zip file downloaded here. Build and publish the chat bot with alias as ‘demo’, and note the bot name and version, which will be utilized by the backend message moderator Lambda function.
2. Create an Amazon Cloud9 environment with m5.large (8GiB RAM) instance type (recommended). The other parameters, such as environment type, platform, and networking, can be set by default values.
3. Open the Cloud9 IDE console and install the missing dependency in the terminal:
   npm install -g jq
4.  Check out this GitHub code repository, go to televisit-demo folder, and run the deploy.sh script:

git clone https://github.com/aws-samples/amazon-chime-sdk.git

cd amazon-chime-sdk/apps/televisit-demo

./deploy.sh

If you see the prompts during the frontend CloudFormation deployment, then you just need to input the Amazon S3 bucket name that hosts the HTML and bundled Javascript file in addition to default values for other parameters. Alternatively, you can deploy the backend and frontend manually by following the READMEs in each folder.

In the backend deployment, two CloudFormation parameters in the template, namely LexBotName and LexBotAlias, should take corresponding values from step 1. The backend template will create several API Gateway endpoints handled by Lambda functions. The frontend stack will configure and build the React web application. It will also create a private S3 bucket as the origin for a CloudFront distribution, and it will upload the bundled Javascript file and HTML file to that S3 bucket. Once the deployment is finished, you should be able to access the telemedicine webpage through the CloudFront domain:

Figure 2. CloudFormation frontend stack outputs

Similar to an existing Amazon Chime SDK chat meeting application, you must register a new user on this web application, and then confirm the new user registration on AWS console for Amazon Cognito User Pool:

Figure 3. Confirm new user registration on Cognito User

Once confirmed, you can sign in to the telemedicine application through that user. After you log into the webportal, you can add a new channel to interact with Chatbot via Amazon Chime SDK chat, and then start a new meeting with live transcription and media capture functions. The Github code repository has some screenshots for the demo.

Note that we have configured the Amazon Chime SDK live transcription by using the Amazon Transcribe Primary Care use case in this demo, as you can see in the StartTranscriptionApiFunction Lambda function:

const response = await chime.startMeetingTranscription({
      "MeetingId": meetingId,
      "TranscriptionConfiguration": {
        "EngineTranscribeMedicalSettings": {
          "Region": "us-east-1",
          "LanguageCode": "en-US",
          "Specialty": "PRIMARYCARE",
          "Type": "CONVERSATION"
        }
      }
    }).promise();

Amazon Transcribe Medical supports primary care and specialty care areas, such as cardiology, neurology, obstetrics,gynecology, pediatrics, oncology, radiology, and urology, and only for English language. Alternatively, you can utilize Amazon Transcribe for other languages.

response = await chime.startMeetingTranscription({
        "MeetingId": meetingId,
        "TranscriptionConfiguration": {
          "EngineTranscribeSettings": {
            "Region": "us-east-1",
            "LanguageCode": <LanguageCode>
          }
        }
      }).promise();

The S3 bucket utilized for capturing meeting recordings have public access blocked and server-side encryption enabled. In addition to streaming live transcription, you can also start batch transcription jobs by using the captured media files. To do so, follow this blog post to concatenate the chunked audio files by using the Lambda function with FFmpeg library layer.

Clean up

• Delete Amazon Lex bot and Cloud9 instance on AWS console.
• Delete both the frontend and backend CloudFormation stack on the AWS console or via AWS CLI. If you see errors to delete the S3 bucket or Custom Resource, then empty the S3 bucket and try to delete the stack again.
• Finally, you will also need to delete the Amazon Chime app instance created during backend CloudFormation deployment by using the following AWS CLI command:

aws chime delete-app-instance --app-instance-arn <appInstanceArn FROM STEP 4>

Conclusion

This post has demonstrated how to create a web application for virtual telehealth visits between patients and clinicians. The new features of Amazon Chime SDK including live transcription and media capture, make it easier to automate the clinical notes analysis and generation. Further intelligent SOAP clinical notes automation will let clinicians focus on patient conversation as well as reduce clinician fatigue from writing and editing clinical summaries. Please note that this demo intends to show some of the possibilities when integrating Amazon Chime SDK and other managed AI services. When you build your telemedicine solutions on AWS, you should include additional security services in a production deployment, including Amazon Macie for detecting sensitive data in S3 buckets and AWS WAF for API Gateway.

To find the latest developments to this solution, check out Amazon Chime SDK applications GitHub repository.