How can I mount an Amazon EFS volume to AWS Batch in a managed compute environment?

4 minute read
0

I want to mount an Amazon Elastic File System (Amazon EFS) volume in AWS Batch. How can I do that in a managed compute environment without creating custom Amazon Machine Images (AMIs)?

Short description

Note: AWS Batch now supports mounting EFS volumes directly to the containers that are created, as part of the job definition. This is a simpler method than the resolution noted in this article. For more information, see Specifying an Amazon EFS file system in your job definition and the efsVolumeConfiguration parameter in Container properties.

Use a launch template to mount an Amazon EFS volume to an EC2 instance and then to a container. This also allows you to mount the EFS volume to containers through AWS Batch without creating a custom AMI.

Important: When you create an Amazon EFS volume, use the same Amazon Virtual Private Cloud (Amazon VPC) and subnets that are assigned to your compute environment.

Resolution

Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version .

1.    Create an Amazon EFS file system.

2.    Note the file system ID (for example: fs-12345678). You need the file system ID to run your launch template.

3.    Create a launch template that includes a user data section and uses the MIME multi-part file format. For more information, see Mime Multi Part Archive on the Cloud-init website.

Example MIME multi-part file

Note: The following example MIME multi-part file configures the compute resource to install the amazon-efs-utils package. Then, the file mounts an existing Amazon EFS file system at /mnt/efs:.

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

--==MYBOUNDARY==
Content-Type: text/cloud-config; charset="us-ascii"

packages:
- amazon-efs-utils

runcmd:
- file_system_id_01=fs-12345678
- efs_directory=/mnt/efs

- mkdir -p ${efs_directory}
- echo "${file_system_id_01}:/ ${efs_directory} efs tls,_netdev" >> /etc/fstab
- mount -a -t efs defaults

--==MYBOUNDARY==--

Important: Replace fs-12345678 with your file system ID.

4.    Create a file called mount-efs.json.

Note: Adjust the size of your volume based on your needs.

Example Amazon Linux 2 launch template

{
  "LaunchTemplateName": "user-data",
  "LaunchTemplateData": {
    "BlockDeviceMappings": [
      {
        "Ebs": {
          "DeleteOnTermination": true,
          "VolumeSize": 30,
          "VolumeType": "gp2"
        },
        "DeviceName": "/dev/xvda"
      }
    ],
    "UserData": "TUlNRS1WZXJzaW9uOiAxLjAKQ29udGVudC1UeXBlOiBtdWx0aXBhcnQvbWl4ZWQ7IGJvdW5kYXJ5PSI9PU1ZQk9VTkRBUlk9PSIKCi0tPT1NWUJPVU5EQVJZPT0KQ29udGVudC1UeXBlOiB0ZXh0L2Nsb3VkLWNvbmZpZzsgY2hhcnNldD0idXMtYXNjaWkiCgpwYWNrYWdlczoKLSBhbWF6b24tZWZzLXV0aWxzCgpydW5jbWQ6Ci0gZmlsZV9zeXN0ZW1faWRfMDE9ZnMtODc0MTc4MDYgICAgIAotIGVmc19kaXJlY3Rvcnk9L21udC9lZnMKCi0gbWtkaXIgLXAgJHtlZnNfZGlyZWN0b3J5fQotIGVjaG8gIiR7ZmlsZV9zeXN0ZW1faWRfMDF9Oi8gJHtlZnNfZGlyZWN0b3J5fSBlZnMgdGxzLF9uZXRkZXYiID4+IC9ldGMvZnN0YWIKLSBtb3VudCAtYSAtdCBlZnMgZGVmYXVsdHMKCi0tPT1NWUJPVU5EQVJZPT0tLQ=="
  }
}

Example Amazon Linux 1 launch template

{
  "LaunchTemplateName": "userdata",
  "LaunchTemplateData": {
    "BlockDeviceMappings": [
      {
        "Ebs": {
          "DeleteOnTermination": true,
          "VolumeSize": 8,
          "VolumeType": "gp2"
        },
        "DeviceName": "/dev/xvda"
      },
      {
        "Ebs": {
          "DeleteOnTermination": true,
          "VolumeSize": 22,
          "VolumeType": "gp2"
        },
        "DeviceName": "/dev/xvdcz"
      }
    ],
    "UserData": "TUlNRS1WZXJzaW9uOiAxLjAKQ29udGVudC1UeXBlOiBtdWx0aXBhcnQvbWl4ZWQ7IGJvdW5kYXJ5PSI9PU1ZQk9VTkRBUlk9PSIKCi0tPT1NWUJPVU5EQVJZPT0KQ29udGVudC1UeXBlOiB0ZXh0L2Nsb3VkLWNvbmZpZzsgY2hhcnNldD0idXMtYXNjaWkiCgpwYWNrYWdlczoKLSBhbWF6b24tZWZzLXV0aWxzCgpydW5jbWQ6Ci0gZmlsZV9zeXN0ZW1faWRfMDE9ZnMtODc0MTc4MDYgICAgIAotIGVmc19kaXJlY3Rvcnk9L21udC9lZnMKCi0gbWtkaXIgLXAgJHtlZnNfZGlyZWN0b3J5fQotIGVjaG8gIiR7ZmlsZV9zeXN0ZW1faWRfMDF9Oi8gJHtlZnNfZGlyZWN0b3J5fSBlZnMgdGxzLF9uZXRkZXYiID4+IC9ldGMvZnN0YWIKLSBtb3VudCAtYSAtdCBlZnMgZGVmYXVsdHMKCi0tPT1NWUJPVU5EQVJZPT0tLQ=="
  }
}

Important: If you add user data to a launch template in the Amazon Elastic Compute Cloud (Amazon EC2) console, then make sure that you do one of the following:
Paste in the user data as plaintext.
-or-
Upload the user data from a file.

If you use the AWS CLI or an AWS SDK, you must first base64-encode the user data. Then, submit that string as the value of the UserData parameter when you call CreateLaunchTemplate, as shown in the example JSON template.

5.    Run the following AWS CLI command to create a launch template based on the mount-efs.json file that you created in step 4:

aws ec2 --region us-east-1 create-launch-template --cli-input-json file://mount-efs.json

Note: Replace us-east-1 with your AWS Region.

Example create-launch-template command output

{
  "LaunchTemplate": {
    "LaunchTemplateId": "lt-06935eb650e40f886",
    "LaunchTemplateName": "user-data",
    "CreateTime": "2019-12-26T09:40:46.000Z",
    "CreatedBy": "arn:aws:iam::12345678999:user/alice",
    "DefaultVersionNumber": 1,
    "LatestVersionNumber": 1
  }
}

6.    Create a new compute environment and associate that environment with your launch template.

Note: When AWS Batch spins up instances, the Amazon EFS volume is now mounted on the instances.

7.    Check if the Amazon EFS volume is mounted with the container instance by using SSH to connect to the instance launched by AWS Batch. Then, run the following Linux df command:

$ df -h

Example df command output

Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        3.9G   92K  3.9G   1% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
/dev/xvda1       50G  854M   49G   2% /
127.0.0.1:/     8.0E     0  8.0E   0% /mnt/efs

Note: /mnt/efs is mounted automatically.

8.    Create a job definition in AWS Batch that includes the volume and mount point.

Example AWS Batch Job definition

{
  "jobDefinitionName": "userdata",
  "jobDefinitionArn": "arn:aws:batch:us-east-1:12345678999:job-definition/userdata:1",
  "revision": 1,
  "status": "ACTIVE",
  "type": "container",
  "parameters": {},
  "containerProperties": {
    "image": "busybox",
    "vcpus": 1,
    "memory": 1024,
    "command": [],
    "volumes": [
      {
        "host": {
          "sourcePath": "/mnt/efs"
        },
        "name": "efs"
      }
    ],
    "environment": [],
    "mountPoints": [
      {
        "containerPath": "/mnt/efs",
        "sourceVolume": "efs"
      }
    ],
    "ulimits": [],
    "resourceRequirements": []
  }
}

9.    Submit an AWS Batch job using the job definition that you created in step 8.


AWS OFFICIAL
AWS OFFICIALUpdated 2 years ago