Why are my list queries returning the wrong number of items when I use DynamoDB with AWS AppSync?

4 minute read
0

My list queries are returning the wrong number of items when I use Amazon DynamoDB with AWS AppSync. How do I resolve this?

Short description

When you use a list query with AWS AppSync and Amazon DynamoDB, DynamoDB performs a Scan operation on the table and returns a certain number of evaluated items. If you provide a FilterExpression operation on the request, then DynamoDB applies the FilterExpression to only these evaluated items.

The number of items that the Scan checks depends on the limit variable that you applied to the query. If no limit is applied, then DynamoDB uses the default limit that's configured in the request mapping template. The Scan returns only the items that are evaluated and match the filter expression. To return the expected number of items, you must adjust the limit variable.

You can increase the default limit on your mapping template, or add a limit variable to the list GraphQL query. However, the maximum number of items that you can evaluate is 1 MB. If your Scan operation exceeds 1 MB of data, then you can use pagination to get the rest of the data. To help you query a database that has thousands of items, you can index your data to filter results by a common field.

Resolution

Increase the default limit

In the following example mapping template for an AWS Amplify generated API, the default limit is 100:

#set( $limit = $util.defaultIfNull($context.args.limit, 100) )
#set( $ListRequest = {
  "version": "2018-05-29",
  "limit": $limit
} )

To evaluate more items, change the default limit in the mapping template. Or, add a limit variable to the list query. For example, in the following list query, the limit is set to 1000:

query MyQuery {
  listEmployees(limit: 1000) {
    items {
      id
      name
      company
    }
  }
}

Paginate with AWS AppSync

To paginate with AWS AppSync, add a nextToken argument to your GraphQL query.

Example query:

query MyQuery {
  listEmployees{
    items {
      id
      name
      company
    }
    nextToken
  }
}

Note: The nextToken value that the query returns is either null or a long string that starts with ey and ends with =. This value represents the last evaluated key. If the value is null, then there are no more items in the table to evaluate. If it's a long string, then there are more items to evaluate.

For example:

"nextToken": "eyJ2ZXJzaW9uIjoyLCJ0b2tlbiI6IkFRSUNBSGg5OUIvN3BjWU41eE96NDZJMW5GeGM4WUNGeG1acmFOMUpqajZLWkFDQ25BRkQxUjREaVVxMkd1aDZNZ2ZTMmhPMUFBQUNIVENDQWhrR0NTcUdTSWIzRFFFSEJxQ0NBZ293Z2dJR0FnRUFNSUlCL3dZSktvWklodmNOQVFjQk1CNEdDV0NHU0FGbEF3UUJMakFSQkF4c0RFY1ZZUGpaRDFxbDcxNENBUkNBZ2dIUWkwaGtJYytoU04vRFMzL3VQT2ZDMnpIV1dBVkg4LzU3TWFwMGZWSHVackt1VmV4emRvTVQrNml6VC9RdDRsSVNGN1pmc3NWTHVvcnJpRE1RZVE5ckNyd3J4dmNOY3ZZUzhTc21PRFFkaTUreVhQcDJ1OENaK29Sd2wrV3VoOGJ0YlBpQXQydjRNdmw2L09jRzRHV2RxRmFjeTFDRjRyK2FPd29velRTV3NqMTd4OUpwVi93cHVYc2tTN2R5TmYxa3JJS3hSL3BMWlY5M3JPSlVocEpDV2xEL3Y1UU5JdGJoOWpyaTI3N09LbUZsVlh3bDRna3hDa1kzeGZMNjZrM2dZY0hpbHlUOE1GUnBLU0VCTFE3RGEzSlJtUG8xUkJvQ051K3dBTkwwd09Vckp0N1BFb0QvTVRrenZDV1lCTXhVaUhaRDdrM3Y5a2FJS2NScHY0ajhuTDJ2Nk9EZ3plTjgwN1RtRFViM21rRUsrakRRcjUvd3ZQVW56cGFaN1RnR21yT21FaTlGQklOUnl6dk9rdDRWazZEaVU3RCtNYUJSdm5iNnR0VklPa2lDdFlhODRqenhlOFlFRUZGOElyTksrQm9yL28vdktxMVczSUxsU1VWWFd0N0hPWjV4TDBudHVTeGlBdW9ZK1Y0NEkzMXlPQkJ1T1AwMVpUek1TdGUvZCtIT1RRUEt2SGVGanF5Y0tpNGNTQUdZN3BobGs5eWJJem9hOTM0YldJOUFyRmF0WDY4UnkzTkF4cWNCbzh4ZklxZGZNN3Rlam02NldMT0Z6T3F6MDRrK1B0K0lXdWhOeS9CWEN2YXh2dk09In0="

To get the rest of the items in the table, run another query with the nextToken as a query variable. Continue to include the nextToken variable until all the items are evaluated.

Example query:

query MyQuery {
  listEmployees(nextToken:"eyJ......="){
    items {
      id
      name
      company
    }
    nextToken
  }
}

Index your data

In the following example query, the filter expression is set to return a number of items that are based on employees that equal AnyCompany:

query MyQuery {
  listEmployees(filter: {company: {eq: "AnyCompany"}}) {
    items {
      id
      name
      company
    }
    nextToken
  }
}

To list all employees based on their company, create a global secondary index for the field to query against. For example, in Amplify, you can use the @index directive to define secondary indexes:

type Employee @model{
  id:ID
  name : String
  email : AWSEmail
  company: String @index(name:"employeeByCompany", queryField: "listEmployeesByCompany")
}

In the following example query, a GSI employeeByCompany is set for the company field, and the query is defined as listEmployeesByCompany:

query MyQuery {
  listEmployeesByCompany(company: "AnyCompany") {
    items {
      id
      name
      company
    }
    nextToken
  }
}

You can specify a limit variable to return an intended number of items in the database. For example, in the following query, the limit variable is set to 5:

query MyQuery {
  listEmployeesByCompany(company: "AnyCompany",limit:5) {
    items {
      id
      name
      company
    }
    nextToken
  }
}

Note: Setting a limit variable returns only the intended number of items, even if you applied additional filterExpressions. For example, in the following query, the limit variable is set to 5 and the filter equals Mary Major:

query MyQuery {
  listEmployeesByCompany(company: "AnyCompany",limit:5) {
    name:{
      eq:"Mary Major"
    }
  }) {
    items {
      id
      name
      company
    }
    nextToken
  }
}

The preceding query returns only 5 items that equal Mary Major. However, there might be more than 5 Mary Majors.

Note: Indexed queries also return a maximum of 1 MB of data at a time. For larger indexed queries, you can use pagination.


AWS OFFICIAL
AWS OFFICIALUpdated 2 years ago
2 Comments

Hi, this is correct but.

I think Appsync limit is not merely based on items length, but on kb of data. E.g. I have a list of items which in turn store a long string of json.

In that case, no matter what is the limit I put I always receive only 54 items. Even if I only return the ids of the items from graphql.

replied a year ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied a year ago