How can I troubleshoot 500 errors with VPC link integration for API Gateway?

Short description

AWS API Gateway with VPC links can fail to forward traffic to backend resources and return an HTTP 500 status code for the following reasons:

• The VPC link is in the Failed state. Or, the endpoint service is deleted or in the Rejected state.
• The registered target for the target group is unhealthy, or isn't in use.
  Note: If no traffic is sent over the VPC link for 60 days, then the link becomes INACTIVE. For more information, see Working with VPC links for HTTP APIs.
• Security groups don't allow traffic on specific ports.
• The network access control list (network ACL) blocks the traffic.
• The target isn't listening on the target ports.
• The domain name certificate doesn't match the TLS-activated Network Load Balancer or Application Load Balancer.

If Amazon CloudWatch Logs is activated for your API, then an error message with the cause of the error appears in the execution logs.

Example HTTP 500 error messages in CloudWatch logs

"Thu Aug 04 17:48:21 UTC 2022 : Execution failed due to configuration error: There was an internal error while executing your request Thu Aug 04 17:48:21 UTC 2022 : Method completed with status: 500"

-or-

"Thu Aug 04 19:50:21 UTC 2022 : Execution failed due to configuration error: Host name 'domain.com.com' does not match the certificate subject provided by the peer (CN=myinstance.com)
Thu Aug 04 19:50:21 UTC 2022 : Method completed with status: 500"

Resolution

Confirm the cause of the error

Turn on CloudWatch API logging. Make sure that you configure execution logging.

Note: When you configure the logging settings, for Log level, choose INFO. Then, choose Log full requests/responses data.

To identify the cause of the errors, review the REST APIs execution logs in CloudWatch.

Check for the error message in the execution logs

Error: Execution failed due to configuration error: There was an internal error while executing your request

Check the following configurations:

• Verify that the VPC link load balancer exists and wasn't deleted.
• Verify that the VPC link is in the AVAILABLE state. If the VPC link is in the FAILED state, then you must create a new VPC link and associate it with your API.
  Note: Deploy the API after you modify the integration request.
• Verify that the VPC link endpoint connection is in the AVAILABLE state.
• If the VPC ID is referenced with a stage variable, then verify that the VPC link ID is correct.
• Verify that the VPC link load balancer listens on the HTTP/HTTPS port that the request is configured for. Check that the listeners are configured for the correct port. Make sure that the network ACLs don't block the request.
• Verify that the target group accepts the request. The network ACLs must allow inbound and outbound traffic, and the security groups must allow inbound traffic for the configured ports.
• If the request returns a 500 error, then the connection might receive TCP RST packets. Verify that the backend server is running. Verify that there's a service running on the backend target of the target port. Verify that the backend target listens on the target port.
• Verify that the Enforce inbound rules on AWS PrivateLink traffic option is turned off. Enforce inbound rules on PrivateLink traffic applies all inbound and outbound security group rules to traffic from the VPC link. If you don't configure the security group to allow traffic from the VPC link, then the security group can block API Gateway traffic. To apply security group rules to traffic that doesn't come from the VPC link, turn off Enforce inbound rules on PrivateLink traffic.

Error: Execution failed due to configuration error: Host name 'domain.com.com' does not match the certificate subject provided by the peer (CN=myinstance.com)

Make sure that the endpoint domain name matches the certificate that the TLS-activated load balancer target returns.

Error: Execution failed due to configuration error: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

This error occurs when the API Gateway can't find the root certificate authority. Even when insecureSkipVerification is set to true on the integrations TlsConfig property, API Gateway performs basic certificate validation. The validation reviews the following information:

• The certificates expiration date
• The hostname
• The presence of a root certificate authority

Certificates that you configure at the target instance must be trusted from the supported certificate authority provider list.

Error: Execution failed due to configuration error: Cannot verify ECDH ServerKeyExchange signature

This error occurs when a key and the corresponding certificate don't match, and the TLS handshake fails. To resolve this issue, check the contents of the files that you used for your configured certificate authorities, certificates, and keys.

Error: Execution failed due to an internal error

This error occurs when Amazon API Gateway didn't connect to the load balancer over the VPC link because the target resets on the load balancer. To resolve this issue, set a longer timeout on the target than the default timeout (350 seconds) that's on the load balancer.

Error: Execution failed due to configuration error: Not a valid protocol version: {Text of string}

This error occurs when the integration responded with an HTTP response that's not valid and doesn't adhere to the HTTP specification. This might mean that the integrated backend is sending back data that isn't valid to Amazon API Gateway.

To troubleshoot this error, take the following actions:

• Reproduce the request that you made from Amazon API Gateway. To test the service endpoint that Amazon API Gateway uses, create a VPC endpoint that's linked to the service endpoint ID. You can also run packet captures to review the response that's returned from the integrated backend.
• To avoid duplicate encryption, change the load balancer protocol to TLS or TCP, depending on the targets response.
• Make sure that insecureSkipVerification is set to true on the integrations TlsConfig property.

Related information

Understanding VPC Links in Amazon API Gateway private Integrations

Monitoring REST API execution with Amazon CloudWatch metrics

Set up a Network Load Balancer for API Gateway private integrations