How do I influence my Direct Connect connection network traffic path to on-premises over multiple circuits?

Short description

You can use multiple Direct Connect circuits in different AWS Regions to increase bandwidth and high availability. Depending on the distribution of the circuits across Regions, you can influence Direct Connect traffic by advertising Border Gateway Protocol (BGP) community tags with on-premises prefixes.

Common setups for multiple Direct Connect connections include:

• Private virtual interfaces created on the Direct Connect connections attached to the same virtual private gateways (VGW) in the same Region.
• Private virtual Interfaces created on the Direct Connect connections attached to the same Direct Connect Gateway (DXGW) and associated with multiple VGWs in any Regions.
• Transit virtual Interfaces created on the Direct Connect connections attached to the same DXGW and associated with multiple transit gateways in any Regions.

Resolution

Follow these instructions to influence your Direct Connect connection network traffic path based on your use case.

Default behavior for private and transit virtual Interfaces and how to influence it

To influence the default behavior, it's a best practice to use Autonomous System (AS) path prepending and the following local preference BGP community tags:

• 7224:7100—Low preference
• 7224:7200—Medium preference
• 7224:7300—High preference

Local preference BGP community tags are evaluated in order from lowest to highest preference (where highest preference is preferred). For each prefix that you advertise over a BGP session, you can apply a community tag to indicate the priority of the associated path for returning traffic.

For more information, see How can I use BGP communities to influence the preferred routing path on Direct Connect links from AWS to my network?

Scenario 1

You have one Direct Connect connection (DX1) in the us-east-1 Region, and a second connection (DX2) in the eu-west-1 Region. A private virtual interface (VIF) is created on DX1 and DX2 with the same prefix's advertised from on-premises devices. DX1 and DX2 VIF's are attached with a DXGW associated with three VGWs in the us-east-1, eu-west-1, and ap-southeast-1 Regions.

If you advertise BGP tag 7224:7300 over the VIF on DX1, traffic from the us-east-1 Region Amazon Virtual Private Cloud (Amazon VPC) is unchanged. Traffic from the eu-west-1 and ap-southeast-1 Regions prefer DX1 because the same region preference is overwritten by the BGP community.

If you advertise BGP tag 7224:7300 over VIFs on DX1 and DX2, traffic from all Amazon VPCs in different regions are load balanced across DX1 and DX2. This is because the same region preference is overwritten by the BGP community.

If you advertise BGP tag 7224:7300 over both VIFs with AS prepending on DX1, traffic from all Amazon VPCs in all regions prefer DX2. This is because the region preferences are overwritten as equal. Then, AWS performs validation on the AS path length finding that the path prefix for DX2 is shorter than DX1.

If you only advertise one AS prepending over VIF on DX1 without BGP community tags, traffic from Amazon VPCs in the us-east-1 and eu-west-1 Regions remains unchanged. Traffic from the ap-southeast-1 Region prefers DX2. This is because the region preference has a higher priority than the AS path length.

Scenario 2

You have Direct Connect connections DX1 and DX2 in the us-east-1 Region with private virtual interfaces using the same prefix advertised from on-premises devices. The private virtual interfaces are attached with the DXGW associated with three VGWs in the us-east-1 and eu-west-1 Regions.

If you advertise BGP tag 7224:7300 over the VIF on DX1, traffic from the us-east-1 and eu-west-1 Regions prefer DX1 because the same region preference is overwritten by the BGP community.

If you advertise BGP tag 7224:7300 over both VIFs on DX1 and DX2, traffic from VPCs in different regions remain unchanged. This is because the region preference is overwritten by the BGP community tag for load balancing.

If you advertise BGP tag 7224:7300 over both VIFs with AS prepending on DX1, traffic from Amazon VPCs in all regions prefers DX2. This is because the region preference is overwritten as equal. Then, AWS performs validation on the AS path length finding that the path prefix for DX2 is shorter than DX1.

If you advertise one AS prepending over the VIF on DX1 without BGP community tags, traffic from Amazon VPCs in the us-east-1 and eu-west-1 Regions prefers DX2. This is because the us-east-1 and eu-west-1 Regions have the same region preference and a shorter AS path for DX2.

Scenario 1 and Scenario 2

If you advertise on-premises prefixes without influencing factors, the default traffic egress behavior is the following:

• Traffic sourced from the Amazon VPC in the us-east-1 Region is preferred for DX1 because it’s in the same region. Traffic sourced from the Amazon VPC in the eu-west-1 Region is preferred for DX2 because it's in the same region. Because the ap-southeast-1 is a remote region, traffic sourced from the Amazon VPC is load balanced across DX1 and DX2.
• Traffic sourced from the Amazon VPC in the us-east-1 Region is load balanced across DX1 and DX2 because they are in the same region. Traffic sourced from the Amazon VPC in the eu-west-1 Region is load balanced across DX1 and DX2.

Default behavior for public virtual interfaces and how to influence it

In the following scenario, you have two Direct Connect connections in the same Region that use the same public virtual interface and prefix advertised from on- premises devices.

Egress traffic from the AWS direction is load balanced across public VIFs without influencing factors. To influence the default behavior, It's a best practice to use Autonomous System (AS) prepending with a public ASN. If you can only use a private ASN, it's a best practice to advertise more specific prefixes from the preferred public VIF, so that the longest prefix match determines the routing path.

Common asymmetric routing scenarios for public virtual interfaces

You can use the following BGP prefixes with Direct Connect:

• 7224:9100—Local AWS Region
• 7224:9200—All AWS Regions for a continent
• 7224:9300—Global (all public AWS Regions)

For more information, see the scope BGP communities.

In the following scenario, you have a Direct Connect connection in the us-east-1 Region with a public virtual interface. You're advertising the same on-premises public prefix with the community tag 7224:9100 over the Direct Connect connection and the local internet service provider.

If you're accessing an Amazon Simple Storage Service (Amazon S3) bucket in the us-east-1 Region, the inbound and outbound traffic is over the Direct Connect connection.

If you're accessing an Amazon S3 bucket in the eu-west-1 Region, the inbound traffic is over the Direct Connect connection, and the outbound traffic is over the local internet service provider. This is because the prefix isn't propagated to the eu-west-1 Region causing asymmetric routing.

If you’re accessing an Amazon S3 bucket in the us-east-1 Region and the inbound traffic from on-premises uses the local internet service provider, asymmetric routing occurs. This is because AWS prefers to send outbound traffic over the Direct Connect connection instead of the local internet service provider when the same prefixes are received.

Related information

Configure redundant connections with AWS Direct Connect

How do I set up an Active/Active or Active/Passive Direct Connect connection to AWS from a public virtual interface? 

AWS Direct Connect virtual interface types

---