AWS Direct Connect – Part 1

During some work with a client, we had quite a deep dive into AWS Direct Connect options. I thought I’d write up some of the material here in case it’s helpful to anyone else.

In this post, I will address what Direct Connect is and finding a Direct Connect Partner as well as the physical peering requirements. In the next post, I will go into detail on the BGP peering and route policy design.

Let’s start with an overview of the objective: You have or are planning on having, some resources in AWS (EC2, S3, etc) that you need to access. However, you still have resources on-premise that need either to access the cloud or migrate to the cloud.

The easiest way to begin is to set up a site-to-site VPN from your on-premises data center directly to your VPN using the VGW (Virtual Private Gateway) VPC construct. This may meet your needs in the short term as long as you only need connectivity to one VPC.

As we move beyond 1 VPC, we can simply deploy another VGW. This increases the tunnels to be managed and comes with the design limit that you cannot route between the VPCs in this construct without hair-pinning back to the on-premises datacentre (possibly incurring data transfer cost in addition to latency).

A better design pattern is to use the Transit Gateway regional construct to terminate the IPSec VPN and manage the routing between the VPCs.

However, with all these architectures, we are still using the internet as transport.

If you require a dedicated connection to AWS with SLA, low latency, high throughput, low packet loss, and perhaps better data transfer costs, then Direct Connect is the service you need.

Direct Connect brings with it a certain number of moving parts that need to be designed for.

One of the first things you’ll notice is that there needs to be somewhere to physically plug you into the AWS network. This is the role of the AWS Direct Connect Partners.

Direct Connect Partners

Direct Connect connectivity is facilitated through Direct Connect Partners. These are either in cohosting facilities or via a connection provided to your premises by the partner in question.

https://aws.amazon.com/directconnect/partners/?partner-solutions-cards.sort-by=item.additionalFields.partnerNameLower&partner-solutions-cards.sort-order=asc&awsf.partner-solutions-filter-location=*all

Partners are AWS approved for different connection types:

• Dedicated Connections – you have a private cabled connection to the AWS backbone
• Hosted Connection – you share a physical connection to the AWS backbone with a guaranteed bandwidth slice
• Hosted VIF – the least flexible, share of an oversubscribed link to the AWS backbone and connectivity to 1 VIF

In a typical Direct Connect design, we’ll see a setup something similar to the above. One of the most common design patterns is to extend your WAN into the Direct Connect partners facility.

We will assume a “dedicated connection” for the purpose of this blog post (this basically means that the physical connection between your devices and the AWS devices is dedicated to you).

Design Considerations

There are a number of steps to go through to get there.

Let’s start by identifying the major design considerations:

Enterprise WAN Design – any Direct Connect design or deployment will depend heavily on your current WAN or SDWAN design. Integrated into this design should be a Cloud Edge block which will take care of the physical connection and logical peering with the AWS network.

Direct Connect Physical Peering – the physical options and configuration required to physically connect your WAN Cloud Edge devices to the AWS network devices

Direct Connect Logical Peering – through the establishment of BGP peering sessions, the mutual advertisement of routes can take place. Planning and design here are critical.

Enterprise WAN Design

Once you have chosen your AWS Direct Connect partner, you need to understand how you can extend your WAN design into their facility. Whilst there are multiple options here, I will outline one that I see most often.

If you want to deploy your SDWAN devices directly into the AWS region, you will need a different design approach. Let me know if you want more informatino on this type of design.

Most clients create a Cloud Edge block whereby they place 2 redundant routers at the colocation facility and extend their WAN through these devices.

These devices need to meet certain criteria for physical and logical peering as can be seen below.

Physical Peering

By this stage, you have physically installed your devices in the cohosting facility racks. They need to cable your devices directly to the AWS backbone via their cross-connect. However, in order to do that they need a Letter of Authority (LOA).

The Letter of Authority (LOA) can be downloaded from the AWS console after setting up Direct Connect.

The LOA can be printed from the AWS console after you have signed up for Direct Connect and begun your configuration. From the moment you sign up, Amazon needs around 72 hours to prepare the connection on their side after which you are ready to be cabled up by the AWS Partner.

Depending on the service you have purchased, you will have to ensure you have the correct physical presentation to terminate calling from your network devices (routers) to the facility cross-connect patch panel. This is a critical element to get right in the planning as AWS support very specific connection types.

• Your network must use single-mode fiber with a 1000BASE-LX (1310 nm) transceiver for 1 gigabit Ethernet, a 10GBASE-LR (1310 nm) transceiver for 10 gigabit, or a 100GBASE-LR4 for 100 gigabit Ethernet.
• Auto-negotiation for the port must be disabled. Auto-negotiation is supported only if the port speed is 1 Gbps. Port speed and full-duplex mode must be configured manually.
• 802.1Q VLAN encapsulation must be supported across the entire connection, including intermediate devices.
• Your device must support Border Gateway Protocol (BGP) and BGP MD5 authentication.
• (Optional) You can configure Bidirectional Forwarding Detection (BFD) on your network. Asynchronous BFD is automatically enabled for AWS Direct Connect virtual interfaces, but does not take effect until you configure it on your router.

For dedicated connections, Link Aggregation is possible for increased bandwidth.

Physical peering resiliency design can be achieved in a few different ways. It’s a prudent design pattern to peer using 2 devices on your side, terminated to 2 different AWS devices on their side. Amazon also manages the physical cabling on their side to limit any planned or unplanned maintenance from impacting the BGP peering sessions.

Logical Peering – Next Post

In the next post, we will go into more detail on the BGP peering and routing policy over Direct Connect.

See you in the next one…