Introduction

This is part 2 of a two-part tutorial series where we delve into a case study on how a private Serverless REST API can be built with Amazon API Gateway (APIGW) and accessed privately by clients in another Amazon VPC.

In part 1, we built the private API in a VPC. In part 2, we will show how clients in another VPC can privately access our private API via an Amazon VPC peering connection and Route53 resolver endpoints.

Architecture

In this part, we will build the Client VPC, VPC Peering connection, and Route53 resolver endpoints.

Picture showing steps an EC2 instance client accessing a private API in another VPC via VPC peering

Building the client VPC Infrastructure

Creating the VPC

Create a file called client-vpc.tf where we will define all the networking configurations necessary for our private API:

  
➜  tf-private-apigw git:(main) touch client-vpc.tf

Next, define the VPC CIDR, enable DNS hostnames and DNS support to allow resources within the client VPC to be automatically assigned DNS names, and enable Amazon DNS for name resolution respectively:

  
resource "aws_vpc" "client_vpc" {
    cidr_block = "172.128.0.0/16"
    enable_dns_hostnames = true
    enable_dns_support = true

    tags = {
        Name = "client-vpc"
    }
}

`‍`Creating the VPC Peering connection

In order for the client VPC to communicate with the API VPC, we must create a layer 3 connection between the VPCs.

In our case, we chose VPC peering over AWS Transit Gateway because our scenario is simple. We will set up the peering connection for the Client VPC now and the other leg of the peering connection in the API VPC later on.

Create the VPC peering connection for the client VPC, in the client-vpc.tf file:

  
resource "aws_vpc_peering_connection_accepter" "api_client_vpc_peering" {
  vpc_peering_connection_id = aws_vpc_peering_connection.api_client_vpc_peering.id
  auto_accept               = true

    tags = {
    Name = "api-client-vpc-peering",
    Side = "Accepter"
  }
}

`‍`Subnets, route tables & routing rules

Next, create private subnets, route tables and associate them together. Notice that the private route tables have a route to the API VPC CIDR.

  
resource "aws_subnet" "client_private_sn_az1" {
    vpc_id                 = aws_vpc.client_vpc.id
    cidr_block             = "172.128.1.0/24"
    availability_zone = data.aws_availability_zones.available.names[0]
    map_public_ip_on_launch = false

    tags = {
        Name = "client-private-subnet-az1"
    }
  
}

resource "aws_route_table" "client_private_rt_az1" {
    vpc_id = aws_vpc.client_vpc.id

    route {
        cidr_block = "10.0.0.0/16"
        vpc_peering_connection_id = aws_vpc_peering_connection.api_client_vpc_peering.id
    }

    tags = {
        Name = "client-private-rt-az1"
    }
  
}

resource "aws_subnet" "client_private_sn_az2" {
    vpc_id                 = aws_vpc.client_vpc.id
    cidr_block             = "172.128.2.0/24"
    availability_zone = data.aws_availability_zones.available.names[0]
    map_public_ip_on_launch = false

    tags = {
        Name = "client-private-subnet-az2"
    }
  
}

#  route table for private subnet az2
resource "aws_route_table" "client_private_rt_az2" {
    vpc_id = aws_vpc.client_vpc.id

    route {
        cidr_block = "10.0.0.0/16"
        vpc_peering_connection_id = aws_vpc_peering_connection.api_client_vpc_peering.id
    }

    tags = {
        Name = "client-private-rt-az2"
    }
  
}

#  route table associations for private subnets
resource "aws_route_table_association" "client_rta2_az2" {
    subnet_id = aws_subnet.client_private_sn_az2.id
    route_table_id = aws_route_table.client_private_rt_az2.id
}
resource "aws_route_table_association" "client_rta1_az1" {
    subnet_id = aws_subnet.client_private_sn_az1.id
    route_table_id = aws_route_table.client_private_rt_az1.id
}

DNS Resolution

For clients in the Client VPC to be able to resolve the DNS name of our private API Endpoint in our API VPC, we will need to set up an outbound Route53 resolver endpoint and a resolver rule in the Client VPC. This will ensure DNS queries for “execute-api.eu-central-1.amazonaws.com" will be resolved by the Authoritative DNS servers in the API VPC.

Setting up Outbound DNS resolution in the Client VPC

The first step is to create the security groups in the security-groups.tf file to lock down what traffic is allowed to hit the Route53 resolver endpoint:

  
resource "aws_security_group" "outbound_resolver_ep_sg" {
  name        = "private-api-outbound-resolver-endpoint-sg"
  description = "Security group for Route 53 outbound endpoints"
  vpc_id      = aws_vpc.client_vpc.id

  ingress {
    from_port   = 53
    to_port     = 53
    protocol    = "tcp"
    cidr_blocks = ["${aws_subnet.client_private_sn_az1.cidr_block}", "${aws_subnet.client_private_sn_az2.cidr_block}"]
  }
  ingress {
    from_port   = 53
    to_port     = 53
    protocol    = "udp"
    cidr_blocks = ["${aws_subnet.client_private_sn_az1.cidr_block}", "${aws_subnet.client_private_sn_az2.cidr_block}"]
  }

  egress {
    from_port   = 53
    to_port     = 53
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  egress {
    from_port   = 53
    to_port     = 53
    protocol    = "udp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  lifecycle {
    create_before_destroy = true
  }
}

‍Create the outbound Route53 resolver endpoint in the client-vpc.tf file:

  
resource "aws_route53_resolver_endpoint" "outbound_resolver_ep" {

  name      = "private-api-outbound-resolver-endpoint"
  direction = "OUTBOUND"
  security_group_ids = [aws_security_group.outbound_resolver_ep_sg.id]

  ip_address {
    subnet_id = aws_subnet.client_private_sn_az1.id
    ip        = "172.128.1.10"
  }

  ip_address {
    subnet_id = aws_subnet.client_private_sn_az2.id
    ip        = "172.128.2.10"
  }

  tags = {
    Name = "private-api-resolver-endpoint"
  }
}

‍Create a resolver rule, to forward all DNS queries for our private API endpoint’s domain name “execute-api.eu-central-1.amazonaws.com” to the inbound Route53 resolver endpoint in our API VPC. We will set up the inbound Route53 Resolver endpoint in the API VPC later in this tutorial:

  
resource "aws_route53_resolver_rule" "private_api_resolver_rule" {
  name        = "private-api-resolver-rule"
  domain_name = var.private_api_domain_name
  rule_type   = "FORWARD"
  
  resolver_endpoint_id = aws_route53_resolver_endpoint.outbound_resolver_ep.id
  target_ip     {
    ip = "10.0.0.2"
  }
  target_ip     {
    ip = "10.0.1.10"
  }
  target_ip     {
    ip = "10.0.2.10"
  }

  depends_on = [ aws_route53_resolver_endpoint.outbound_resolver_ep ]
  tags = {
    Name = "private-api-resolver-rule"
  }
}

resource "aws_route53_resolver_rule_association" "private_api_resolver_rule_assoc" {
  resolver_rule_id = aws_route53_resolver_rule.private_api_resolver_rule.id
  vpc_id = aws_vpc.client_vpc.id
}

‍Next, we check the resources that would be created/modified by Terraform by running terraform plan in the root of our project directory:

  
tf-private-apigw git:(main) terraform plan

If there are no syntax errors to fix and we are ok with the changes highlighted, the next step is to deploy to AWS using terraform apply —auto-approve.

Our folder structure now looks like this:

  
|- src/
	|- archives/
	|- handlers/
		|- libs/
			|- ddbDocClient.mjs
		|- create.mjs
		|- get.mjs
		|- update.mjs
		|- delete.mjs
|- locals.tf
|- provider.tf
|- terraform.tfvars.tf
|- variables.tf
|- api-vpc.tf
|- lambda.tf
|- security-groups.tf
|- apigw.tf
|- client-vpc.tf

`‍`Setting up VPC Peering & Route43 Resolver in the API VPC

VPC Peering connection to the client VPC

Create the other leg of the VPC Peering connection in the API VPC:

  
resource "aws_vpc_peering_connection" "api_client_vpc_peering" {
  vpc_id        = aws_vpc.api_vpc.id
  peer_vpc_id   = aws_vpc.client_vpc.id
  auto_accept   = true
  accepter {
    allow_remote_vpc_dns_resolution = true
  }
  requester {
    allow_remote_vpc_dns_resolution = true
  }
  tags = {
    Name = "api-client-vpc-peering",
    Side = "Requester"
  }
}

‍Update the private route tables with the client VPC’s CIDR range so all return traffic to the client VPC will be routed to the VPC Peering connection:

  
resource "aws_route_table" "private_rt_az1" {
  vpc_id = aws_vpc.api_vpc.id

  route {
    cidr_block = "172.128.0.0/16"
    vpc_peering_connection_id = aws_vpc_peering_connection.api_client_vpc_peering.id
  }
  tags = {
    Name = "private_rt_az1"
  }
}

resource "aws_route_table" "private_rt_az2" {
  vpc_id = aws_vpc.api_vpc.id

  route {
    cidr_block = "172.128.0.0/16"
    vpc_peering_connection_id = aws_vpc_peering_connection.api_client_vpc_peering.id
  }
  tags = {
    Name = "private_rt_az2"
  }
}

`‍`DNS Resolution

Here we will set up the inbound Route 53 Resolver endpoint with an inbound direction, to enable the resolution of external DNS queries to the API VPC from the Client VPC.

We will start by setting up the security group for the Route 53 Resolver endpoint in the security-groups.tf file:

  
resource "aws_security_group" "inbound_resolver_ep_sg" {
  name        = "api-client-inbound-resolver-endpoint-sg"
  description = "Security group for Route 53 inbound endpoints"
  vpc_id      = aws_vpc.api_vpc.id

  ingress {
    from_port   = 53
    to_port     = 53
    protocol    = "tcp"
    cidr_blocks = ["${aws_subnet.client_private_sn_az1.cidr_block}", "${aws_subnet.client_private_sn_az2.cidr_block}"]
  }
  ingress {
    from_port   = 53
    to_port     = 53
    protocol    = "udp"
    cidr_blocks = ["${aws_subnet.client_private_sn_az1.cidr_block}", "${aws_subnet.client_private_sn_az2.cidr_block}"]
  }

  egress {
    from_port   = 53
    to_port     = 53
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  egress {
    from_port   = 53
    to_port     = 53
    protocol    = "udp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  lifecycle {
    create_before_destroy = true
  }
}

‍Next, we set up the Route53 inbound resolver in the api-vpc.tf file:

  
resource "aws_route53_resolver_endpoint" "inbound_resolver_ep" {
  name = "private-api-inbound-resolver-endpoint"
  direction      = "INBOUND"
  security_group_ids = [aws_security_group.inbound_resolver_ep_sg.id]
  ip_address {
    subnet_id = aws_subnet.private_sn_az1.id
    ip = "10.0.1.10"
  }
  ip_address {
    subnet_id = aws_subnet.private_sn_az2.id
    ip = "10.0.2.10"
  }
  tags = {
    Name = "private-api-inbound-resolver-endpoint"

  }
}

Subnet Reservations

We optionally set up subnet reservations for the static IP addresses assigned to the inbound Route53 Resolver endpoints in each availability zone. This will ensure Amazon VPC doesn’t automatically assign any IP address from that range to other VPC resources, which could create conflicts.

  
resource "aws_ec2_subnet_cidr_reservation" "private_sn_az1_rsv" {
  cidr_block       = "10.0.1.0/28"
  reservation_type = "explicit"
  subnet_id        = aws_subnet.private_sn_az1.id
}
resource "aws_ec2_subnet_cidr_reservation" "private_sn_az2_rsv" {
  cidr_block       = "10.0.2.0/28"
  reservation_type = "explicit"
  subnet_id        = aws_subnet.private_sn_az2.id
}

Testing the endpoints from the Client VPC

Just like we did in part 1 of this tutorial, we will setup an instance in the Client VPC from which we will make API calls to the private API endpoint in the API VPC.

Setting up an EC2 client

The first step is to create a security group for the client instance in the security-groups.tf file:

  
resource "aws_security_group" "api_client_sg" {
  name        = "api-client-sg"
  description = "Security group for API clients"
  vpc_id      = aws_vpc.client_vpc.id

  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["${aws_subnet.private_sn_az1.cidr_block}"]

  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }


  lifecycle {
    create_before_destroy = true
  }
}

Next, we create an EC2 instance in the ec2.tf file:

  
resource "aws_instance" "client_vpc_instance" {
  ami                    = local.ami 
  instance_type          = "t2.micro"
  subnet_id              = aws_subnet.client_private_sn_az1.id
  vpc_security_group_ids = [aws_security_group.api_client_sg.id, ]
  iam_instance_profile   = aws_iam_instance_profile.ec2_instance_profile.name

  tags = {
    Name = "client-vpc-instance"
  }
}

‍‍Since our client instance is in a private subnet, we will use AWS System Manager Session Manager to access the instance for testing via a System Manager VPC Interface endpoint.

AWS System Manager Session Manager service requires two Interface endpoints (ssm and ssmmessages) to be able to access our private EC2 instance. We will set up the endpoints in the client-vpc.tf file.

Firstly, we create the security groups in the security-groups.tf file for the interface endpoints. The security group will allow only HTTPS traffic on port 443:

  
resource "aws_security_group" "ssm_ep_sg" {
  name        = "ssm-endpoint-sg"
  description = "Security group for SSM endpoints for client-vpc clients"
  vpc_id      = aws_vpc.client_vpc.id

  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["${aws_subnet.client_private_sn_az1.cidr_block}", "${aws_subnet.client_private_sn_az2.cidr_block}"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
  lifecycle {
    create_before_destroy = true
  }
}

‍Next, we set up the Interface endpoints for SSM and SSM messages:

  
resource "aws_vpc_endpoint" "ssm_ep" {
  vpc_endpoint_type   = "Interface"
  private_dns_enabled = true
  service_name        = "com.amazonaws.${var.region}.ssm"
  vpc_id              = aws_vpc.client_vpc.id
  subnet_ids          = [aws_subnet.client_private_sn_az1.id]
  security_group_ids  = [aws_security_group.ssm_ep_sg.id]
  tags = {
    Name = "ssm-endpoint"
  }
}
resource "aws_vpc_endpoint" "ssm_messages_ep" {
  vpc_endpoint_type   = "Interface"
  private_dns_enabled = true
  service_name        = "com.amazonaws.${var.region}.ssmmessages"
  vpc_id              = aws_vpc.client_vpc.id
  subnet_ids          = [aws_subnet.client_private_sn_az1.id]
  security_group_ids  = [aws_security_group.ssm_ep_sg.id]
  tags = {
    Name = "ssm-messages-endpoint"
  }
}

Run terraform validate, to verify the syntax and structure of our updated Terraform code and configuration files. Then run terraform plan to see the changes that Terraform will make to our infrastructure based on the updated configurations. Finally, run terraform apply —auto-approve to apply the changes to our infrastructure on AWS.

Once Terraform is done applying the changes successfully, you will see similar outputs like this in our CLI:

terraform outputs after a successful terraform apply

Testing System Manager Session Manager access

Follow the steps below to launch the System Manager Session Manager to our client-vpc-instance:

1. Open the Amazon EC2 console

2. Click on the Instance ID of our client-vpc-instance

3. Click on “Connect”

4. Click on Session Manager and Connect

5. If you see the CLI then the session was successfully established

We are now ready to test the private API endpoint from the Client VPC.

Testing the Endpoint

All tests will be run in the System Manager Session Manager connection to our client-vpc-instance.

Below are the scripts that we will run to test the private API Endpoints. We obtain the endpoint from the outputs from our Terraform deployment.

CreateClaim

We will make 3 calls to our claim API endpoint to create 3 items in the DynamoDB claimsTable. Make sure you update the —location with your own claim_url or claim_id_url output if you are following along:

  
curl --location 'https://9yccs3c029.execute-api.eu-central-1.amazonaws.com/dev/claim?policyId=123&memberId=123&memberName=JohnDoe' \
--header 'Content-Type: application/json' \
--data '{
  "policyType": "Health", 
"claimAmount": 500, 
"description": "malaria treatment", 
"status": "pending"
}'

curl --location 'https://9yccs3c029.execute-api.eu-central-1.amazonaws.com/dev/claim?policyId=456&memberId=456&memberName=MaryJane' \
--header 'Content-Type: application/json' \
--data '{
  "policyType": "Health", 
"claimAmount": 650, 
"description": "alergy treatment", 
"status": "pending"
}'

curl --location 'https://9yccs3c029.execute-api.eu-central-1.amazonaws.com/dev/claim?policyId=123&memberId=123&memberName=JohnDoe' \
--header 'Content-Type: application/json' \
--data '{
  "policyType": "Health", 
"claimAmount": 1500, 
"description": "snake bite treatment", 
"status": "pending"
}'

‍The image below shows the results when we create 3 claims:

Picture showing curl requests and results

We can equally see the items in our claimsTable in the Amazon DynamoDB Console:

Get

  
curl --location 'https://9yccs3c029.execute-api.eu-central-1.amazonaws.com/dev/claim/ce202035-0270-4b42-8382-3b1d7de53eef?policyId=123&memberId=123&memberName=JohnDoe'

Update

  
curl --location --request PUT 'https://9yccs3c029.execute-api.eu-central-1.amazonaws.com/dev/claim?policyId=123&memberId=123&memberName=JohnDoe' \
--header 'Content-Type: application/json' \
--data '{
  "policyType": "Health", 
"claimAmount": 2500, 
"description": "malaria treatment booster", 
"status": "pending"
}'

Delete

  
curl --location --request DELETE 'https://9yccs3c029.execute-api.eu-central-1.amazonaws.com/dev/claim/2afe60e7-3f11-485a-8a69-16389ff52fbd?policyId=123&memberId=123&memberName=JohnDoe'

Traffic flow

Let’s have a look at the DNS resolution from the instance in the Client VPC. We will do a name lookup for our private API endpoint:

As you can see, the client instance can resolve the domain name of our private API endpoint and obtain the private IP addresses of the Interface endpoints for the private API in the API VPC in multiple availability zones (10.0.1.109 & 10.0.2.150).

Notice how the Client VPC’s Amazon DNS server at 172.128.0.2 resolves the domain name of the private API and its Interface VPC Endpoint to the same private IP addresses, 10.0.1.109 in private_sn_az1 and 10.0.2.150 in private_sn_az2.

Let’s look at how the traffic flows from the client VPC to the private API Gateway Endpoint:

Picture showing steps involved when an EC2 instance client tries to access a private API in another VPC

The client calls the private API endpoint (in our case, GET https://9yccs3c029.execute-api.eu-central-1.amazonaws.com/dev/claim/2afe60e7-3f11-485a-8a69-16389ff52fbd?policyId=123&memberId=123&memberName=JohnDoe).

DNS Resolution Flow

The DNS query is sent to the VPC+2 IP (172.128.0.2) in the Client VPC that connects to the outbound Route 53 Resolver endpoint.
The Route 53 Resolver forwarding rule is configured to forward the queries for “execute-api.eu-central-1.amazonaws.com” to the inbound endpoints in the API VPC. The query is forwarded to the Route53 outbound Resolver endpoint in the Client VPC.
The outbound endpoint forwards the query to the Route53 inbound Resolver in the API VPC which goes over the VPC Peering connection.
The inbound Route53 Resolver endpoint in the API VPC forwards the DNS query to its VPC + 2 (10.0.0.2) which returns the the Layer 3 IP address of the private API’s Interface endpoint to the client instance in the Client VPC via the same path in reverse.

Packet Forwarding Flow

The client instance in the Client VPC sends the packet containing the payload with the destination IP of the private API Endpoint in the API VPC.
The VPC peering connection forwards the traffic based on the destination IP in the packet to the Interface Endpoint of the private API
Amazon API Gateway passes the payload to our private Lambda through the integration request.
Depending on the request in the payload, the Lambda functions perform CRUD operations on DynamoDB Table via the DynamoDB Gateway Endpoint.

As we did in the first part of this tutorial, we will use the AWS Network Manager VPC Reachability Analyzer to see how the Layer 3 IP Packet moves from the EC2 instance in the client VPC, through the VPC Peering link, to the API VPC and finally hitting the private API Interface Endpoint. Below is a snippet of an analysis.

AWS Network Manager Reachability Analyzer path details

Cleaning Up

To clean up all resources created by Terraform, run terraform destroy --auto-approve in the project root directory:

  
tf-private-apigw git:(main) terraform destroy --auto-approve

Recap

In the first part of this tutorial, we created a VPC for our private API Gateway endpoint, and in this part, we created the layer 3 networking and DNS infrastructure so clients in the Client VPC can make API requests to the private API Gateway endpoint successfully.

Research/References

https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resolver.html

https://repost.aws/knowledge-center/api-gateway-private-endpoint-connection

https://repost.aws/knowledge-center/vpc-peering-connection-create

https://repost.aws/knowledge-center/ec2-systems-manager-vpc-endpoints

https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started-privatelink.html

https://docs.aws.amazon.com/whitepapers/latest/best-practices-api-gateway-private-apis-integration/rest-api.html

Fon Nkwenti

Senior Serverless Developer & Project Manager

Fon Nkwenti is a Senior Serverless Developer and Project Manager at Serverless Guru. He loves organizing tech communities and events.

Private Serverless REST API with API Gateway: Part 2

Introduction

Architecture

Building the client VPC Infrastructure

Creating the VPC

‍Creating the VPC Peering connection

‍Subnets, route tables & routing rules

DNS Resolution

Setting up Outbound DNS resolution in the Client VPC

‍Setting up VPC Peering & Route43 Resolver in the API VPC

VPC Peering connection to the client VPC

‍DNS Resolution

Subnet Reservations

Testing the endpoints from the Client VPC

Setting up an EC2 client

Testing System Manager Session Manager access

Testing the Endpoint

CreateClaim

Get

Update

Delete

Traffic flow

DNS Resolution Flow

Packet Forwarding Flow

Cleaning Up

Recap

Research/References

The dream team

Looking for skilled architects & developers?

More from Serverless Guru

Choose Your Character: Same Service, Different Tools ( Part 2 )

Choose Your Character: Same Service, Different Tools ( Part 1 )

Our Rebrand to Server Guru (April Fools!)

Join the Community

Private Serverless REST API with API Gateway: Lambda, DynamoDB, VPC Endpoints & Terraform - Part 2

Looking for Senior AWS Serverless Architects & Engineers?

Introduction

Architecture

Building the client VPC Infrastructure

Creating the VPC

‍Creating the VPC Peering connection

‍Subnets, route tables & routing rules

DNS Resolution

Setting up Outbound DNS resolution in the Client VPC

‍Setting up VPC Peering & Route43 Resolver in the API VPC

VPC Peering connection to the client VPC

‍DNS Resolution

Subnet Reservations

Testing the endpoints from the Client VPC

Setting up an EC2 client

Testing System Manager Session Manager access

Testing the Endpoint

CreateClaim

Get

Update

Delete

Traffic flow

DNS Resolution Flow

Packet Forwarding Flow

Cleaning Up

Recap

Research/References

Fon Nkwenti

More from Serverless Guru

Choose Your Character: Same Service, Different Tools ( Part 2 )

Choose Your Character: Same Service, Different Tools ( Part 1 )

Our Rebrand to Server Guru (April Fools!)

Private Serverless REST API with API Gateway: Lambda, DynamoDB, VPC Endpoints & Terraform - Part 2

Navigating Post-Payment: AWS Step Functions for E-Commerce Excellence

Private Serverless REST API with API Gateway: Lambda, DynamoDB, VPC Endpoints & Terraform - Part 1

`‍`Creating the VPC Peering connection

`‍`Subnets, route tables & routing rules

`‍`Setting up VPC Peering & Route43 Resolver in the API VPC

`‍`DNS Resolution

`‍`Creating the VPC Peering connection

`‍`Subnets, route tables & routing rules

`‍`Setting up VPC Peering & Route43 Resolver in the API VPC

`‍`DNS Resolution