Running Terraform in an Air-Gapped Environment – Part 2

Introduction

In our previous post, we discussed setting up Terraform in an air-gapped environment by pre-downloading providers and modules. This time, we’ll focus on running Terraform in an air-gapped AWS environment and how to leverage VPC Endpoints (VPCe) to enable secure communication with AWS services without relying on internet connectivity.

When Terraform operates in an AWS environment isolated from the public internet, VPC Endpoints act as the backbone, allowing Terraform to interact privately with AWS services. This guide explores the essential VPC endpoints required for Terraform and provides a step-by-step process to configure them.

📚 This guide is at a beginner level difficulty. 📚

What Are VPC Endpoints?

VPC Endpoints allow AWS resources in your VPC (Virtual Private Cloud) to privately connect to AWS services without going over the public internet. They use the AWS PrivateLink service and Elastic Network Interfaces (ENIs) to route traffic securely through the AWS network.

There are two types of VPC Endpoints:

1. Gateway Endpoints: For S3 and DynamoDB (free and highly efficient).
2. Interface Endpoints: For all other services like EC2, IAM, STS, RDS, etc. (incur costs).

Step-by-Step Guide to Configuring VPC Endpoints

1. S3 Gateway Endpoint

The S3 Gateway Endpoint is required for:

• Storing Terraform state files in S3.
• Downloading modules or remote files from an S3 bucket.

Create the S3 Gateway Endpoint:

aws ec2 create-vpc-endpoint \
    --vpc-id <vpc-id> \
    --service-name com.amazonaws.<region>.s3 \
    --route-table-ids <route-table-id>

Route Table: Add the endpoint to the route table associated with your private subnets.

Permissions: Restrict access to specific buckets using an endpoint policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::my-terraform-state-bucket",
        "arn:aws:s3:::my-terraform-state-bucket/*"
      ]
    }
  ]
}

2. DynamoDB Gateway Endpoint

The DynamoDB Gateway Endpoint is required if you’re using DynamoDB for state locking.

Create the DynamoDB Gateway Endpoint:

aws ec2 create-vpc-endpoint \
    --vpc-id <vpc-id> \
    --service-name com.amazonaws.<region>.dynamodb \
    --route-table-ids <route-table-id>

No additional endpoint policies are needed for DynamoDB.

3. Interface Endpoints for Terraform-Managed AWS Services

Terraform needs interface VPC Endpoints for most AWS services. Below are the key services and how to configure them:

aws ec2 create-vpc-endpoint \
    --vpc-id <vpc-id> \
    --service-name com.amazonaws.<region>.ec2 \
    --subnet-ids <subnet-id> \
    --security-group-ids <security-group-id>

aws ec2 create-vpc-endpoint \
    --vpc-id <vpc-id> \
    --service-name com.amazonaws.<region>.iam \
    --subnet-ids <subnet-id> \
    --security-group-ids <security-group-id>

aws ec2 create-vpc-endpoint \
    --vpc-id <vpc-id> \
    --service-name com.amazonaws.<region>.sts \
    --subnet-ids <subnet-id> \
    --security-group-ids <security-group-id>

aws ec2 create-vpc-endpoint \
    --vpc-id <vpc-id> \
    --service-name com.amazonaws.<region>.secretsmanager \
    --subnet-ids <subnet-id> \
    --security-group-ids <security-group-id>

4. Configure Security Groups for VPC Endpoints

For interface VPC Endpoints, ensure the associated security groups allow incoming traffic from the private subnet CIDRs.

Example Security Group Rules:

• Allow HTTPS (port 443) traffic from private subnet CIDRs.

5. Enable Private DNS for Interface Endpoints

Enable private DNS names so that Terraform can use the standard AWS service URLs (e.g., ec2.amazonaws.com) without modification.

aws ec2 modify-vpc-endpoint \
    --vpc-endpoint-id <endpoint-id> \
    --private-dns-enabled

Validating the Setup

After creating the necessary VPC Endpoints:

1. Run terraform init and ensure it can access the S3 bucket and DynamoDB table.
2. Deploy a sample resource (e.g., EC2 instance or IAM role) to verify connectivity to the respective AWS services.

Conclusion

Running Terraform in an air-gapped AWS environment requires thoughtful planning and VPC Endpoint configurations. By setting up the necessary endpoints (S3, DynamoDB, EC2, IAM, etc.), you enable Terraform to interact with AWS services securely without internet access.

This setup ensures a secure, compliant, and efficient workflow, even in the most restrictive environments.

All code in this post can be found on my GitHub.

I’ll be posting more of these guides, going over a wide range of difficulty, so subscribe below! My newsletter sends out friendly emails when I make new posts.
Want to learn more about how I can assist you with your cloud and DevOps needs? Visit my homepage to get in touch and let’s find out how I can support your next project!