Salek Ali.
System Operational
aws kubernetes observability prometheus grafana terraform

Amazon Managed Prometheus + Grafana on EKS: Open Source Terraform Module (Part 2 of 5)

How the metrics module works: AMP workspace, AMG with SSO, IRSA-scoped ingestion, pre-built dashboards, and the two-phase Terraform apply pattern that ties it all together.

Dr Salek Ali 14 April 2026
Amazon Managed Prometheus + Grafana on EKS: Open Source Terraform Module (Part 2 of 5)

This is Part 2 of my five-part series on the eks-observability-stack. Part 1 covered the architecture and design decisions. This post digs into the metrics layer: Amazon Managed Prometheus (AMP) for storage, Amazon Managed Grafana (AMG) for dashboards, and the Terraform code that wires them together.

If you just want the code, it’s all in modules/metrics/.

What the Metrics Module Deploys

The metrics module creates five things:

  1. An AMP workspace for storing Prometheus metrics
  2. An AMG workspace with AWS SSO authentication
  3. An IRSA role scoped to aps:RemoteWrite on that specific workspace
  4. A Kubernetes namespace (monitoring) for collector workloads
  5. Three pre-built Grafana dashboards (cluster health, node metrics, application metrics)

Optionally, it also deploys an ADOT Collector via Helm if you need a managed metrics scraper. But most teams already have kube-prometheus-stack or a similar setup handling the scrape side, so the ADOT piece is off by default.

AMP: The Prometheus Backend

The AMP submodule is deliberately simple. One resource, one optional rule group:

resource "aws_prometheus_workspace" "this" {
  alias = "${var.cluster_name}-amp"
  tags  = var.tags
}

resource "aws_prometheus_rule_group_namespace" "alerts" {
  count = var.alert_rules_yaml != "" ? 1 : 0

  name         = "eks-observability-alerts"
  workspace_id = aws_prometheus_workspace.this.id
  data         = var.alert_rules_yaml
}

That’s the entire submodule. AMP handles storage, HA, retention, and scaling. You don’t configure any of that. You get a prometheus_endpoint output and you point your remote-write config at it.

The alerting rules are injected from the alerting module (covered in Part 5). If alerting is disabled, alert_rules_yaml is empty and the rule group resource is skipped entirely. No conditionals scattered through the code, just a clean count.

Why AMP Instead of Self-Hosted Prometheus

I covered this in Part 1, but it bears repeating with specifics. Self-hosted Prometheus on EKS means:

  • Persistent volumes that need monitoring themselves (ironic)
  • Retention management that someone will forget to configure until the disk fills
  • OOM kills when label cardinality explodes (and it will)
  • HA configuration with Thanos or Cortex, which is its own operational burden
  • Upgrades that occasionally break scrape configs

AMP gives you the same PromQL interface, the same remote-write protocol, and none of the above. The trade-off is cost. AMP charges per metric sample ingested and per query processed. For most EKS clusters, the bill is between $50 and $300/month. Compare that to the engineering hours you spend babysitting a self-hosted Prometheus that exists only to monitor your other infrastructure.

AMG: The Grafana Frontend

The AMG submodule creates a workspace with SSO authentication and a short-lived service account token for Terraform:

resource "aws_grafana_workspace" "this" {
  name                     = "${var.cluster_name}-amg"
  account_access_type      = "CURRENT_ACCOUNT"
  authentication_providers = ["AWS_SSO"]
  permission_type          = "SERVICE_MANAGED"
  data_sources             = ["PROMETHEUS"]
  tags                     = var.tags
}

resource "aws_grafana_workspace_service_account" "terraform" {
  name         = "terraform-dashboard-provisioner"
  grafana_role = "ADMIN"
  workspace_id = aws_grafana_workspace.this.id
}

resource "aws_grafana_workspace_service_account_token" "terraform" {
  name               = "terraform"
  service_account_id = aws_grafana_workspace_service_account.terraform.service_account_id
  seconds_to_live    = 3600
  workspace_id       = aws_grafana_workspace.this.id
}

A few things to note:

SSO authentication. AMG uses AWS IAM Identity Center (formerly AWS SSO). Your team members sign in through your existing identity provider. No separate Grafana user database. No password resets. No “admin/admin” default credentials that someone forgets to change.

Service account token with a one-hour TTL. The token exists only to let Terraform create dashboards and data sources. It expires after 3600 seconds. This is intentional. You don’t want a long-lived admin token sitting in your Terraform state. Run the apply, dashboards get created, token expires. If you need to update dashboards later, the next apply generates a fresh token.

SERVICE_MANAGED permissions. AMG manages its own IAM role for reading from AMP. You don’t need to create cross-service IAM policies manually.

IRSA: Scoped Ingestion Credentials

Every workload that writes metrics to AMP needs an IAM role. The stack creates these via a reusable IRSA module:

module "amp_ingest_irsa" {
  source = "../irsa"

  role_name                 = "${local.metrics_name_prefix}-amp-ingest"
  cluster_oidc_provider_arn = var.cluster_oidc_provider_arn
  cluster_oidc_issuer_url   = var.cluster_oidc_issuer_url
  service_account_name      = "amp-iamproxy-ingest-service-account"
  namespace                 = var.namespace

  inline_policies = {
    "${local.metrics_name_prefix}-amp-remote-write" = jsonencode({
      Version = "2012-10-17"
      Statement = [{
        Effect   = "Allow"
        Action   = ["aps:RemoteWrite"]
        Resource = module.amp.workspace_arn
      }]
    })
  }
}

This role can do exactly one thing: write metrics to one specific AMP workspace. It cannot read metrics. It cannot delete the workspace. It cannot touch any other AWS service. The trust policy restricts it to a single Kubernetes service account in a single namespace.

If you enable the optional ADOT Collector, it gets its own separate IRSA role with the same scoped policy. Two workloads, two roles, both least-privilege.

No static AWS credentials anywhere in the cluster.

The Two-Phase Apply

This is the most important operational detail in the metrics module, and the thing most people get wrong when trying to manage Grafana dashboards with Terraform.

The problem: Terraform’s Grafana provider needs an endpoint URL and an auth token to configure dashboards. But the AMG workspace (which produces that endpoint) is created by Terraform itself. You can’t configure a provider with values that don’t exist yet.

The solution: two-phase apply.

Phase 1: Create the infrastructure.

terraform apply

This creates the AMP workspace, AMG workspace, IRSA roles, and namespace. Grafana resources are skipped because grafana_endpoint defaults to empty.

After phase 1 completes, Terraform outputs the Grafana endpoint and service account token.

Phase 2: Configure Grafana.

export TF_VAR_grafana_endpoint=$(terraform output -raw grafana_endpoint)
export TF_VAR_grafana_auth=$(terraform output -raw grafana_service_account_token)
terraform apply

Now the Grafana provider has valid credentials. This second apply creates the AMP data source in Grafana and deploys all three dashboards.

The root module handles this with a simple guard:

locals {
  grafana_enabled = var.enable_metrics && var.grafana_endpoint != ""
}

resource "grafana_data_source" "amp" {
  count = local.grafana_enabled ? 1 : 0

  type       = "prometheus"
  name       = "Amazon Managed Prometheus"
  uid        = "AMP"
  url        = module.metrics[0].amp_endpoint
  is_default = true

  json_data_encoded = jsonencode({
    httpMethod    = "POST"
    sigV4Auth     = true
    sigV4AuthType = "default"
    sigV4Region   = data.aws_region.grafana[0].id
  })
}

The data source uses SigV4 authentication. Grafana signs requests to AMP using its own IAM role. No API keys, no basic auth, no credentials stored in Grafana’s database.

Why Not Use a terraform_remote_state Loop?

I’ve seen stacks that try to solve this with remote state references or depends_on chains. It doesn’t work cleanly because Terraform providers are initialized before resources are planned. If the provider block references a resource output, Terraform fails during the plan phase.

The two-phase pattern is explicit, predictable, and documented. It adds one extra command to your initial setup. After that, subsequent applies are single-phase because the variables are already set.

Pre-Built Dashboards

The stack ships three dashboards as JSON files. They’re deployed via the Grafana Terraform provider:

resource "grafana_dashboard" "this" {
  for_each = local.grafana_enabled ? local.builtin_dashboards : {}

  config_json = each.value
  overwrite   = true

  depends_on = [grafana_data_source.amp]
}

Cluster Health Dashboard

Five panels covering the basics:

PanelPromQL (simplified)Purpose
Node CPU Utilisation100 - avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100CPU pressure across nodes
Node Memory Utilisation100 * (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)Memory pressure across nodes
Total Pod Countkube_pod_status_phase{phase="Running"} by namespacePod distribution and density
PVC Usagekubelet_volume_stats_used_bytes / kubelet_volume_stats_capacity_bytesStorage capacity warnings
Cluster Request Ratesum(rate(http_requests_total[5m]))Overall traffic volume

This is your “is the cluster healthy right now” dashboard. You glance at it and know whether something needs attention.

Node Metrics Dashboard

Four panels with a $node template variable so you can drill into specific nodes:

  • CPU Usage per Node: Per-node CPU with the idle rate inverted
  • Memory Usage per Node: Available vs total memory ratio
  • Disk I/O per Node: Read and write bytes per second
  • Network I/O per Node: Receive and transmit bytes per second

When the cluster health dashboard shows a CPU spike, you switch to this dashboard, select the affected node, and see exactly what’s happening.

Application Metrics Dashboard

Four panels for service-level indicators:

  • Request Rate: sum by(job)(rate(http_requests_total[5m])) grouped by service
  • Error Rate: 5xx responses as a percentage of total requests, with thresholds at 1% (yellow) and 5% (red)
  • P95 Latency: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) with a 500ms red threshold
  • Active Connections: Current open connections per service

These metrics require your applications to expose Prometheus-format metrics. If you’re running a standard HTTP service with a /metrics endpoint, these dashboards work out of the box. The sample app (covered in Part 5) demonstrates the exact metric names and labels expected.

Connecting Your Existing Prometheus Setup

Most teams already have something collecting metrics. The stack doesn’t force you to rip that out. You have two options:

Option A: Keep your existing scraper, add AMP as a remote-write target.

If you’re running kube-prometheus-stack or a standalone Prometheus, add a remote_write block pointing to AMP:

remoteWrite:
  - url: "https://aps-workspaces.ap-southeast-2.amazonaws.com/workspaces/ws-xxx/api/v1/remote_write"
    sigv4:
      region: ap-southeast-2
    queue_config:
      max_samples_per_send: 1000
      max_shards: 200
      capacity: 2500

The Kubernetes service account running your Prometheus needs the IRSA role the stack creates. Annotate the service account:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: amp-iamproxy-ingest-service-account
  namespace: monitoring
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/my-cluster-metrics-amp-ingest

Option B: Enable the ADOT Collector.

Set enable_adot_collector = true in the metrics module. The stack deploys an ADOT Collector via Helm that scrapes Prometheus targets and remote-writes to AMP. It comes pre-configured with its own IRSA role. Zero manual annotation needed.

Cost Considerations

AMP pricing has three components:

  1. Metric samples ingested: $0.90 per 10 million samples (first 2 billion/month)
  2. Metric samples stored: $0.03 per million samples/month
  3. Query samples processed: $0.10 per 10 million samples scanned

For a typical 10-node EKS cluster running 50 services with standard Kubernetes metrics (node-exporter, kube-state-metrics, cAdvisor) plus application metrics, expect roughly:

  • Ingestion: 500M to 1B samples/month = $45 to $90
  • Storage: Scales with retention (default 150 days)
  • Queries: Depends on dashboard refresh rate and ad-hoc usage

AMG pricing is $9/editor/month and $5/viewer/month. Most teams need 2 to 5 editors and a handful of viewers.

Total: $100 to $300/month for a production metrics setup. That’s less than one day of an engineer’s time spent debugging a self-hosted Prometheus outage.

What’s Next

Part 3 covers the logging layer: Fluent Bit collecting from every pod, routing to CloudWatch Logs or OpenSearch, and how to switch backends without re-architecting anything.

The full stack is on GitHub. Star it, fork it, open issues.

If you want this deployed on your EKS clusters, configured for your compliance requirements, and integrated with your existing monitoring, book a free 30-minute discovery call. I’ll scope what you need and give you a straight answer on timeline and cost.