Integrating Kubecost with Amazon Managed Service for Prometheus

This blog post was co-written by Linh Lam, Solution Architect, Kubecost

Customers can track their Kubernetes control plane and Amazon Elastic Compute Cloud (Amazon EC2) costs using AWS Cost and Usage Reports. However, they often need deeper insights to accurately track Kubernetes costs across namespaces, clusters, pods, and more. We recently announced that AWS and Kubecost collaborated to deliver cost monitoring for EKS customers. Today, in partnership with Kubecost, we are launching a streamlined integration between Kubecost and Amazon Managed Service for Prometheus. This approach provides cluster operators with Amazon Elastic Kubernetes Service (Amazon EKS) cost insights powered by Kubecost for a single Amazon EKS cluster and backed by a scalable Amazon Managed Service for Prometheus workspace.

Amazon Managed Service for Prometheus is a Prometheus-compatible monitoring and alerting service that makes it easy to monitor containerized applications and infrastructure at scale. You can use the open-source Prometheus query language to monitor and alert for the performance of containerized workloads without having to worry about scaling the underlying monitoring infrastructure. The service automatically scales the ingestion, storage, alerting, and querying of operational metrics as workloads grow or shrink. Furthermore, it’s integrated with AWS security services to enable fast and secure access to data. This lets you concentrate on your workloads instead of having to manage your monitoring stack.

Kubecost is built on OpenCost, which was recently accepted as a Cloud Native Computing Foundation (CNCF) Sandbox project, and is actively supported by AWS. Kubecost provides fine-grained visibility into your cluster, letting you break down costs by Kubernetes resources, such as pods, nodes, namespaces, and labels. This costs-visibility allows teams to have transparent and accurate cost data based on their actual AWS bill.

Now let’s configure Kubecost to be backed by an Amazon Managed Service for Prometheus workspace.

Prerequisites

To get started with this post, you should have the following:

• An Amazon Elastic Kubernetes Service (Amazon EKS) cluster.
• Helm 3.9+, kubectl, eksctl, jq, and the awscli.
• AWS Identity and Access Management (IAM) credentials with permissions to create IAM roles.
• An IAM OIDC provider created for the Amazon EKS cluster.

Installing Kubecost

If your cluster is running Kubernetes 1.23 or above, you will need to install the Amazon EBC CSI driver to your cluster. This driver allows your cluster to manage the lifecycle of Amazon EBS volumes for persistent volumes. Run the following command to create an IAM service account with the policies needed to use the Amazon EBS CSI Driver.

eksctl create iamserviceaccount   \
    --name ebs-csi-controller-sa   \
    --namespace kube-system   \
    --cluster <CLUSTER_NAME>   \
    --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy  \
    --approve \
    --role-only \
    --role-name AmazonEKS_EBS_CSI_DriverRole

Where <CLUSTER_NAME> is the name of your Amazon EKS cluster.

Export the ARN of the AmazonEKS_EBS_CSI_DriverRole, which was created by the previous command:

export SERVICE_ACCOUNT_ROLE_ARN=$(aws iam get-role --role-name AmazonEKS_EBS_CSI_DriverRole | jq -r '.Role.Arn')

Now install the Amazon EBS CSI add-on for EKS using the AmazonEKS_EBS_CSI_DriverRole by issuing the following command:

eksctl create addon --name aws-ebs-csi-driver --cluster <CLUSTER_NAME> \
    --service-account-role-arn $SERVICE_ACCOUNT_ROLE_ARN --force

Where <CLUSTER_NAME> is the name of your Amazon EKS cluster. To verify that the Amazon EBS CSI driver add-on is working as expected, run the following command:

kubectl get pods -n kube-system | grep ebs

This command should list several running pods.

Now install Kubecost on the Amazon EKS cluster by running the following command:

helm upgrade -i kubecost \
oci://public.ecr.aws/kubecost/cost-analyzer --version <VERSION> \
--namespace kubecost --create-namespace \
-f https://tinyurl.com/kubecost-amazon-eks

Where <VERSION> is the current version of the kubecost/cost-analyzer chart. At the time of publication, the latest version is 1.98.0. You can find all available versions of the EKS optimized Kubecost bundle here. We recommend finding and installing the latest available Kubecost cost analyzer chart version.

Next, you must set up IAM roles for Kubecost service accounts (IRSA). Using the OIDC provider for the cluster, you grant IAM permissions to your cluster’s service accounts. You must grant appropriate permissions to the kubecost-cost-analyzer and kubecost-prometheus-server service accounts. These will be used to send and retrieve metrics from the workspace. Run the following commands on the command line:

eksctl create iamserviceaccount \
    --name kubecost-cost-analyzer \
    --namespace kubecost \
    --cluster <CLUSTER_NAME> --region <REGION> \
    --attach-policy-arn arn:aws:iam::aws:policy/AmazonPrometheusQueryAccess \
    --attach-policy-arn arn:aws:iam::aws:policy/AmazonPrometheusRemoteWriteAccess \
    --override-existing-serviceaccounts \
    --approve

eksctl create iamserviceaccount \
    --name kubecost-prometheus-server \
    --namespace kubecost \
    --cluster <CLUSTER_NAME> --region <REGION> \
    --attach-policy-arn arn:aws:iam::aws:policy/AmazonPrometheusQueryAccess \
    --attach-policy-arn arn:aws:iam::aws:policy/AmazonPrometheusRemoteWriteAccess \
    --override-existing-serviceaccounts \
    --approve

<CLUSTER_NAME> is the name of the Amazon EKS cluster where you want to install Kubecost and <REGION> is the region of the Amazon EKS cluster. Note that these commands each generate an AWS CloudFormation stack that creates a new IAM role, assigns the AmazonPrometheusQueryAccess and AmazonPrometheusRemoteWriteAccess managed policies, and configures a trust relationship with the OIDC provider for the cluster. This allows the service account to assume the IAM role.

If you don’t yet have an Amazon Managed Service for Prometheus workspace, then the following AWS Command Line Interface (AWS CLI) command will create a workspace called kubecost-amp in the current region. You may skip this step if you have a pre-existing workspace.

aws amp create-workspace --alias kubecost-amp --region <REGION>

<REGION> is the region where you want the workspace to be created. Now you can update the Kubecost configuration so that it uses your Amazon Managed Service for Prometheus workspace as the source for your cluster metrics. Run the following command from the command line:

export AWS_REGION=<REGION>
export AMP_WORKSPACE_ID=<WORKSPACE-ID>

Where <WORKSPACE-ID> is the Workspace ID of the cluster and <REGION> is the current region of the workload. You can get the Workspace ID of the workspace from the Summary section of the Amazon Managed Service for Prometheus console. See the following figure.

Figure 1: The Amazon Managed Service for Prometheus workspace detail, which lists the workspaces ARN, Workspace ID, Endpoint – remote write URL, and the Endpoint – query URL

Run the following command to create a file called config-values.yaml, which contains the defaults that Kubecost will use for connecting to your Amazon Managed Service for Prometheus workspace.

cat << EOF > config-values.yaml
global:
  amp:
    enabled: true
    prometheusServerEndpoint: http://localhost:8005/workspaces/${AMP_WORKSPACE_ID}
    remoteWriteService: https://aps-workspaces.${AWS_REGION}.amazonaws.com/workspaces/${AMP_WORKSPACE_ID}/api/v1/remote_write
    sigv4:
      region: ${AWS_REGION}

sigV4Proxy:
  region: ${AWS_REGION}
  host: aps-workspaces.${AWS_REGION}.amazonaws.com
EOF

Now, run the following command to configure Kubecost to begin using your workspace:

helm upgrade -i kubecost \
oci://public.ecr.aws/kubecost/cost-analyzer --version <VERSION> \
--namespace kubecost --create-namespace \
-f https://tinyurl.com/kubecost-amazon-eks \
-f config-values.yaml

<VERSION> is the current version of the kubecost/cost-analyzer chart. At the time of publication, the latest version is 1.98.0.

Finally, restart the Prometheus deployment, which reloads the service account configuration.

kubectl rollout restart deployment/kubecost-prometheus-server -n kubecost

After a few minutes, your Kubecost cluster should be available and ready to view.

You can enable port-forwarding to expose the Kubecost dashboard:

kubectl port-forward deployment.apps/kubecost-cost-analyzer 9090:9090 -n kubecost

The dashboard will be available to view here.

Figure 2: The Kubecost dashboard, backed by an Amazon Managed Service for Prometheus workspace

Conclusion

We’re excited that this release allows customers to store and analyze their Kubecost metrics using Amazon Managed Service for Prometheus as a backend. This release helps customers better support cost monitoring for their Amazon EKS workloads without having to deal with managing the monitoring stack.

Using Kubecost in your Amazon EKS workloads lets you better monitor costs associated with containerized workloads. Your Amazon EKS cost insights are powered by Kubecost and backed by a scalable Amazon Managed Service for Prometheus workspace.

If you need support, you can submit a support request via AWS Support.

If you would like to learn more from the Kubecost team, contact them here.

Get started today by integrating your workload with Amazon EKS cost monitoring.

About the authors: