Dashboards as code: A new approach to visualizing AWS APIs

You manage your infrastructure with code, why not manage your dashboards the same way? With Steampipe’s dashboards-as-code approach you write HCL to define dashboard widgets, and you write SQL to fill them with data extracted from APIs.

Here are some common questions about your AWS resources:

• How many resources do I have?
• How old are they?
• Are any public?
• Is encryption enabled, and if so using which keys?
• Is versioning enabled?
• How do AWS Identify and Access Management (IAM) users, groups, and policies interact?

In our last post we showed how a Steampipe “mod” runs compliance benchmarks. Here we’ll explore a new kind of mod, based on Steampipe’s dashboard subsystem. The AWS Insights mod for Steampipe is an open source tool (Apache 2.0) that defines over 650 queries and displays their results on 84 dashboards, each of which addresses questions like these.

To see your own AWS accounts in these dashboards, here are the prerequisites:

• Install Steampipe
• Install/configure the AWS plugin

Here’s the setup:

git clone https://github.com/turbot/steampipe-mod-aws-insights.git
cd steampipe-mod-aws-insights
steampipe dashboard

Now visit localhost:9194 to view and interact with the dashboards as shown here.

Screencast 1: Running the Steampipe AWS Insights dashboard

These dashboards leverage the core system we’ve discussed here and here. Steampipe’s FDW (foreign data wrapper) works with the AWS plugin to map AWS APIs to Postgres tables. Queries against those tables work concurrently across regions and accounts, enabling rapid assessment of large swaths of AWS infrastructure. Like the benchmarks and controls provided by the compliance mod, these dashboards wrap SQL queries in HCL-defined resources.

Dashboard resources

Here are the HCL resources used in the dashboards. Please see the docs for a complete overview of dashboard-related HCL resources.

Cards

The AWS Identity and Access Management (IAM) Access Key Report shown in the screencast begins with a series of cards that display individual values. This one reports that there are 5 keys whose ages are between 90 and 365 days.

HCL definition for the card:

card {
  type  = "info"
  width = 2
  sql   = query.aws_kms_key_90_365_days_count.sql
}

SQL query for the card:

query "aws_kms_key_90_365_days_count" {
 sql = <<-EOQ
 select
 count(*) as value,
 '90-365 Days' as label
 from
 aws_kms_key
 where
 creation_date between symmetric (now() - '90 days'::interval) and (now() -> '365 days'::interval);
 EOQ
}

You don’t need to know about these HCL and SQL constructs in order to use the dashboard, but it’s useful to see how easy this code is to read and write. Like the Compliance mod, this mod is straightforward to modify and extend.

Here are the other elements used in the dashboards.

Inputs

The AWS IAM User Detail dashboard is driven by a selectable list of IAM users.

HCL definition for the input:

input "user_arn" {
  title = "Select a user:"
  sql   = query.aws_iam_user_input.sql
  width = 4
}

You can define an input that uses a static list of choices, as you’d do with an HTML SELECT tag, but Steampipe uniquely affords the opportunity to populate the list of choices with the result of a SQL query.

SQL query for the input:

query "aws_iam_user_input" {
  sql = <<-EOQ
    select
      title as label,
      arn as value,
      json_build_object(
        'account_id', account_id
      ) as tags
    from
      aws_iam_user
    order by
      title;
  EOQ
}

Tables

This table provides an overview of the selected IAM user.

HCL definition for the table:

table {
  title = "Overview"
  type  = "line"
  width = 6
  query = query.aws_iam_user_overview
  args  = [ self.input.user_arn.value ]

The value of type argument is line which means the columns of this single-row table will display vertically. The args argument tells the table’s query to expect the user_arn input defined above as a parameter.

SQL query for the table:

query "aws_iam_user_overview" {
  sql = <<-EOQ
    select
      name as "Name",
      create_date as "Create Date",
      permissions_boundary_arn as "Boundary Policy",
      user_id as "User ID",
      arn as "ARN",
      account_id as "Account ID"
    from
      aws_iam_user
    where
      arn = $1
  EOQ

  param "arn" {}
}

Charts

This is the Amazon Simple Storage Service (Amazon S3) Buckets by Region chart.

HCL definition for the chart:

chart {
  title = "Buckets by Region"
  sql   = query.aws_s3_bucket_by_region.sql
  type  = "column"
  width = 4
}

Other values for the type argument: bar, donut, line, pie. If you write type = "donut" the chart looks like this.

SQL query for the chart:

query "aws_s3_bucket_by_region" {
  sql = <<-EOQ
    select
      region,
      count(i.*) as total
    from
      aws_s3_bucket as i
    group by
      region;
  EOQ
}

Flow Diagrams

This is the Attached Policies diagram for Dwight Schrute.

A flow diagram works with query results that represent nodes and edges. Here’s the data that drives this diagram.

The query that produces that data is a UNION of queries against these tables: aws_iam_user, aws_iam_group, and aws_iam_policy.

Benchmark dashboards

In our last post we showed how to run the Compliance mod, display its output in the terminal, and export the output to HTML. With the advent of Steampipe dashboards, the Compliance mod now also runs that way. The setup is the same as before.

git clone https://github.com/turbot/steampipe-mod-aws-compliance.git
cd steampipe-mod-aws-compliance

Instead of running steampipe check, run steampipe dashboard.

Screencast 2: Running the PCI benchmark as a dashboard

This screencast showcases the PCI v3.2.1 benchmark. Under the covers, Steampipe runs all the same controls – across all your AWS regions and accounts if you’ve configured a connection aggregator. In dashboard mode you can review and interact with the controls as they run.

Dashboards as code

As with the Compliance mod, the HCL and SQL files that produce this mod are version-controlled artifacts that live in an evolving public GitHub repo that you can view and – we hope! – contribute to.

There are lots of ways to build dashboards, and indeed lots of ways to do so using Steampipe. You can connect tools like Metabase and Tableau to Steampipe, and use those tools to build interactive dashboards like the ones shown here. The code that drives them lives in databases, though; you interact with it through a UX layer; and there’s no easy way to manage it as you manage your other code artifacts.

We believe dashboards are strategic assets that should be developed in a code-forward and repo-friendly way. That doesn’t mean your HCL and SQL files are static artifacts. Quite the opposite! When you’re developing a Steampipe dashboard your HCL and SQL code comes to life, with realtime feedback as you type.

Screencast 3: Live feedback while editing a dashboard

Again, you don’t need to write HCL+SQL to use this mod. But now that you’ve seen how it works, we hope you’ll be tempted to try turning some of your Steampipe queries into interactive dashboards. It’s easier than you think, it’s fun, and it’s wildly productive. If you do give it a try, please drop by our Slack workspace and show the community what you’ve done!

Resources

• Install Steampipe
• Install/configure the AWS plugin
• Install the AWS Insights mod

This post was contributed by Bob Tordella and Jon Udell from steampipe.io
The content and opinions in this post are those of the third-party authors and AWS is not responsible for the content or accuracy of this post.