Using AWS Managed Grafana with Timestream for Observability: Infrastructure Setup

Network Ninja software engineers recently implemented the AWS Managed Grafana Service as a fully managed internal dashboarding solution, allowing us, among other things, to monitor network traffic across hundreds of applications (example of the final result below).

Screenshot of AWS managed Grafana network traffic dashboard.

We chose Grafana because:

It allows us to visualize and analyze data from multiple sources,
It co-exists with our AWS infrastructure (allowing us to manage access via IAM policies), and
It supports single sign-on (SSO).

Since there wasn’t a comprehensive guide to accomplishing what we wanted, we wrote one - and broke it up into three digestible sections. Part 1 (which follows below) covers setting up Timestream and connecting it to Grafana, while Parts 2 and 3 will explain Timestream Queries and Scheduled Tasks & Graph Annotations, respectively.

Tech Infrastructure

For simplicity, we chose to use a combination of AWS Timestream and Cloudwatch for our metrics datastore. Using a push-based model instead of polling means no inventory and service discovery to worry about - that works well in a cloud environment where instances may come and go. We also like the balance of memory and magnetic storage, since most of the time when we’re looking at dashboards, we’re looking at recent history.

If there’s a downside to using Timestream in Grafana, it’s the lack of a query builder. However, in practice, building the SQL queries for the dashboard panels is fairly straightforward. We also went with Telegraf to collect the metrics on our systems. It supports Timestream as a backend directly and has a ton of plugins, including a statsd plugin for any ad hoc metrics.

TL;DR

This guide covers the setup of AWS Managed Grafana using:

Timestream for time-series data,
Cloudwatch for application and infrastructure monitoring, and
Telegraf as the service agent to collect and send metrics from our applications.

Step 1: Configuring Timestream

Create your Timestream database(s). Note that:

Both Cloudformation and Terraform support Timestream.
You can also create Timestream databases through the AWS console.
You will need to create a Timestream database per region.

Next, create (or augment your existing) EC2 instance roles to allow writing to your Timestream database. We chose to allow our instances to create tables, so we could easily add new stats. Here’s an example policy with needed permissions and a resource constraint on the specific Timestream database:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "timestream:DescribeDatabase",
      "Effect": "Allow",
      "Resource": "arn:aws:timestream:::database/my-timestream-db"
    },
    {
      "Action": "timestream:WriteRecords",
      "Effect": "Allow",
      "Resource": "arn:aws:timestream:::database/my-timestream-db/table/*"
    },
    {
      "Action": [
        "timestream:CreateTable",
        "timestream:TagResource"
        ],
      "Effect": "Allow",
      "Resource": "arn:aws:timestream:::database/my-timestream-db/table/*"
    },
    {
      "Action": "timestream:DescribeEndpoints",
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

Don’t forget to attach your roles, if needed.

Step 2: Create an AWS Managed Grafana Instance

Navigate to Amazon Managed Grafana in the console and click Create workspace.

Screenshot of using AWS managed Grafana console to create a workspace.

Screenshot of using AWS managed Grafana console to name a workspace.

Choose your settings and click Next. We chose to let AWS manage the permissions.

Screenshot of using AWS managed Grafana console to name a workspace.