Running Grafana Enterprise on Fly.io

Engineer looking at Grafana dashboard

Fly.io offers a managed Grafana dashboard, an invaluable tool for monitoring your applications. While Fly's out-of-the-box offering is undeniably handy, there's limitations to be made aware of:

  • Fly.io deploys the Open Source version of Grafana, which unfortunately means missing out on some advanced features, such as alerting.
  • Customization is somewhat restricted; for instance, modifications to the grafana.ini file aren't possible.

In this article I'll guide you through the steps to deploy Grafana Enterprise as a Fly app, so you can leverage the full power of Grafana Enterprise. Once we have Grafana Enterprise up and running, I'll show you how to get Fly metrics that we have on fly-metrics.net into our custom Grafana instance. Lastly, I'll show you how to copy the dashboard from fly-metrics.net to our Grafana Enterprise instance.

Creating the app

Let's start with the basic configuration for running Grafana enterprise on Fly. Create a file called fly.toml in the root of your project and add a file called fly.toml with the following contents:

[build]
  image="grafana/grafana-enterprise:10.0.3-ubuntu"

[[mounts]]
  source="grafana_data"
  destination="/var/lib/grafana"

[http_service]
  internal_port = 3000
  force_https = true
  auto_stop_machines = false
  auto_start_machines = false

The image we're using is specified in the [build] section. In this case we are using the Ubuntu version of Grafana Enterprise. There's also an Alpine version, but I chose to use the Ubuntu version because that's what I am most familiar with. As of writing, 10.0.3 is the latest version. Grafana stores its data in /var/lib/grafana, so to be able to persist the data across restarts we mount a volume called grafana_data on that directory. Lastly, in the [http_service] section we configure the port that Grafana listens to, which is port 3000.

You can now deploy using the following command:

fly launch --copy-config --auto-confirm --ha=false --now

This creates a new app based off the existing fly.toml file. A 1GB volume will be created to store the Grafana data and lastly, the app will be deployed to a single instance. Run fly open to check out your Grafana Enterprise installation. You can log in with username admin and password admin.

Configuring Grafana

The easiest way to configure Grafana is by defining environment variables. E.g. to change the default color mode to 'light', we can define the following environment variable in our fly.toml file.

[env]
  GF_DEFAULT_THEME="light"

Similarly, we can use Fly secrets to configure sensitive information that we might not want in our codebase:

fly secrets set GF_SMTP_PASSWORD='secr3t!'

For detailed information about configuring Grafana, see the Grafana documentation.

Getting Fly metrics into Grafana

Now we're getting to the fun part. Fly exposes an API endpoint from which we can fetch Prometheus metrics. We can add this as a data source to our new Grafana instance. On your Grafana Enterprise instance, press "Connections", search for "Prometheus" and click "Create a Prometheus data source". There's a few important input fields:

  • The name should be prometheus_on_fly. This name is important. I will get to that later.
  • Prometheus server URL: https://api.fly.io/prometheus/<org-slug>. The organization slug you can find by running fly orgs list.
  • Custom HTTP Headers: Add a customer header called Authorization and value Bearer <api-token>. You can get the API token by running fly auth token.

Press Save & test. If everything went well, you should see a green message saying "Successfully queried the Prometheus API". Now we have access to all the metrics that we have on fly-metrics.net!

Copying the dashboard from fly-metrics.net

Although we have access to the data, we don't have the cool dashboard yet that we see on fly-metrics.net. Luckily, it's quite easy to transfer this to our new instance.

Open fly-metrics.net and navigate to the dashboard that you want to copy. Next, press the 'Dashboard settings' button in the top-right corner. By default, you should see a message along of the lines of "Dashboard not editable". Press "Make editable" and save. After refreshing, you will see a sidebar menu with a bunch of options. Press "JSON model" and copy the complete JSON object. Now go back to your Grafana Enterprise instance, go to Dashboards and press "New" -> "Import". Paste the JSON object into the "Import via panel json" field and press "Load". Voilà!

In the previous section I mentioned that the name of the Prometheus data source is important. This is because if you look carefully at the JSON model you will see references to a data source called prometheus_on_fly. If you had chosen a different name for your data source, you would have gotten errors. If you want to change the name of the data source, that's fine, but you will have to change all references of prometheus_on_fly to the name of your data source.

Conclusion

In this article I showed you how to deploy Grafana Enterprise on Fly. We configured it in such a way that everything from fly-metrics.net can be found on our Grafana Enterprise instance. You can now start leveraging all the cool features that Grafana Enterprise has to offer and configure it to your liking. You can for example add plugins that you like or connect other data sources such as your Postgres or MySQL database.

The full source code used in this article can be found on GitHub.

If you have feedback or questions, please reach out to me on Twitter: @dejagersh.