Monitor a Lightsail instance using Prometheus and Grafana

Intro

The purpose of this article is to monitor the Lightsail instance using Grafana. By default, you can monitor the Lightsail instance from the AWS account, but this will require you to log into AWS and get on the Lightsail services section.

Lightsail’s monitoring info offered in AWS is not very detailed.

That is the main reason why I needed to use Grafana.

The flow is like this:

• We’ll install Prometheus with Node Exporter, to read all the data and create an endpoint for those values to be displayed
• Prometheus will act as our data source
• Grafana will read the Prometheus info as the data source and put it into a Dashboard.

We’ll follow the official AWS tutorial to add Prometheus to Lightsail but with some changes.

In this setup, we have 1 instance of Lightsail. If you have a setup with multiple instances of Lightsail using a Load Balancer, check the Monitor multiple Lightsail instances at once using Prometheus and Grafana.

This simple setup allows Node Exporter and Prometheus to run on the same machine.

Requirements

• A Lightsail instance that uses Ubuntu 20.04
• A static IP attached to the instance
• Ports 9090 and 9100 opened on the instance
  Lightsail >  Instance > Networking
  9090 is used by Prometheus, 9100 is used by Node Exporter.

Install Prometheus with Node Exporter

1. Connect to your Lightsail instance using SSH

2. Create an account named exporter for Node Exporter and prometheus for Prometheus

sudo useradd --no-create-home --shell /bin/false exporter

sudo useradd --no-create-home --shell /bin/false prometheus

3. Create local system directories

sudo mkdir /etc/prometheus /var/lib/prometheus

sudo chown prometheus:prometheus /etc/prometheus

sudo chown prometheus:prometheus /var/lib/prometheus

4. Download Prometheus binary package

• Go to Prometheus download page

Select the operating system to be linux and architecture amd64.

The right click on prometheus latest version,  “Copy link” and save it in the notepad for later.

Scroll down and do the same for node_exporter.

5. Connect via SSH to your Lightsail instance again

6. Go to home

cd ~

7. Download the prometheus binary package to your instance

curl -LO prometheus_url_address_you_saved_in_notepad

Example:

curl -LO https://github.com/prometheus/prometheus/releases/download/v2.52.0/prometheus-2.52.0.linux-amd64.tar.gz

8. Download the node_exporter binary package to your instance

curl -LO node_exporter_url_address_you_saved_in_notepad

9. Extract the contents for the downloaded packages one by one for Prometheus and Node Exporter
Make sure the archive names match the ones in the links you copied in step 4.

tar -xvf prometheus-2.52.0.linux-amd64.tar.gz

tar -xvf node_exporter-1.8.1.linux-amd64.tar.gz

10. Copy the extracted prometheus and promtool to the /usr/local/bin directory

sudo cp -p ./prometheus-2.52.0.linux-amd64/prometheus /usr/local/bin

sudo cp -p ./prometheus-2.52.0.linux-amd64/promtool /usr/local/bin

11. Change ownership of prometheus and promtool folders to the prometheus user we created at the start

sudo chown prometheus:prometheus /usr/local/bin/prom*

12. Copy the consoles and console_libraries to /etc/prometheus. Make it recursive copy of all directories within the hierarchy using -r

sudo cp -r ./prometheus-2.52.0.linux-amd64/consoles /etc/prometheus

sudo cp -r ./prometheus-2.52.0.linux-amd64/console_libraries /etc/prometheus

13. Change ownership of the copied folders to prometheus user

sudo chown -R prometheus:prometheus /etc/prometheus/consoles

sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries

14. Copy the configuration file prometheus.yml to /etc/prometheus and change ownership to prometheus user

sudo cp -p ./prometheus-2.52.0.linux-amd64/prometheus.yml /etc/prometheus

sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml

15. Copy the node_exporter file from ./node_exporter* to /usr/local/bin

sudo cp -p ./node_exporter-1.8.1.linux-amd64/node_exporter /usr/local/bin

16. Change ownership of the file to the exporter user

sudo chown exporter:exporter /usr/local/bin/node_exporter

Configure Prometheus

1. Connect via SSH to your Lightsail instance again

2. Create a backup copy to your prometheus.yml config file

sudo cp /etc/prometheus/prometheus.yml /etc/prometheus/prometheus.yml.backup

3. Open the prometheus.yml config file in Vim

sudo vim /etc/prometheus/prometheus.yml

Quickly edit with Vim:

• Press i to enter insert mode and make changes to the file
• Press ESC to exit insert mode
• Press : to enter command mode and add wq! to save the file

Here are some more essential commands for Vim.

4. Edit the prometheus.yml 

• scrape_inteval – this defines how often Prometheus will scrape/collect data from the targets available
• job_name – This is used to identify the exporters available for Prometheus
• targets – These will point to the actual machines from where you’re going to scrape data. They are written in the format IP:PORT.

We’re going to edit the default prometheus.yml file by:

• Setting the correct static  IP on the prometheus job_name
• Adding a basic_auth for the prometheus
• Creating a job_name for node_exporter that we’re going to name it node
• Adding a basic_auth for node

# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
       - targets:
         # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["your_static_IP:9090"]

     basic_auth: 
       username: prometheus_uberadmin 
       password: password123

  - job_name: "node"

    static_configs:
      - targets: ["your_static_IP:9100"]

    basic_auth:
      username: node_exporter_uberadmin
      password: password123

5. Save the file

Press the Esc key to exit insert mode, and type :wq! to save your changes and quit Vim.

6. Create a file that will hold the basic_auth for Prometheus

sudo vim /etc/prometheus/web.yml

Add the following text to the file and save it.

basic_auth_users:
  prometheus_uberadmin: $2a$12$GxinAvcHwkGQclBiJmz6ce5is67Bxj7mvnKyY4L0DxJooqFgA7hvu

Note: the password needs to be generated using a bcrypt hash of the string you want to use.
Use an online bcrypt tool for that if it’s easier.

The example above is the hash for password123.

7. Save the file

Press the Esc key to exit insert mode, and type :wq! to save your changes and quit Vim.

8. Start Prometheus

sudo -u prometheus /usr/local/bin/prometheus --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /var/lib/prometheus --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries

9. After the running service is validated, press Ctrl+C to stop it.

10. Open systemd configuration file in Vim

sudo vim /etc/systemd/system/prometheus.service

11. Insert the following lines into the file

[Unit]
Description=PromServer
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file /etc/prometheus/prometheus.yml \
--storage.tsdb.path /var/lib/prometheus/ \
--web.config.file /etc/prometheus/web.yml \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

The instructions above are going to be used by Linux to start Prometheus on the server.

12. Press the Esc key to exit insert mode, and type :wq! to save your changes and quit Vim.

13. Load the new information into the systemd

sudo systemctl daemon-reload

14. Restart Prometheus

sudo systemctl start prometheus

15. Check Prometheus status

sudo systemctl status prometheus

16. Press Q to exit the status

17. Enable Prometheus on instance boot

sudo systemctl enable prometheus

18. Check Prometheus in the browser

Go to HTTP://your_static_ip:9090

You will be prompted to add a user and password. Use the ones you set in the basic_auth.

After you will be redirected to the Prometheus dashboard.

19. Check the targets

Go to Status > Targets

In there, you should now see the node job listed.

Configure Node Exporter

1. Connect via SSH to your Lightsail instance again

2. Create a file that will hold the basic_auth for Node Exporter

First, create a folder for the file in /etc/

sudo mkdir /etc/node_exporter

Create and edit the file

sudo vim /etc/node_exporter/web.yml

Add the following text to the file and save it.

basic_auth_users:
  node_exporter_uberadmin: $2a$12$GxinAvcHwkGQclBiJmz6ce5is67Bxj7mvnKyY4L0DxJooqFgA7hvu

Note: the password needs to be generated using a bcrypt hash of the string you want to use.
Use an online bcrypt tool for that if it’s easier.

The example above is the hash for password123.

3. Save the file

Press the Esc key to exit insert mode, and type :wq! to save your changes and quit Vim.

4. Create a systemd service for node_exporter in Vim

sudo vim /etc/systemd/system/node_exporter.service

5. Press the i key to enter insert mode in Vim

6. Edit the file as follows to collect all the available data from the server

[Unit]
Description=NodeExporter
Wants=network-online.target
After=network-online.target


[Service]
User=exporter
Group=exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
--web.config.file="/etc/node_exporter/web.yml"

[Install]
WantedBy=multi-user.target

7. Save the file

Press the Esc key to exit insert mode, and type :wq! to save your changes and quit Vim.

8. Reload systemd process

sudo systemctl daemon-reload

9. Start the node_exporter

sudo systemctl start node_exporter

10. Check node_exporter status

sudo systemctl status node_exporter

11. Press Q to exit the status command

12. Enable Node Exporter on instance boot

sudo systemctl enable node_exporter

Configure Grafana

I assume you already have Grafana installed. In this tutorial, we’re skipping the part about installing Grafana.

You can check the Resources at the bottom where we have some links on how to install Grafana.

I recommend having Grafana on a different machine since you will use it in the future to monitor different servers probably.

1. Add Data source
Go to Data sources > Add new data source

2. Select a Prometheus data source

3. Add URL and credentials

Complete the Prometheus server URL with the HTTP://static_ip:9090

And on Authentication select Basic Authentication and add the credentials we created. In our case prometheus_uberadmin and password123.

Scroll down and click Save & Test.

4. Create a Dashboard

Go to Dashboards > New > New Dashboard

4. Click on Import a dashboard

You can go to this page and find a dashboard that matches your needs https://grafana.com/grafana/dashboards

I recommend one of these 2 to get a good overview of your instance in Grafana:

• Node Exporter Dashboard EN 20201010-StarsL.cn
• Node Exporter for Prometheus Dashboard based on 11074

Paste the ID of the dashboard in the import section and load it.

5. Select the Data source

Add a name for the Dashboard and select the Prometheus data source we created in step 3.

Once you click Import a new dashboard will be generated with the details pulled from Prometheus.

Conclusion

Following these steps will allow you to install Prometheus + Node Exporter on a Lighsail instance from AWS and feed that into a Grafana dashboard.

You can check Grafana’s documentation on how you can do more, or tweak the existing dashboard.

As mentioned before this is an example with a single Lightsail instance.

In a future article, I will explain how you can do it on multiple Lightsail instances and feed that information on a single Grafana dashboard. That would be very useful if you have a setup with a Load balancer in Lightsail.

Resources

The starting point for this article was the tutorial provided by AWS on how to get started with Prometheus on Lightsail.

• Tutorial: Install Prometheus on a Linux-based Lightsail instance
• Install Grafana
• Grafana Docker Compose