The author voluntarily contributed this tutorial as a part of Pepipost Write to Contribute program.
As a developer, many times you would have worried, whether your services are up and running or not. Not only that, sometimes as an infrastructure guy you might be also worried about your server’s health too. What is the current RAM or disk utilization? or whether they are going to be fully occupied which in turn can completely bring the system down. These are just the basics and in fact there are tons of more such things which need to be monitored and fixed in everyday's life.
“Never send a human to do a machine's job”
At some time, deploying humans to check all these and that too for 24*7 can be a real pain.
That's exactly the purpose of writing this Prometheus monitoring tutorial series around monitoring and alerting. In this tutorial, you will learn how to set up Prometheus as a universal monitoring system and how to use its exporter to define and fetch the metrics which are really important to track.
Prometheus is an open-source system for monitoring and alerting tool. It is developed in the GO language. Prometheus tool is currently a standalone open source project and maintained independently by any organization. You can check more details here.
Here Is This Tutorial, Prometheus Is Installed And Tested On A Test Server With Minimum Configuration:
There are multiple ways to install Prometheus. You can use Prometheus docker image or use any of the available configuration management systems like Ansible, chef, puppet and salt stack. For more information on installation, visit the official installation guide here.
Here, is the link to download the docker image for Prometheus.
It also has pre-compiled binaries available. I am going to use this binary for installation because it is easy to set up and easy to understand because of we already familiar with the utilization of binary files.
Visit Prometheus download page. It will give you a list of pre-compiled binaries for drawins, linux, and windows. You can download according to your OS. Below, the installation is explained for Linux OS.
You can simply fire below command in your Linux terminal:
You will see a file with name prometheus-2.11.1.linux-amd64.tar.gz will get downloaded.
tar -xvzf prometheus-2.11.1.linux-amd64.tar.gz
$ mv prometheus-2.11.1.linux-amd64 prometheus $ cd prometheus/$ ll
Folder contains below file:
Visit localhost:9090 on your web browser:
Your Prometheus is up and running!
If you notice in prometheus/ folder, It created a folder with the name ‘data’. Prometheus starts storing metrics in this /data folder only.
Now get all metric list by hitting the URL to localhost:9090/metrics
Here is a sample production command:
~/prometheus/prometheus --storage.tsdb.path=/var/lib/prometheus/data/ --web.external-url=http://myurl.com:9090
--storage.tsdb.path: Specify the path where you want to save Prometheus data.
--web.external-url: You can use this option if you want to bind your address with your URL.
You can get below error in case of your folder don’t have appropriate permission:
level=error ts=2019-08-06T14:25:19.791Z caller=main.go:731 err="opening storage failed: lock DB directory: open /var/lib/lock: permission denied"
You can try appending “sudo” to your command OR you can give appropriate permission to your folder.
1) Create a file:
2) Just paste below code:
[Unit] Description=Prometheus Server Documentation=https://prometheus.io/docs/introduction/overview/ After=network-online.target [Service] User=root Restart=on-failure #Change this line if you download the #Prometheus on different path user ExecStart=~/prometheus/prometheus --storage.tsdb.path=/var/lib/prometheus/data/ --web.external-url=http://myurl.com:9090 [Install] WantedBy=multi-user.target
Save and exit.
3) Reload the Systemctl Daemon:
sudo systemctl daemon-reload
4) Start the Prometheus service:
sudo systemctl start prometheus
Till now you learned how to do basic Prometheus setup. Now, you will learn, how to set up Prometheus exporter.
Exporters can be any scripts or services which will fetch specific metrics from your system and gives data in Prometheus format. There are primarily two ways by which you can fetch metrics and store into Prometheus:
You can use both methods but usually, people prefer the first one to fetch metrics.
So now we are going to setup node exporter. It will fetch your server metrics which will be RAM/DISK/CPU utilization, network, io etc.
$ wget https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz $ tar -xvzf node_exporter-0.18.1.linux-amd64.tar.gz $ mv node_exporter-0.18.1.linux-amd64 node_exporter $ cd node_exporter $ ./node_exporter
You should see below output once the node exporter is started:
Just visit to localhost:9100/metrics
Create a file in below path:
Just paste below code:
[Unit] Description=Node exporter After=network-online.target [Service] User=root Restart=on-failure #Change this line if you download the #Prometheus on different path user ExecStart=~/node_exporter/node_exporter [Install] WantedBy=multi-user.target
Reload the systemctl daemon:
sudo systemctl daemon-reload
Start the Prometheus service:
sudo systemctl start node-exporter
Open file ~/prometheus/prometheus.yml add below configuration:
scrape_configs: # The job name is added as a label job=<job_name> to any timeseries scraped from this config. - job_name: 'prometheus' # metrics_path defaults to '/metrics' # scheme defaults to 'http'. static_configs: - targets: ['localhost:9090'] # Add below block for node_exporter - job_name: node_exporter scrape_interval: 1m scrape_timeout: 1m metrics_path: "/metrics" static_configs: - targets: ['localhost:9100']
Save and exit.
~/prometheus/prometheus --storage.tsdb.path=/var/lib/prometheus/data/ --config.file=~/prometheus/prometheus.yml --web.external-url=http://myurl.com:9090
Also, don’t forget to make the same changes in your service file:/etc/systemd/system/prometheus.service
sudo systemctl restart prometheus
Now visit the URL localhost:9090. In Expression field you can search for “node_filesystem_size_bytes” by clicking on the “Execute” button. You will get the below stats:
After clicking on the graph:
Like this, you can explore each metrics like memory, CPU, etc.
There are already lots of exporters is available on the internet like Nginx exporter, MongoDB exporter, MySQL server exporter, etc. Just download them and start using it. You can check more info about exporter here.
You have now successfully installed Prometheus and node exporters.
If you see this official architecture diagram, we have successfully set up the Prometheus, Pushgateway/Exporters, and Prometheus UI.
In the next part, we will see how we can set up an alert manager and how to setup alert over the metrics.
Stay tuned for the new updates.
If you loved the tutorial, rate us and leave a comment below.
NOTE: PROMETHEUS IS METRIC STORAGE SYSTEM. DON’T TRY TO STORE ANY KIND OF LOGS. IT IS NOT RECOMMENDED.