Setting up disk alerts for disk utilization

Objectives

Getting timely alerts about resource utilization is key to managing your database, avoiding problems, and mitigating downtime. If you know in advance that your database is running out of disk, take steps to scale those resources.

In this tutorial, you use the IBM Cloud API and the IBM Cloud CLI to set up an alert that emails you whenever the disk utilization of your database exceeds 90%. This specific example creates an alert on a Databases for Elasticsearch deployment, but it is applicable to all the databases in the IBM Cloud Databases catalog.

Getting productive

ibmcloud plugin install monitoring

ibmcloud plugin install cloud-databases

Retrieve your monitoring service instance

In this step, you retrieve the necessary credentials to gain access to your monitoring instance.

Begin by logging in to the IBM Cloud CLI with the following command:

ibmcloud login -sso

Follow the on-screen instructions to log in.

Next, target the appropriate region for your instance:

ibmcloud target -r <REGION>

Then, list the existing monitoring instances in that region with the following command:

ibmcloud monitoring service-instances

Note the service instance name that has “Platform Metrics” enabled.

Now create an authorization token for the API, by using a command like:

The following steps work only on Bash.

AUTH_TOKEN=$(ibmcloud iam oauth-tokens | awk '{print $4}')

You can now retrieve the ID of your monitoring service instance for the API with the following command:

GUID=$(ibmcloud resource service-instance <instance_name_from_step_above> --output json | jq -r '.[].guid')

Set up the notification channel

Configure a notification channel to be notified when an alert is triggered. To set up your notification channel, use the following command:

curl -X POST https://<region>.monitoring.cloud.ibm.com/api/notificationChannels -H "Authorization: Bearer $AUTH_TOKEN" -H "IBMInstanceID: $GUID" -H "content-type: application/json"  --data-raw '{"notificationChannel":{"id":null,"version":null,"teamId":"","name":"<notification_channel>","type":"EMAIL","enabled":true,"sendTestNotification":true,"options":{"notifyOnOk":true,"notifyOnResolve":true,"emailRecipients":["email@email.com"]}}}'

You see the following output:

{"notificationChannel":{"id":39209,"version":1,"customerId":34292,"enabled":true,"sendTestNotification":true,"createdOn":1678967870764,"modifiedOn":1678967870764,"name":"thursTest","options":{"notifyOnOk":true,"emailRecipients":["email@email.com"],"notifyOnResolve":true},"type":"EMAIL"}}% 

You have now created a notification channel for your alerts.

Make a note of the id field that is returned by the API call.

Create the alert

Now that you have a notification channel, create your alert rule. Your alert rule describes the metric query to be monitored, the threshold value, and the action to take when the threshold is crossed. In this case, you're monitoring your ibm_service_instance_name to ensure that max utilization doesn't exceed 90%. If that happens, an alert is triggered and you're notified.

This alert is triggered at 90% disk utilization. However, 50-70% disk utilization is preferred.

To retrieve the name of the database instance you want to set up the alert for, list all your database instances with a command like:

ibmcloud cdb ls

Make sure to select a database in the same region as the monitoring instance.

You see output like the following:

Retrieving instances for all database types in all resource groups in all locations under IBM as …
OK
Name                          Location   State
Databases for PostgreSQL-76   us-south   inactive
testelastic                   eu-gb      active
Databases for MySQL-9j        us-south   active

Now, use the name of your database to create the alert by using a command like:

curl --request POST \
  --url https://<region>.monitoring.cloud.ibm.com/api/alerts \
  --header "Authorization: Bearer  $AUTH_TOKEN" \
  --header 'Content-Type: application/json' \
  --header "IBMInstanceID: $GUID" \
  --data-raw  '{
	"alert":
		{
			"type": "MANUAL",
			"name": "Disk Alert",
			"description": "",
			"enabled": true,
			"severity": 1,
			"timespan": 60000000,
			"notificationChannelIds": [
				<id_from_previous_step>
			],
			"filter": "ibm_service_instance_name in (\"<db_instance_name_from_previous_step>\")",
		  "condition": "max(max(ibm_databases_for_elasticsearch_disk_used_percent)) > 0.9"
		}
}'

This command takes the max disk utilization of any member available, regardless of the number of members.

Check that your alert is created

An alert is created whenever you reach 90% of disk size. You receive the alert to the same email that you created in the notification channel. You can also use the following command to check current active alerts:

ibmcloud monitoring alert list --name <monitoring instance name>

You now receive an alert whenever your Databases for Elasticsearch instance disk utilization exceeds 90%, so you can act before the disk is too full.

Next Steps

To modify your alert or find out more about Monitoring, see Getting started withIBM Cloud Monitoring.

Cloud Databases service metrics

This tutorial uses IBM Cloud® Databases for Elasticsearch. However, the same process applies to other Cloud Databases services:

• IBM Cloud® Databases for DataStax
• IBM Cloud® Databases for EnterpriseDB
• IBM Cloud® Databases for etcd
• IBM Cloud® Databases for MongoDB
• IBM Cloud® Databases for PostgreSQL
• IBM Cloud® Databases for Redis
• IBM Cloud® Databases for MySQL
• IBM Cloud® Messages for RabbitMQ