Introduction
Monitoring MongoDB in production is essential for maintaining database availability, performance, and reliability. Percona Monitoring and Management (PMM) provides built-in alert templates, custom alert rules, dashboards, and notification integrations to help database administrators identify potential issues before they impact applications.
In this article, we will walk through configuring MongoDB alerts in PMM 3.9.1, setting up email notifications through Grafana, and creating alert rules for replica set health and database performance.
Configure Gmail SMTP for PMM Email Alerts
PMM uses Grafana's SMTP configuration to send email notifications. In this implementation, Gmail SMTP is used instead of AWS SNS.
Step 1: Generate a Gmail App Password
A Gmail App Password allows PMM to authenticate with Gmail SMTP without using your regular Google account password.
Sign in to your Google account.
Enable 2-Step Verification if it is not already enabled.
Open Google Account security settings.
Navigate to App passwords.
Generate an app password for the PMM notification service.
Copy the generated password securely.
App Passwords may not be available for some Google Workspace accounts or accounts using certain security configurations.
Important: Never publish your App Password in a blog, screenshot, configuration example, or public repository.
Step 2: Configure Gmail SMTP
Configure Grafana's SMTP settings with the following values:
| Setting | Value |
|---|---|
| SMTP host | smtp.gmail.com:587 |
| SMTP authentication user | Your Gmail address |
| SMTP password | Gmail App Password |
| Security | STARTTLS |
| Skip TLS verification | false |
| From address | Your Gmail address |
| From name | PMM MongoDB Alerts |
For a Docker-based PMM installation, SMTP settings can be supplied through Grafana environment variables when deploying the PMM Server container.
Example configuration:
GF_SMTP_ENABLED=true
GF_SMTP_HOST=smtp.gmail.com:587
GF_SMTP_USER=your-email@gmail.com
GF_SMTP_PASSWORD=YOUR_GMAIL_APP_PASSWORD
GF_SMTP_SKIP_VERIFY=false
GF_SMTP_FROM_ADDRESS=your-email@gmail.com
GF_SMTP_FROM_NAME=PMM MongoDB AlertsReplace the example email address and password with your own values. Store credentials securely and avoid committing them to source control.
The password should be entered without the display spaces Google may insert for readability.
Note: For an existing PMM Docker deployment, verify the supported configuration and apply the settings using an appropriate container configuration or redeployment procedure. Do not delete the persistent PMM data volume.
1. Configure Email Notifications in PMM
The first step is to configure a contact point for receiving alert notifications.
Navigate to:
Alerts → Contact points → Create contact point
Configure the following settings:
Name: GSuite
Integration: Email
Email address: Your monitoring team's email address
Save the contact point.
Test the notification
Open the contact point and select the option to send a test notification.
In our implementation, PMM successfully sent the test notification, and the alert email was received.
This confirms that the email integration is working.
Important: A successful test notification does not automatically route every alert to this email address. Configure the notification policies to ensure that the appropriate alerts are delivered to the correct contact point.
2. Understand Built-in Alert Templates
PMM provides built-in alert templates for common database monitoring scenarios.
Navigate to:
Alerts → Templates
For MongoDB, useful templates include:
MongoDB down
MongoDB ReplicaSet has no Primary
MongoDB Replication Lag is high
MongoDB Primary Changed
MongoDB member is in an unusual state
The exact templates available depend on the PMM version and installed configuration.
Using built-in templates helps standardize monitoring and reduces the effort required to write custom alert expressions.
3. Create a MongoDB Node Down Alert
A node-down alert identifies when a monitored MongoDB instance becomes unavailable.
Navigate to:
Alerts → Templates → MongoDB down → Create alert rule
Configure the rule as follows:
| Setting | Configuration |
|---|---|
| Rule name | MongoDB - Node Down - Critical |
| Duration | 60 seconds |
| Severity | Critical |
| Folder | MongoDB |
| Evaluation group | MongoDB - 1m |
Keep the template expression unchanged unless there is a specific requirement to customize it.
The duration determines how long the condition must remain breached before the alert transitions to firing.
4. Configure a Replica Set No Primary Alert
A MongoDB replica set requires an elected primary to accept writes under normal replica set operation.
If the replica set has no primary, applications may experience write failures or service disruption.
Navigate to:
Alerts → Templates → MongoDB ReplicaSet has no Primary → Create alert rule
Recommended configuration:
| Setting | Configuration |
|---|---|
| Rule name | MongoDB - No Primary - Critical |
| Duration | 60 seconds |
| Severity | Critical |
| Folder | MongoDB |
| Evaluation group | MongoDB - 1m |
Keep the original template expression unchanged.
This alert helps detect a replica set that has lost its primary.
5. Configure Replication Lag Monitoring
Replication lag measures how far a secondary member is behind the primary.
High replication lag can affect read consistency, failover readiness, and recovery objectives.
Use the built-in MongoDB Replication Lag is high template.
Review the threshold and duration before enabling the rule. Select values appropriate for the application's write volume, replication requirements, and recovery objectives.
6. Configure a Custom WiredTiger Cache Alert
PMM also allows administrators to create custom alert rules using MetricsQL or PromQL expressions.
For example, a WiredTiger cache utilization alert can compare the current cache usage against the configured maximum cache size.
The following expression was validated against the metrics available in our PMM environment:
(
sum by (agent_id, instance, job) (
mongodb_mongod_wiredtiger_cache_bytes
)
/
sum by (agent_id, instance, job) (
mongodb_mongod_wiredtiger_cache_max_bytes
)
) > 0.80This expression identifies monitored series where WiredTiger cache usage exceeds 80% of the configured maximum.
The expression uses a ratio, so 0.80 represents 80%.
Note: Metric names and labels may differ between exporter versions and monitoring environments. Validate the available metrics before deploying the rule to another customer environment.
7. Recommended MongoDB Alert Coverage
A production monitoring implementation should cover the following areas.
Critical alerts
MongoDB node unavailable
Replica set has no primary
Excessive replication lag
Oplog window below the required recovery window
Backup failure or missing successful backup
WiredTiger cache utilization above the defined threshold
Connection utilization above the defined threshold
Warning alerts
Long-running queries
High operation latency
High host memory usage
Excessive swap usage
WiredTiger eviction or concurrency pressure
High disk utilization and I/O latency
Security and operational events
Failed authentication attempts
MongoDB user or role changes
TLS certificate expiry
Index build failures
Replica set elections and unexpected state changes
Some security events require MongoDB audit logs or log-based monitoring rather than standard time-series metrics. Backup monitoring also requires an appropriate backup integration or successful-backup signal.
8. Validate the Alert Delivery Workflow
Before enabling alerts in production, validate the entire notification workflow:
Confirm that the required metrics are available.
Verify the alert expression and threshold.
Configure the pending duration.
Select the correct severity and folder.
Configure the evaluation group.
Set up the notification policy and contact point.
Test the alert condition safely.
Confirm that the expected email is received.
Document the rule and its recovery procedure.
Avoid forcing failures on production database nodes simply to test an alert. Use a safe test rule or a non-production environment where possible.
Conclusion
PMM 3.9.1 provides a practical way to standardize MongoDB monitoring through built-in templates and custom alert rules.
By combining database availability alerts, replica set monitoring, replication lag detection, WiredTiger metrics, and reliable email notifications, DBAs can improve operational visibility and respond to database issues more quickly.
For customer implementations, always validate metric availability, topology, alert thresholds, and notification routing before enabling production alerts.
Note : This procedure is based on a PMM 3.9.1 configuration walkthrough. Validate all settings against the customer's environment before deployment.
0 Comments