[EXPERIMENTAL] Mimir integration with VictoriaMetrics#1607
Closed
gsanchietti wants to merge 39 commits into
Closed
[EXPERIMENTAL] Mimir integration with VictoriaMetrics#1607gsanchietti wants to merge 39 commits into
gsanchietti wants to merge 39 commits into
Conversation
gsanchietti
changed the title
gsanchietti
changed the title
gsanchietti
force-pushed
the
alertmanager_victoria
branch
5 times, most recently
from
010e499 to
e94f63d
Compare
The netdata alert script is inside the configuration directory and it is preserved across upgrade
This was a leftover when mwan alert was moved from netdata chart to mwan hooks
This is required to keep the alert fireing inside mimir
- Add vmalert init script (vmalert.initd) to start/stop vmalert service - Add vmalert UCI configuration file (vmalert.conf) with datasource settings - Add comprehensive alert rules for host and hardware monitoring (vmalert-rules/host.yaml) - CPU usage alerts (warning at 70%, critical at 85%) - Memory usage alerts (warning at 80%, critical at 90%) - Disk space alerts (warning at 80%, critical at 90%) - Disk inodes alerts - System load alerts - Network error and drop alerts - Process zombie and blocked alerts - Update Makefile to install vmalert configuration and rules - Add detailed documentation of vmalert setup and metrics mapping - Alerts are currently in blackhole mode (evaluated but not forwarded) - Rules adapted for Telegraf metric names instead of Prometheus names - Support for Mimir integration when configured via ns-plug Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add telegraf-services Python script that queries ubus to collect the
running state of all procd-managed services. Outputs JSON for
telegraf inputs.exec with data_format = json_v2.
- Track known persistent services in /var/run/telegraf-services-known.json
so services that disappear from the ubus list (stopped or failed to
start) are still emitted with running=0, keeping VictoriaMetrics alert
rules effective even when procd removes the instance entirely.
- Add services.conf telegraf input using inputs.exec + json_v2 parser.
Tags each metric with service, instance, and has_respawn so vmalert
rules can target only persistent daemons (has_respawn=true).
- Add services.yaml vmalert rule: ServiceDown fires when
procd_service_running{has_respawn="true"} == 0 for 2 minutes.
Uses alertgroup=services label (not service=host) so the metric's
own service label (e.g. nginx, rpcd) is preserved in the alert.
- Add comprehensive telegraf/README.md documenting architecture, all
metrics, service monitoring design, PromQL query examples, and
manual test procedures.
- Add scripts/test-service-monitor.sh for end-to-end simulation:
injects a bad nginx config, verifies metric drops to 0, waits for
ServiceDown to fire, then restores nginx and confirms alert resolves.
- Update victoria-metrics/README_VMALERT.md with service monitoring
section and cross-references.
End-to-end verified on 192.168.100.238:
nginx broken config → procd removes instance from ubus →
telegraf-services emits running=0 via state file →
VictoriaMetrics stores metric →
ServiceDown{service=nginx} fires after 2m →
nginx restored → alert resolves
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add telegraf-mwan Python script that reads /var/run/mwan3/iface_state/
to collect WAN interface connectivity state. Each file in that directory
is named after an mwan3 interface and contains 'online' or 'offline'.
No-ops silently when mwan3 is not running (directory absent).
- Add mwan.conf telegraf input using inputs.exec + json_v2 parser,
producing mwan_interface metrics tagged with interface and status.
- Add mwan.yaml vmalert rule: WANDown fires when
mwan_interface_online == 0 for 2 minutes. The interface and status
labels come directly from the metric, so each WAN interface fires its
own distinct alert.
- Update telegraf/README.md with WAN Monitoring section: architecture,
metric reference, query examples, alert details, and manual test
procedure.
- Update Makefile to install telegraf-mwan and mwan.conf.
End-to-end verified on 192.168.100.238:
wan2 iface_state=offline → mwan_interface_online{interface=wan2}=0 →
WANDown{interface=wan2, alertgroup=mwan} fired after 2 minutes ✓
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Netdata health checks and alert forwarding are now disabled permanently. Alerts are handled by vmalert (via Telegraf → VictoriaMetrics) instead. Changes: - 30_ns-plug_alerts: emptied to disable all health.d rules, set health=no, remove health_alarm_notify.conf from system on upgrade. Keep python plugins for non-alerting metrics (fping latency, dashboards). - Deleted: health_alarm_notify.conf, netadata_enable_alerts, netadata_disable_alerts - ns-plug-alert: removed netdata subcommand and ~140 lines of NETDATA_ALERT_MAP, cmd_netdata, _netdata_fire, _netdata_resolve helper functions - Makefile: removed install of netadata hooks and health_alarm_notify.conf, removed /etc/netdata dir creation - README.md: updated Alerts section to reflect vmalert-based alerting Persistence: 30_ns-plug_alerts runs at every sysupgrade + fresh install, ensuring netdata alerting stays disabled across updates. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
gsanchietti
force-pushed
the
alertmanager_victoria
branch
2 times, most recently
from
2caa025 to
4158b17
Compare
Migrate ping monitoring from netdata's fping plugin to telegraf's native ping input plugin. This provides better performance, no external dependencies, and improved system compatibility. Changes: - Add ns.telegraf API handler for ping monitor configuration - Add ns.telegraf.json ACL definition for telegraf-manager role - Add telegraf.conf.d/ping.conf with native ping plugin configuration - Remove ns.netdata API handler and ACL (netdata integration) - Update ns-api Makefile to install new API handler - Update telegraf Makefile to install ping.conf and add inputs.ping tag The new API provides the same interface: - get-configuration: retrieve current ping hosts - set-hosts: configure hosts to ping The ping plugin uses native method (method="native") which sends ICMP packets directly without external ping command, requiring CAP_NET_RAW capability or root privileges. Metrics are tagged with influxdb_db="ping-metrics" for proper InfluxDB database routing. BREAKING CHANGE: ns.netdata API is removed. Clients must migrate to ns.telegraf API for ping monitor configuration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Migrate the following APIs: - ns.report: latency-and-quality-report - ns.dashboard: interface-traffic Replace data from Netdata with data from Victoria Metrics: netdata is now deprecated and will be dismissed in the future. Assisted-by:: Copilot:Sonnet4.6
gsanchietti
force-pushed
the
alertmanager_victoria
branch
from
4158b17 to
15cffa5
Compare
Member
Author
|
Replaced by #1633 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Raise Mimir alerts using VictoriaMetrics and VictoriaLogs.
Keep netdata with its own alerting for backward compatibility.
TODO: