Zabbix

Zabbix lives at http://zabbix.mesh.nycmesh.net

Zabbix is used primarily for historical data collection and Slack. There are a handful of dashboards configured for a few devices, but for the most part, the rest of its configuration is unused.

Data Collection

Zabbix is fed through the following sources:

Data gathered via SNMP from various OSPF devices (mainly OmniTiks) discovered through OSPF2Zabbix
Data forwarded from the UISP API by UISP2Zabbix

Custom Templates

We have a variety of custom templates, some of which were set up manually at one point, the rest either auto-generated or managed by one of the above tools.

Alerting

Alerting can be found in the #zabbix-alerts channel. ~~Currently, the channel is pretty noisy, and alerts~~Alerts need to be tuned to what we ~~_really_~~really care about, such as the antennas on the larger links. To make a trigger show up there, add the slack tag to it. The trigger can be any severity level. By default, many triggers are straight-up disabled. Alerting is, unfortunately, a manual process. We're still figuring out what is important and what isn't.

Weekly reports of noisy triggers are published in #zabbix-reports, where the top 20 noisiest triggers are aggregated. This can help us identify problems over time.

There is a plan to use certain triggers to automatically switch over links. For example, we'd like to disable the AF60xr on Vernon and use a backup link when it rains.

More Info

For (outdated-ish) information on how this was set up, including how Slack alerting was configured, refer to this doc: https://docs.google.com/document/d/1mJI8DWe882P6GCEGdT0xazxwrrCQZD7qEBcsDEjDU7Q/edit?usp=sharing