Metrics & Monitoring

[SomeoneWeird]

Improvement Suggestion

The current iteration of our dashboards in dpanel is a stopgap to getting something more fierce in place, that would allow the user to do monitoring, and view metrics, across all running pups (and the system itself).

Without taking into consideration requirements or feedback that could change the following, I suggest two iterations to improve:

Phase 1 - Dashboard Tab

We build an overarching "Dashboards" tab into dpanel that collates exposed metrics from each pup into a single view.

We can add a dropdown at the top, and let users create customised dashboards that only include specific metrics from specific pups, mixed together.

The outcome that we should support is that a user running multiple Dogecoin Core nodes (such as existing core, core "green", sakura etc) should be able to easily view side-by-side metrics and make quick, high level comparisons.

This may require is increasing the density of metric collection, and expanding how we keep & store some of this data.

Prior to phase 2, we might want to look at running a prometheus-compatible collector that runs on the host, instead of having our custom metrics endpoint. This would mean that we make a breaking change here, instead of there, also means that if a user decides to optionally not install prometheus, metrics are support backwards-compatible supported, plus the fact that a lot of software has out of the box support for emitting prometheus-style metrics.

Phase 2 - Prom & Grafana

Without a Team behind us, we should not be wanting to re-create extensive timescale databases and frontend graphing components ourselves.

I propose that for hardware that would support it, we allow Dogebox users to install a "Monitoring Suite" pup that has both grafana and prometheus configured. If installed, metrics would be routed into this pup, rather than processed on the host, and our dashboard screen should iframe grafana (with some pre-configured dashboards for core etc).

This should be optional, as we don't want to leave users running on less-powerful hardware behind, so if this pup is not installed, we should just fall back to the UI improved in phase 1.

This will need to be specced out a little more to figure out what we can actually do via the grafana API, as they likely let you embed configured dashboard components, but don't let you embed a non-configured, configurable component.

[edtubbs]

I think a separate metrics tab makes sense. We can tweak src/pages/page-pup-library-listing/index.js for small layout changes (grid vs. single row) but .metric-container limits us.

[elusiveshiba]

I've been working on a proof of concept for this.

Added a new 'Monitoring' page which has 2 main sections - 'System' and 'Pup metrics'

The 'configure' button opens a modal with buttons for each metric so the user can enable/disable them as needed:

When a supported service is enabled, a new 'Services' section is added:

What do you think @SomeoneWeird @edtubbs ?

[patowens]

@elusiveshiba I like what you've POC'd here. It fits with the 'staple and stable features' phase of the project. As an appliance operator, I can visit the monitoring tab and answer:

A. Appliance health:

• Is my appliance chugging? Sheesh, look at that, 99% of Disk utilised.. What's consumed it? It's Foo, Bar and Baz collectively.

B. Curious questions and comparisons

• I want to compare Pup X with Y. Great, I'll enable some metrics from both so I can see them on the one page in a glance
• I need to keep tabs on Foo, Bar and Baz because I run the Lemonade stand pup to sell things online. _Great, I'll enable metrics from those 3 pups (Gigawallet, Lemonade stand)

---

Feedback/Considerations:

1. Consider Dashboards - not as a singular thing you configure, but as one of many that you can create. Why? This lets me as the user have dedicated dashboards that answer/tell specific stories. Rather than having one long dashboard that answers many questions (it could get muddied)
2. Consider renaming "Monitoring" to Monitor. Because it's not "Exploring".
3. I see your screenshot examples show pups with 7 metrics. A horizontal scrollable row or some left/right pagination could keep the list tidy and uniform, whilst catering for pups that have 10-20+ metrics.

[elusiveshiba]

I've made a bunch of updates that allow a higher level of customisation. Some UI inspiration from Home Assistant

Multiple dashboards

Edit mode - rename and reorder dashboards, reorder, add and remove components

Add or remove components from available pups/system status/services

Filter metrics shown for specific pups