📊 Monitoring (daemon)

📌 Description

Collecting system and application metrics on the node: CPU, RAM, disk, network, GPU, Docker and systemd state. Metrics are available through the CLI and the daemon API, and are streamed to the platform for the dashboard and alerts.

🎯 Scenarios

• asc status — a server and application summary in the terminal.
• asc stats — CPU, memory, disk usage and I/O, and network I/O per application (like docker stats --no-stream); asc stats --live keeps refreshing in place instead of exiting after one sample (like plain docker stats); root sees all users' applications grouped by owner.
• The platform dashboard shows node load graphs in real time.
• An "application went down" alert → a Telegram notification (via the platform).
• The AI assistant answers "what's eating my memory?" with monitoring data.

🏗️ Technical design

• System metrics: CPU, memory, disk (usage + I/O), network (rx/tx, errors, drops), GPU — directly from procfs (/proc/stat, /proc/meminfo, /proc/loadavg, /proc/net/dev, /proc/uptime) and statvfs(3) for disks; no external crates. Nothing distro-specific here; a separate collector behind the same abstraction will come later for macOS.
• Application metrics: per-container stats (Docker API), per-unit (systemd cgroups), per-process.
• Network interfaces: an interface list (IP, MAC, status, speed, type), traffic statistics; interface management is post-MVP.
• Storage: a ring buffer in memory + a short history in SQLite; long history lives on the platform side.
• Delivery: a push stream into the nodeservice tunnel (5–15 s interval, adaptive); healthcheck events — instantly.
• Application health statuses: running / stopped / unhealthy / unknown — based on the healthcheck from asc.yaml.

🧩 Implementation (current increment)

• The src/daemon/monitor/ module: system.rs — procfs parsers (pure functions over &str, covered by unit tests) and snapshot capture; mod.rs — Monitor: a background sampler in the daemon; the interval and history depth are set in config.toml ([monitor] interval_ms = 100, history_samples = 300 — 30 s of in-memory history at 100 ms). The old interval_secs key from configs written before DMN-072 is still parsed so those files load, but it no longer sets the interval and is never written back — it disappears from config.toml on the next save (DMN-075). Honouring it was why an updated daemon could still deliver one sample per 10 s: every install older than DMN-072 carries interval_secs = 10.
• Live stream (DMN-072): Monitor holds a tokio::sync::broadcast channel; every push() fans the sample out to subscribers (capacity 16 — a receiver that falls behind just skips ahead on Lagged rather than blocking the sampler). MonitorService.StreamSystemMetrics subscribes to the channel instead of being polled by GetSystemMetrics — the platform panel gets frames as fast as the daemon actually samples them. The subscription is taken before the buffer is read, and the sample already in the buffer is sent as the first frame (DMN-075), so a panel that just connected paints at once instead of waiting out a sampling interval.
• CPU usage is computed as the delta of two /proc/stat reads; network rates (bytes/s) — as counter deltas between samples.
• GPU metrics: gpu.rs — one entry per card (vendor, model, utilisation, VRAM used/total, temperature, power). NVIDIA comes from nvidia-smi --query-gpu=... --format=csv,noheader,nounits: the proprietary driver exposes no kernel interface with utilisation, so the vendor tool — shipped with every driver package — is the only portable source; the call is bounded by a 5 s timeout and the child is killed if it hangs. AMD comes from the amdgpu sysfs interface (gpu_busy_percent, mem_info_vram_total/_used, hwmon/*/temp1_input, power1_average), read like every other metric here. Sources are probed once: a machine with no GPU never spawns a process again, and its gpus list simply stays empty. Because a sample may spawn a process, the daemon's sampler runs the whole collection on a blocking thread.
• API: MonitorService in proto (GetSystemMetrics, GetMetricsHistory) + REST routes GET /v1/metrics and GET /v1/metrics/history?limit=N — both transports on top of the shared layer, like the rest of the API (DMN-005).
• Real listening ports (DMN-103): MonitorService.ListListeningPorts + REST GET /v1/ports/listening, CLI asc ports --listening. Not the same as declared app ports (asc ports without the flag, AppService.ListAppPorts) — this parses /proc/net/{tcp,tcp6,udp,udp6} directly for what is actually bound: TCP sockets in LISTEN, UDP sockets whose remote address is all-zero. Process attribution walks /proc/<pid>/fd/* (bounded, tolerant of permission errors — a non-root caller simply sees fewer pids). Docker container attribution is resolved by cross-referencing the container list (docker-proxy/dockerd never say which container they proxy for on their own), and a port an installed-but-stopped app declares shows up flagged as reserved rather than looking free.
• Host processes (DMN-119): ProcessService.ListProcesses / SignalProcess + REST GET /v1/processes[?kernelThreads=true] and POST /v1/processes/{pid}/signal, capability processes. src/daemon/monitor/processes.rs parses /proc/<pid>/{stat,status,cmdline,cgroup}: comm is everything up to the last ) of stat (a process may name itself a (b) c), the effective uid comes from the Uid: line, a Docker container id from the cgroup path (both /docker/<id> and docker-<id>.scope), kernel threads are told apart by PF_KTHREAD and hidden by default. CPU is top-style (100 = one core) and a delta against the previous reading, kept in a process-wide slot: a UI polling every few seconds gets the share since its own last poll without the daemon sleeping, and only a cold call (no reading, or one older than 60 s) samples for ~300 ms. The container id is resolved to a name and owning app (its container or compose project) through the Engine — non-fatal, like the port attribution. Signals are a closed set (TERM, KILL, HUP, INT, STOP, CONT, USR1, USR2 — never a raw number, the numbering is per-architecture). A process is identified by (pid, start_ticks): the caller passes back the start time it saw and a reused pid is refused (FAILED_PRECONDITION / REST 409) instead of signalling a stranger. pid 1, the daemon itself and kernel threads are refused before kill(2) (PERMISSION_DENIED / 403); signalling needs a root context, listing is open to anyone, as ps is.
• CLI: asc status shows CPU (usage, load average), memory, disks and — when there is one — every GPU (model, utilisation, VRAM) — the metrics are sampled by the CLI itself without contacting the daemon (autonomy).
• Per-app metrics: the usage() method on the AppDriver trait returns cumulative counters (CPU time in microseconds, resident memory in bytes, block-device read/write bytes, network rx/tx bytes — the I/O counters are Option, None when the runtime cannot report them); sources by runtime — the Docker Engine API (/containers/<id>/stats, one-shot: blkio_stats.io_service_bytes_recursive for disk I/O, networks for network I/O, both real because a container has its own cgroup and network namespace), the systemd unit's cgroup v2 (cpu.stat + memory.current + io.stat for disk I/O; network I/O is always None — a unit shares the host's network namespace, so there is nothing per-unit to read), /proc/<pid>/stat + statm + io for processes (disk I/O from read_bytes/write_bytes; network I/O always None for the same reason as systemd). CPU% is computed as the delta of two samples (~500 ms), like docker stats, and can exceed 100% on multi-core machines.
• App uptime (DMN-089): the uptime_secs field on AppService.GetAppStats/StreamAppStats samples — seconds since the app's current run started. The start-time source differs per kind: docker reads State.StartedAt off the inspect call it already makes, process reads starttime from /proc/<pid>/stat combined with /proc/stat's btime, systemd reads ActiveEnterTimestampMonotonic (microseconds since boot — no locale-dependent date parsing needed) anchored to the same btime. Unset while the app is stopped.
• CLI asc stats: an ID / KIND / CPU % / MEM / QUOTA / DISK / NET/s / DISK/s / NET I/O / DISK I/O table over running applications — QUOTA is a compact usage bar (like asc app disk) when the app has a disk quota set, else a dash, DISK is the app directory's total size regardless; NET/s and DISK/s are the current throughput ("rx/s / tx/s" and "read/s / write/s"), the delta of the same two samples the CPU% comes from; NET I/O and DISK I/O are "rx / tx" and "read / write" cumulative byte totals since the app started (like docker stats's own NET I/O / BLOCK I/O columns — not a rate); all four are a dash when the runtime cannot report the pair (network is only available for Docker apps, see above); sorting --sort cpu|mem (cpu by default); for root — grouping by owner, like asc app list. Stopped applications are shown with dashes. --live reprints the same table in place (screen clear + redraw) every ~500 ms — the same interval a single sample already costs for the CPU delta, so no extra sleep — until Ctrl+C.
• Next increments: per-app metrics in the API (MonitorService), history in SQLite, healthcheck statuses, pushing into the tunnel.

🔗 Related tasks

DMN-006, DMN-103, DMN-119, NODE-003, NODE-038, NODE-051, FE-004, FE-109, FE-156 in ROADMAP.md.