diff --git a/buildos.js b/buildos.js index 9d224270f..2f5ae0c87 100644 --- a/buildos.js +++ b/buildos.js @@ -393,6 +393,8 @@ try { var helpFiles = { "odoc": OPENAF_BUILD_HOME + "/js/odoc.js", "ow.test": OPENAF_BUILD_HOME + "/js/owrap.test.js", + "ow.desktop": OPENAF_BUILD_HOME + "/js/owrap.desktop.js", + "ow.screen": OPENAF_BUILD_HOME + "/js/owrap.screen.js", "ow.format": OPENAF_BUILD_HOME + "/js/owrap.format.js", "ow.template": OPENAF_BUILD_HOME + "/js/owrap.template.js", "ow.java": OPENAF_BUILD_HOME + "/js/owrap.java.js", diff --git a/docs/channel-graph.md b/docs/channel-graph.md new file mode 100644 index 000000000..0bb73375b --- /dev/null +++ b/docs/channel-graph.md @@ -0,0 +1,276 @@ +# Graph channels: operate and navigate a knowledge graph + +[Index](./index.md) | [Core](./openaf.md#ow-ch---channels) | [ODoc](./odoc.md) + +The `graph` channel type exposes a knowledge graph stored in a single JSON file as a regular OpenAF channel. It reads and writes the wiki graph format used by Mini-A (`graph.json`), and `ow.ch.utils.graph` adds navigation helpers (neighbors, shortest path, bounded traversal). Contracts live in [owrap.ch.js](../js/owrap.ch.js); tests are in [autoTestAll.Channels.js](../tests/autoTestAll.Channels.js) (`testGraphChannel`). + +## File format + +```json +{ + "version": 2, + "nodes": { "doc:a.md": { "id": "doc:a.md", "type": "document", "props": { "title": "A" } } }, + "edges": [ { "from": "doc:a.md", "to": "doc:b.md", "type": "LINKS_TO", "provenance": "EXTRACTED", "props": { "page": "a.md" } } ], + "communities": [], "summaries": {}, "semantic_cache": {}, "surprise": [] +} +``` + +Node ids are prefixed by kind (`doc:`, `concept:`, ...). An edge is identified by `from`, `to`, `type`, `provenance`, `props.page` and `props.section`; `provenance` is `EXTRACTED`, `INFERRED` or `AMBIGUOUS` (the default when missing or invalid). Sections the channel does not manage (`communities`, `summaries`, `semantic_cache`, `surprise`, ...) are kept untouched when saving. + +## Create a graph channel + +```javascript +$ch("g").create(true, "graph", { file: "graph.json" }); +``` + +| Option | Type | Default | Description | +| --- | --- | --- | --- | +| `file` | String | _required_ | Path to the graph JSON file | +| `readOnly` | Boolean | `false` | Reject changes and never write the file | +| `create` | Boolean | `true` | If `false` the file must already exist; otherwise an empty graph is started | +| `autosave` | String | `"always"` | `"always"` saves on every change, `"debounced"` saves at most every `saveDebounceMs`, `"off"` only saves on `save`/destroy | +| `saveDebounceMs` | Number | `5000` | Minimum interval between saves when `autosave` is `"debounced"` | +| `lock` | String | | Path to a dummy file used as a filesystem lock while loading and saving | +| `store` | Map | | Delegate nodes, edges and the remaining sections to another channel type instead of keeping them in memory (see [Keep the graph out of memory](#keep-the-graph-out-of-memory)). Can't be combined with `file` | + +Saves write a temporary file and atomically move it over `file`. Destroying the channel flushes pending changes (unless read-only). A missing file with `create: false`, or a file that is not a JSON map, makes `create` throw. + +## Keep the graph out of memory + +By default the whole graph lives in memory and `file` is rewritten on every save. With `store` the graph is kept in other OpenAF channels (nodes, edges, adjacency and the remaining sections, one channel each, named `graph::::`), so the memory used is bounded by that channel's own caching instead of by the graph size. Memory stays the default. + +```javascript +$ch("g").create(true, "graph", { store: { type: "mvs", options: { file: "graph.db", compact: true } } }); +``` + +| `store` field | Type | Description | +| --- | --- | --- | +| `type` | String | Channel type used for every part (e.g. `mvs`) | +| `options` | Map or Function | Options for the underlying channels. A Map gets a distinct `map` name per part added (so one `mvs` file holds them all); a Function receives the part name (`nodes`, `edges`, `adj`, `meta`) and returns that part's options | +| `compress` | Boolean | Passed as `shouldCompress` when creating the underlying channels | +| `bulk` | Boolean | Use the underlying channel's `setAll` for `setAll` batches (defaults to `true` for `mvs`, `false` otherwise) | + +With a store, `file`, `autosave`, `saveDebounceMs` and `lock` don't apply: every change goes straight to the store channel. `create: false` throws if the store is empty, and `readOnly` works as usual. Destroying the graph channel destroys the underlying ones. The `ow.ch.utils.graph` helpers work unchanged. + +### When does it make sense? + +Measured with `mvs` (80,000 nodes with small props and 80,000 edges, JDK 21; indicative numbers): + +| | In memory (JSON file) | `mvs` store | +| --- | --- | --- | +| Open an existing graph | ~2.2 s | ~0.05 s | +| Heap after open + 2,000 `neighbors`, 200 `get`, one `path` | ~154 MB | ~12 MB | +| Writing 160,000 entries one `set` at a time | ~2.9 s | ~75 s (~25x slower) | +| Writing 160,000 entries in one `setAll` | ~1.4 s | ~6 s (~4x slower) | +| Point reads (`get`, `neighbors`) | baseline | ~3x slower | +| Whole-graph scans (`stats`, `find`) | baseline | ~10-50x slower | + +Use a store when the graph is large relative to the heap, is mostly read, or you can't afford to load it at start-up. Stay in memory for small graphs (a few thousand nodes), write-heavy loads or frequent `stats`/`find`. Notes: + +- Load in batches: `setAll` writes a whole batch to the store at once (one `mvs` commit, and each node's adjacency written once), about 12x faster than the same entries with `set`. It follows the same rules as sequential `set` calls (an edge needs its nodes to exist earlier in the batch or already stored). Batching is on for `mvs`; set `store.bulk` to `true` or `false` to force it for other types that implement `setAll` properly. +- `mvs` commits on every change and reuses space lazily, so a long write session can bloat the file badly (7 GB for the load above). Set `compact: true` in the `mvs` options so it is compacted when the channel is destroyed (61 MB for the same graph). +- `getKeys`, `getAll`, `find` and `stats` still materialize node ids and edges in memory while they run; avoid them on very large graphs. +- Unlike the JSON file, a store is not in the Mini-A `graph.json` format. + +## Keys and values + +| Entry | Key | Value | +| --- | --- | --- | +| Node | `{ id }` | `{ id, type, props }` (`type` defaults to the existing type, or `concept` for new nodes) | +| Edge | `{ from, to, type, provenance, page, section }` | `{ from, to, type, provenance, props }` | + +A key with `from` and `to` is an edge; a key with `id` is a node; anything else throws. For edges, `type` may be given in the key or the value; `provenance` may be given in either (key wins); `page` and `section` in the key are stored as `props.page` and `props.section`. + +```javascript +$ch("g").set({ id: "doc:a.md" }, { type: "document", props: { title: "A" } }); +$ch("g").set({ id: "doc:b.md" }, { type: "document", props: { title: "B" } }); +$ch("g").set({ id: "concept:x" }, { type: "concept" }); +$ch("g").set({ from: "doc:a.md", to: "doc:b.md", type: "LINKS_TO", provenance: "extracted" }, { props: { page: "a.md" } }); +$ch("g").set({ from: "doc:b.md", to: "concept:x", type: "MENTIONS" }, {}); + +$ch("g").get({ id: "doc:b.md" }); // { id: "doc:b.md", type: "document", props: { title: "B" } } +$ch("g").size(); // 5 (3 nodes + 2 edges) +``` + +Behavior to rely on: + +- `getKeys`, `getAll`, `forEach` and `size` cover nodes first, then edges. `getSortedKeys` sorts nodes by id followed by edges. +- Setting an edge that already exists (same identity) updates its `props` instead of adding a duplicate. +- Setting an edge whose `from` or `to` node does not exist throws. +- Unsetting a node also removes every edge connected to it. +- With `readOnly: true`, `set` and `unset` throw `graph channel is read-only`. +- `setAll`, `unsetAll` and `getSet` follow the other channel types. + +## Navigate with `ow.ch.utils.graph` + +Every helper takes the channel name first. Node ids without a `:` are treated as `doc:` ids, so `"a.md"` means `"doc:a.md"`. + +| Function | Returns | Description | +| --- | --- | --- | +| `neighbors(aName, anId, aOptions)` | Array | Edges connected to a node. Options: `direction` (`"out"`, `"in"` or `"both"`, default `"both"`), `type`, `provenance` (String or Array) | +| `degree(aName, anId, aOptions)` | Number | Number of edges connected to a node (same options as `neighbors`) | +| `path(aName, aFromId, aToId, aOptions)` | Array | Shortest path (BFS, ignoring edge direction) as node ids; `[]` if either node is missing or none exists. Options: `maxDepth`, `type` | +| `traverse(aName, anId, aOptions)` | Map | `{ nodes, edges }` reachable from a node. Options: `depth` (default `1`), `direction`, `type`, `provenance` | +| `find(aName, aOptions)` | Array | Nodes matching `type` and/or `text` (case-insensitive match on id and props) | +| `stats(aName)` | Map | `nodes`, `edges`, `nodeTypes`, `edgeTypes` and `provenance` counts | +| `save(aName)` | Boolean | Flush pending changes now (`false` if read-only) | + +```javascript +var g = ow.ch.utils.graph; + +g.path("g", "a.md", "concept:x"); // [ "doc:a.md", "doc:b.md", "concept:x" ] +g.path("g", "a.md", "concept:x", { maxDepth: 1 }); // [] +g.neighbors("g", "b.md", { direction: "out" }); // [ { from: "doc:b.md", to: "concept:x", type: "MENTIONS", ... } ] +g.traverse("g", "a.md", { depth: 2 }).nodes.length; // 3 +g.find("g", { type: "concept" }); // [ { id: "concept:x", ... } ] +g.stats("g"); // { nodes: 3, edges: 2, ... } +``` + +## Example: model a folder tree + +Directories and files become nodes and containment becomes a `CONTAINS` edge, so `path`, `neighbors` and `find` answer questions about the tree. Parents are set before children because an edge needs both of its nodes to exist. + +```javascript +// Build a small tree to model +var root = io.createTempDir("graphDemo"); +io.mkdir(root + "/src/app"); io.mkdir(root + "/docs"); +io.writeFileString(root + "/README.md", "demo"); +io.writeFileString(root + "/src/app/main.js", "//"); +io.writeFileString(root + "/docs/guide.md", "#"); + +$ch("tree").create(true, "graph", { file: root + "/tree-graph.json" }); + +var base = String(io.fileInfo(root).canonicalPath); +var rel = p => String(p).substring(base.length).replace(/^\//, ""); // path relative to root +var up = p => rel(p).replace(/\/?[^\/]*$/, ""); // relative path of the parent + +$ch("tree").set({ id: "dir:" }, { type: "directory", props: { name: "." } }); +listFilesRecursive(root) + .filter(f => f.filename != "tree-graph.json") + .sort((a, b) => a.filepath.length - b.filepath.length) // parents before children + .forEach(f => { + var id = (f.isDirectory ? "dir:" : "file:") + rel(f.canonicalPath); + $ch("tree").set({ id: id }, { type: f.isDirectory ? "directory" : "file", props: { name: f.filename, size: f.size } }); + $ch("tree").set({ from: "dir:" + up(f.canonicalPath), to: id, type: "CONTAINS" }, {}); + }); + +var g = ow.ch.utils.graph; +g.path("tree", "dir:", "file:src/app/main.js"); // [ "dir:", "dir:src", "dir:src/app", "file:src/app/main.js" ] +g.neighbors("tree", "dir:src", { direction: "out" }); // one edge: dir:src -> dir:src/app +g.find("tree", { type: "file", text: ".md" }); // README.md and docs/guide.md nodes +g.stats("tree"); // { nodes: 7, edges: 6, nodeTypes: { directory: 4, file: 3 }, edgeTypes: { CONTAINS: 6 }, ... } + +$ch("tree").destroy(); +io.rm(root); +``` + +Edges created without a `provenance` are counted as `AMBIGUOUS`; add `provenance: "EXTRACTED"` to the edge key when the relation is known for certain, as it is for containment here. + +## More use cases + +The graph does not have to come from Mini-A: any set of things and relations fits. The snippets below assume the helpers `N` and `E` and the shortcut `G`: + +```javascript +$ch("g").create(true, "graph", { file: "infra.json" }); +var G = ow.ch.utils.graph; +var N = (id, type, props) => $ch("g").set({ id: id }, { type: type, props: props || {} }); +var E = (from, to, type, prov) => $ch("g").set({ from: from, to: to, type: type, provenance: prov || "EXTRACTED" }, {}); +``` + +### Service dependencies and impact analysis + +Model services and datastores with `DEPENDS_ON` edges, then ask what breaks when something goes down. Following edges backwards (`direction: "in"`) from a node lists everything that depends on it, directly or not. + +```javascript +N("svc:web", "service"); N("svc:api", "service"); N("svc:auth", "service"); +N("svc:queue", "service"); N("svc:db", "database"); +E("svc:web", "svc:api", "DEPENDS_ON"); E("svc:api", "svc:auth", "DEPENDS_ON"); +E("svc:api", "svc:db", "DEPENDS_ON"); E("svc:auth", "svc:db", "DEPENDS_ON"); +E("svc:queue", "svc:db", "DEPENDS_ON"); + +// Blast radius if the database fails (includes the database itself) +G.traverse("g", "svc:db", { depth: 5, direction: "in", type: "DEPENDS_ON" }).nodes.map(n => n.id); +// [ "svc:db", "svc:api", "svc:auth", "svc:queue", "svc:web" ] + +// Most critical component: how many services depend directly on it +G.degree("g", "svc:db", { direction: "in" }); // 3 + +// Entry points: services nothing else depends on +G.find("g", { type: "service" }).filter(n => G.degree("g", n.id, { direction: "in" }) == 0).map(n => n.id); +// [ "svc:web", "svc:queue" ] + +// How does the web tier reach the database? +G.path("g", "svc:web", "svc:db"); // [ "svc:web", "svc:api", "svc:db" ] +``` + +Rebuild the graph from your CMDB, Kubernetes manifests or Terraform state on a schedule, and keep the file in version control to diff the architecture between releases. + +### Review uncertain relations + +`provenance` records how sure you are about a relation. Edges created without it are `AMBIGUOUS`, and heuristics (naming conventions, log correlation, an LLM) can mark theirs `INFERRED`. Use that to build a review queue and promote the edges a person confirms. Provenance is part of the edge identity, so promoting means unsetting the old edge and setting a new one. + +```javascript +N("person:ana", "person"); +$ch("g").set({ from: "person:ana", to: "svc:api", type: "OWNS" }, {}); // AMBIGUOUS + +// Review queue: everything that is not certain +G.neighbors("g", "person:ana", { provenance: ["INFERRED", "AMBIGUOUS"] }); + +// Promote after confirmation +$ch("g").getKeys().filter(k => k.from).forEach(k => { + var e = $ch("g").get(k); + if (e.provenance == "AMBIGUOUS") { + $ch("g").unset(k); + $ch("g").set({ from: e.from, to: e.to, type: e.type, provenance: "EXTRACTED" }, { props: e.props }); + } +}); +G.stats("g").provenance; // only EXTRACTED edges left +``` + +### Ownership and "who do I ask?" + +Add people, teams and topics as nodes with `OWNS`, `MEMBER_OF` or `KNOWS` edges. `path` finds the shortest chain between any two nodes, which turns into an escalation route or an explanation of why two things are related. `find` with `text` searches ids and properties, so it works as a quick lookup. + +```javascript +G.find("g", { type: "person", text: "ana" }); // who is Ana? +G.neighbors("g", "svc:api", { direction: "in", type: "OWNS" }).map(e => e.from); // owners of the API +G.path("g", "person:ana", "svc:db"); // how is Ana connected to the database? +``` + +### Export for visualization + +Edge keys carry everything needed to draw the graph, so exporting to Mermaid (or Graphviz, or `oafp`) is a few lines. + +```javascript +var lines = [ "graph LR" ]; +$ch("g").getKeys().filter(k => k.from).forEach(k => + lines.push(" " + k.from.replace(/\W/g, "_") + " -->|" + k.type + "| " + k.to.replace(/\W/g, "_"))); +io.writeFileString("infra.mmd", lines.join("\n")); +``` + +### Other ideas + +- **Document and code maps**: `IMPORTS` or `LINKS_TO` edges between files, to find orphans (`degree` of 0) or circular references. +- **Data lineage**: `PRODUCES`/`CONSUMES` edges between jobs and datasets; `traverse` with `direction: "out"` shows everything downstream of a changed table. +- **Incident and runbook index**: link alerts, services and runbooks, then `neighbors` of an alert gives the runbook to follow. +- **Shared, read-only reference data**: ship a curated `graph.json` with an application and open it with `readOnly: true`. + +## Working with Mini-A + +The format is the one Mini-A saves in `/graph.json` (default `./.mini-a-wiki-graph`), so you can inspect or edit a graph produced by Mini-A without loading its code: + +```javascript +$ch("wiki").create(true, "graph", { file: "./.mini-a-wiki-graph/graph.json", readOnly: true }); +ow.ch.utils.graph.stats("wiki"); +``` + +Limits to keep in mind: + +- The whole file is rewritten on save. Do not let Mini-A and a writable graph channel update the same file at the same time; use `readOnly: true` for shared access, or the `lock` option when both sides cooperate on a lock file. +- Semantic extraction, community detection and summaries stay in Mini-A. Changing nodes or edges through the channel can leave `communities` and `summaries` stale until Mini-A rebuilds them. +- Mini-A edge identity also includes `props.page` and `props.section`; use the same `page`/`section` values when updating or removing edges it created. +- Everything is held in memory, so very large graphs cost memory proportional to their size. + +## Source documentation + +The same contracts are documented in the source as Markdown ODoc entries (`ow.ch.types.graph` and `ow.ch.utils.graph`), available with `help ow.ch.types.graph` in the console or `openaf -helpscript ow.ch.types.graph`. See [odoc.md](./odoc.md#opt-in-to-markdown) for the format. diff --git a/docs/channel-timeseries.md b/docs/channel-timeseries.md new file mode 100644 index 000000000..25a171aa9 --- /dev/null +++ b/docs/channel-timeseries.md @@ -0,0 +1,242 @@ +# Time series channels: record, retain and query labelled metrics + +[Index](./index.md) | [Core](./openaf.md#ow-ch---channels) | [ODoc](./odoc.md) + +The `timeseries` channel type stores numeric points identified by a metric name, a set of labels and a timestamp (the Prometheus/OpenMetrics model) as a regular OpenAF channel. It keeps data in memory with an optional append-only NDJSON file, or delegates it to other channels (e.g. `mvs`), applies retention by age and/or count, maintains RRD-like rollups, and `ow.ch.utils.timeseries` adds queries (`range`, `latest`, ...), a collector for `ow.metrics`, a channel subscriber and OpenMetrics import/export. Contracts live in [owrap.ch.js](../js/owrap.ch.js); tests are in [autoTestAll.Channels.js](../tests/autoTestAll.Channels.js) (`testTimeSeriesChannel`, `testTimeSeriesChannelStore`, `testTimeSeriesIntegrations`). + +## Create a timeseries channel + +```javascript +$ch("m").create(true, "timeseries", { file: "metrics.ndjson", retention: "7d" }); +``` + +| Option | Type | Default | Description | +| --- | --- | --- | --- | +| `file` | String | | Path to the NDJSON log. Without `file` (and `store`) the channel only lives in memory | +| `readOnly` | Boolean | `false` | Reject changes and never write the file | +| `create` | Boolean | `true` | If `false` the file (or store) must already exist | +| `autosave` | String | `"always"` | `"always"` appends on every change, `"debounced"` appends at most every `saveDebounceMs`, `"off"` only appends on `save`/destroy | +| `saveDebounceMs` | Number | `5000` | Minimum interval between appends when `autosave` is `"debounced"` | +| `lock` | String | | Path to a dummy file used as a filesystem lock while loading and writing | +| `compactRatio` | Number | `0.5` | The file is rewritten when more than this ratio of its lines is dead (overwritten, deleted or expired) | +| `store` | Map | | Keep the data on channels of another type instead of memory (see [Keep the data out of memory](#keep-the-data-out-of-memory)). Can't be combined with `file` | +| `chunk` | String/Number | `"1h"` | Time span of each stored chunk (only with `store`) | +| `retention` | String/Number | | How long to keep raw points (`"7d"`, `"90m"`, or ms). Same as `raw.keep` | +| `maxPoints` | Number | | Maximum raw points per series. Same as `raw.maxPoints` | +| `raw` | Map | | `{ keep, maxPoints }` for the raw points | +| `rollups` | Array | | `[ { step: "1m", keep: "1d" }, { step: "1h", keep: "30d" } ]` (see [Rollups](#rollups)) | +| `retentionRef` | String | `"series"` | `"series"` measures `keep` back from the newest point of each series, `"now"` from the current time | +| `housekeepEveryMs` | Number | `0` | Minimum interval between retention runs triggered by writes (`0` runs on every write) | + +Durations accept numbers (ms) or abbreviations understood by `ow.format.fromTimeAbbreviation` (`500ms`, `30s`, `5m`, `1h`, `7d`, `1h30m`). The channel never starts threads of its own. + +## Keys and values + +| Entry | Key | Value | +| --- | --- | --- | +| Point | `{ metric, labels, ts }` | `{ v }` where `v` is a finite number | + +`labels` is optional and its values are stored as strings; `ts` is epoch ms, a Date or an ISO string (and defaults to now on `set`). A series is a metric plus its labels; a point is a series plus a `ts`. What `get` and `getAll` return is always `{ metric, labels, ts, v }`. + +```javascript +$ch("m").set({ metric: "cpu", labels: { host: "a" }, ts: 1000 }, { v: 0.4 }); +$ch("m").set({ metric: "cpu", labels: { host: "a" }, ts: 2000 }, { v: 0.6 }); +$ch("m").set({ metric: "cpu", labels: { host: "b" }, ts: "2026-01-01T00:00:00Z" }, { v: 0.1 }); +$ch("m").set({ metric: "uptime" }, { v: 3600 }); // ts defaults to now + +$ch("m").get({ metric: "cpu", labels: { host: "a" } }); // latest: { metric: "cpu", labels: { host: "a" }, ts: 2000, v: 0.6 } +$ch("m").get({ metric: "cpu", labels: { host: "a" }, ts: 1000 }); // exact point +$ch("m").size(); // 4 (points) +``` + +Behavior to rely on: + +- Setting the same series and `ts` again overwrites the point; out-of-order points are inserted in order. +- `get` without `ts` returns the latest point of the series. +- `unset` with `ts` removes that point; **without `ts` it drops the whole series** (points and rollups). `ow.ch.utils.timeseries.drop` removes many series by selector. +- `getKeys`, `getAll` and `forEach` go series by series (sorted by metric and labels), points in time order. `getKeys`/`getAll` accept a selector `{ metric, labels, from, to }`. `getSortedKeys` sorts all points by `ts`. +- `shift`/`pop` follow the other channel types: they return **and remove** the oldest/newest point of the channel. +- `setAll(aKeyFields, aValues)` takes flat values `{ metric, labels, ts, v }`; the key fields default to `metric`, `labels` and `ts`. It does one write per series (and one file append, or one `mvs` commit per chunk). +- With `readOnly: true`, changes throw `timeseries channel is read-only`. +- Channel subscribers run asynchronously (use `$ch(...).waitForJobs()` when you need to wait for them). + +## The NDJSON file + +The file is a log: each write appends one line (`{"o":"s",...}` series, `p` point, `r` rollup bucket, `d` delete, `x` drop series) so a write never rewrites the file. Loading replays the log (a truncated last line is ignored). Overwritten, deleted and expired entries become dead lines; when more than `compactRatio` of the lines are dead (checked every 1000 appended lines and on destroy) the file is rewritten with the live data using a temporary file and an atomic move. `ow.ch.utils.timeseries.compact` does it on request. + +Retention is not logged: it is applied again when the file is loaded and after writes. Use only one writer per file (or `lock`, for cooperating processes that reopen the channel between writes); use `readOnly: true` for shared reading. + +## Query with `ow.ch.utils.timeseries` + +Every function takes the channel name first. A **selector** is `{ metric, labels }` where `metric` and each label value can be a String, an Array (any of), a RegExp or a Function; an empty selector matches everything. + +| Function | Returns | Description | +| --- | --- | --- | +| `series(aName, aSelector)` | Array | `{ metric, labels }` of the matching series | +| `labels(aName, aMetric)` | Map | Label names with their sorted values (for a metric, or for all) | +| `latest(aName, aSelector)` | Array | The newest point of each matching series as flat rows | +| `range(aName, aSelector, aOptions)` | Array | Flat rows `{ ts, metric, ...labels, v }` (see below) | +| `drop(aName, aSelector)` | Number | Removes the matching series; returns how many | +| `stats(aName)` | Map | `series`, `points`, per `metrics` (series, points, first, last) and per `tiers` entries | +| `housekeep(aName, aOptions)` | Number | Applies retention now (`{ now }` measures from that time); returns the removed entries | +| `rebuildRollups(aName)` | | Recomputes every rollup from the raw points | +| `compact(aName)` | Boolean | Rewrites the NDJSON file with the live data only | +| `save(aName)` | Boolean | Flushes pending appends now (`false` if read-only) | + +`range` options: + +| Option | Description | +| --- | --- | +| `from`, `to` | Inclusive bounds (epoch ms, Date or ISO string). Default: everything | +| `step` | Bucket size (ms or `"5m"`). Buckets are aligned to the epoch and a row's `ts` is the start of its bucket. Without `step` the raw points are returned | +| `fn` | `avg` (default), `min`, `max`, `sum`, `count`, `first`, `last`, `rate` (per second) or `increase`. `rate` and `increase` treat a decrease of the value as a counter reset | +| `fill` | For buckets without data: `none` (default), `null`, `zero` or `prev` | +| `tier` | Force the raw tier (`"raw"`) or a rollup (its step, e.g. `"1h"`) | + +```javascript +var ts = ow.ch.utils.timeseries; + +ts.range("m", { metric: "cpu", labels: { host: /^web/ } }, { from: "2026-01-01", step: "5m", fn: "max" }); +// [ { ts: 1767225600000, metric: "cpu", host: "web1", v: 0.93 }, ... ] + +ts.range("m", { metric: "http_requests_total" }, { step: "1m", fn: "rate" }); // requests per second +ts.latest("m", { metric: "cpu" }); // one row per host +ts.labels("m", "cpu"); // { host: [ "a", "b" ] } +``` + +The rows are flat, so they go straight into `oafp`: + +```bash +openaf -c 'ow.loadCh(); $ch("m").create(true,"timeseries",{file:"cpu.ndjson"}); print(stringify(ow.ch.utils.timeseries.range("m",{metric:"cpu"},{step:"2m"}),__,""))' | oafp out=ctable +``` + +Label names that collide with `ts`, `metric` or `v` are prefixed with `_` in the rows. + +## Retention + +- `retention` / `raw.keep` drops raw points older than `keep` before the newest point of the same series (`retentionRef: "series"`, the default; it is deterministic and never empties a series that stopped reporting) or before the current time (`retentionRef: "now"`). +- `maxPoints` keeps the newest N raw points of each series. +- Both run on writes (every write by default, or every `housekeepEveryMs`), when the channel is created, and on demand with `ow.ch.utils.timeseries.housekeep(aName)`. Use `housekeep` (or a scheduled job) to trim series that stopped receiving writes with `retentionRef: "now"`. +- Points removed by retention do not notify channel subscribers. +- A point written older than the retention is accepted and removed right away. + +## Rollups + +Rollups keep aggregated buckets after the raw points are gone (the RRD idea), so long ranges stay cheap and history outlives the raw retention. + +```javascript +$ch("m").create(true, "timeseries", { + file: "metrics.ndjson", + raw : { keep: "6h" }, // raw points for 6 hours + rollups: [ { step: "1m", keep: "1d" }, // 1 minute buckets for a day + { step: "1h", keep: "30d" } ] // 1 hour buckets for 30 days +}); +``` + +Each bucket stores `count`, `sum`, `min`, `max`, `first` and `last` (so `avg`, `min`, `max`, `sum`, `count`, `first`, `last` and `increase` are all answered from it, and a coarser `step` can merge finer buckets exactly when it is a multiple of the bucket size). Buckets are updated on every write. + +- `range` with a `step` uses the finest tier whose buckets divide `step` and that still reaches back to `from` (raw points while they exist). Use `tier` to force one. +- A late point is always stored in the raw tier, and updates every rollup whose bucket has not expired. +- Overwriting or removing a point rebuilds its buckets from the raw points when those are complete. When raw data has been trimmed, the bucket is only adjusted (`count`, `sum`, `first`, `last` follow, but `min` and `max` can't narrow). +- On rollup tiers `rate`/`increase` are approximate: a reset inside a bucket is only detected when its `last` is below its `first`. +- Rollups only exist for the writes made after they were configured; `rebuildRollups` recomputes them from the raw points that are still there. + +## Keep the data out of memory + +By default everything is in memory (and optionally in the NDJSON file). With `store` the data is kept in other OpenAF channels (a `series` index and one chunk channel per tier, named `timeseries::::`) so memory is bounded by that channel's own caching. Points are stored as sorted chunks (key `{ s, b }` with the chunk's start time), so a `range` only reads the chunks it overlaps, and retention drops whole chunks. + +```javascript +$ch("m").create(true, "timeseries", { store: { type: "mvs", options: { file: "metrics.db", compact: true } }, chunk: "1h", retention: "30d" }); +``` + +| `store` field | Type | Description | +| --- | --- | --- | +| `type` | String | Channel type used for every part (e.g. `mvs`) | +| `options` | Map or Function | Options for the underlying channels. A Map gets a distinct `map` name per part (so one `mvs` file holds all); a Function receives the part name (`series`, `raw`, `r`) and returns its options | +| `compress` | Boolean | Passed as `shouldCompress` when creating the underlying channels | +| `bulk` | Boolean | Use the underlying channel's `setAll` to write several chunks at once (defaults to `true` for `mvs`) | + +With a store, `file`, `autosave`, `saveDebounceMs`, `lock` and `compactRatio` don't apply: every change goes straight to the store. `create: false` throws if the store is empty. Destroying the timeseries channel destroys the underlying ones. + +Measured with `mvs` (100,000 points over 20 series, 1 s spacing; indicative numbers on a laptop): + +| | Memory | NDJSON file | `mvs` store | +| --- | --- | --- | --- | +| Load 100,000 points in one `setAll` | ~1.1 s | ~1.0 s | ~1.1 s (1.7 s with two rollups) | +| 2,000 points one `set` at a time | ~0.1 s | ~0.2 s | ~2.5 s (4.7 s with two rollups) | +| Reopen with 100,000 points | | ~0.3 s | no replay | +| `range` of one series, all points | ~10 ms | ~10 ms | ~15 ms | +| `latest` over 20 series | ~2 ms | ~2 ms | ~30 ms | + +Use a store when the data is large relative to the heap or you can't afford to replay a log at start-up; write in batches (`setAll`) because every `mvs` write is a commit. Notes: `getKeys`, `getAll`, `stats` and `size` on a store walk the series index or all chunks of the selected series; set `compact: true` in the `mvs` options so the file is compacted on destroy. + +## Record data + +### From `ow.metrics` + +`collect(aEvery, aName, aSource, aOptions)` records a source periodically, in the style of `ow.ch.utils.keepHistory`: `aEvery` is a number of ms or a cron expression; `aSource` is omitted (all of `ow.metrics.getAll()`), an Array of metric names, or a function returning a map or an array of `{ metric, labels, v, ts }`. Nested maps extend the metric name with `_` (leading underscores of a key are dropped), array items get an `idx` label, booleans become 0/1 and non numeric values are skipped. Options: `labels` (added to every point), `prefix`, `onError`. It returns `{ once, stop }`; **call `stop()` before the process ends or the channel is destroyed** (the collector also stops itself if the channel no longer exists). + +```javascript +$ch("m").create(true, "timeseries", { file: "oaf.ndjson", retention: "2d", retentionRef: "now" }); +var col = ow.ch.utils.timeseries.collect(5000, "m", __, { prefix: "oaf_", labels: { app: "demo" } }); +// ... +col.stop(); +``` + +### From another channel + +`getTimeSeriesSubscriber(aTargetCh, aOptions)` returns a subscriber that records the `set`/`setall` of another channel as points. Options: `metric` (String or `function(key, value)`), `labels` (Array of field names, or a function), `value` (field name or function; defaults to `v`) **or** `values` (Array of field names, one metric `_` each), `ts` (field name or function; defaults to now) and `ops`. + +```javascript +$ch("events").subscribe(ow.ch.utils.timeseries.getTimeSeriesSubscriber("m", { metric: "temp", labels: [ "room" ], value: "t", ts: "when" })); +$ch("events").set({ id: 1 }, { id: 1, room: "kitchen", t: 21.5, when: 1000 }); +``` + +### OpenMetrics (Prometheus text) + +This exporter emits legacy Prometheus text. The strict OpenMetrics mode of +`ow.server.telemetry.passive` and `ow.metrics.fromObj2OpenMetrics` does not apply here. + +`toOpenMetrics(aName, aSelector, aOptions)` returns the latest point of each series (or every point between `from` and `to`) as text: `name{label="value"} value timestamp`. Metric and label names are sanitized, label values escaped. Options: `help` (`{ metric: { help, type } }` for `# HELP`/`# TYPE` lines), `timestamps: false`, `tsUnit` (`"s"` default, `"ms"`). `fromOpenMetrics(aName, aText, aOptions)` records such text as points (options: `labels` added to all, `ts` default timestamp, `tsUnit`) and returns how many. + +```javascript +var txt = ow.ch.utils.timeseries.toOpenMetrics("m", { metric: /^cpu/ }); +// cpu{host="a"} 0.6 2 +$ch("copy").create(true, "timeseries", {}); +ow.ch.utils.timeseries.fromOpenMetrics("copy", txt); +``` + +### Push to a Prometheus push gateway + +`pushPrometheus(aName, aPushGwUrl, aJob, aSelector, aOptions)` sends the latest point of each matching series to a [Pushgateway](https://github.com/prometheus/pushgateway) under the job `aJob`. Series are grouped by the values of `groupBy` labels: each distinct combination is one request to `/metrics/job//