From 06d4a97e132d473f998c1d346b51565c878bb22d Mon Sep 17 00:00:00 2001 From: nmaguiar Date: Fri, 16 Oct 2026 05:13:01 +0100 Subject: [PATCH 01/19] feat(channels): add channel graph support Add a channel graph to the Channels wrapper (js/owrap.ch.js) and wire it into the console (js/openafconsole.js). Document the new graph in docs/channel-graph.md and reference it from docs/index.md and docs/openaf.md. Extend the Channels test suite (tests/autoTestAll.Channels.js and tests/autoTestAll.Channels.yaml) to cover the new functionality. --- docs/channel-graph.md | 149 ++++++++++++ docs/index.md | 1 + docs/openaf.md | 14 ++ js/openafconsole.js | 4 +- js/owrap.ch.js | 413 ++++++++++++++++++++++++++++++++ tests/autoTestAll.Channels.js | 67 ++++++ tests/autoTestAll.Channels.yaml | 8 + 7 files changed, 655 insertions(+), 1 deletion(-) create mode 100644 docs/channel-graph.md diff --git a/docs/channel-graph.md b/docs/channel-graph.md new file mode 100644 index 00000000..ec1a7380 --- /dev/null +++ b/docs/channel-graph.md @@ -0,0 +1,149 @@ +# Graph channels: operate and navigate a knowledge graph + +[Index](./index.md) | [Core](./openaf.md#ow-ch---channels) | [ODoc](./odoc.md) + +The `graph` channel type exposes a knowledge graph stored in a single JSON file as a regular OpenAF channel. It reads and writes the wiki graph format used by Mini-A (`graph.json`), and `ow.ch.utils.graph` adds navigation helpers (neighbors, shortest path, bounded traversal). Contracts live in [owrap.ch.js](../js/owrap.ch.js); tests are in [autoTestAll.Channels.js](../tests/autoTestAll.Channels.js) (`testGraphChannel`). + +## File format + +```json +{ + "version": 2, + "nodes": { "doc:a.md": { "id": "doc:a.md", "type": "document", "props": { "title": "A" } } }, + "edges": [ { "from": "doc:a.md", "to": "doc:b.md", "type": "LINKS_TO", "provenance": "EXTRACTED", "props": { "page": "a.md" } } ], + "communities": [], "summaries": {}, "semantic_cache": {}, "surprise": [] +} +``` + +Node ids are prefixed by kind (`doc:`, `concept:`, ...). An edge is identified by `from`, `to`, `type`, `provenance`, `props.page` and `props.section`; `provenance` is `EXTRACTED`, `INFERRED` or `AMBIGUOUS` (the default when missing or invalid). Sections the channel does not manage (`communities`, `summaries`, `semantic_cache`, `surprise`, ...) are kept untouched when saving. + +## Create a graph channel + +```javascript +$ch("g").create(true, "graph", { file: "graph.json" }); +``` + +| Option | Type | Default | Description | +| --- | --- | --- | --- | +| `file` | String | _required_ | Path to the graph JSON file | +| `readOnly` | Boolean | `false` | Reject changes and never write the file | +| `create` | Boolean | `true` | If `false` the file must already exist; otherwise an empty graph is started | +| `autosave` | String | `"always"` | `"always"` saves on every change, `"debounced"` saves at most every `saveDebounceMs`, `"off"` only saves on `save`/destroy | +| `saveDebounceMs` | Number | `5000` | Minimum interval between saves when `autosave` is `"debounced"` | +| `lock` | String | | Path to a dummy file used as a filesystem lock while loading and saving | + +Saves write a temporary file and atomically move it over `file`. Destroying the channel flushes pending changes (unless read-only). A missing file with `create: false`, or a file that is not a JSON map, makes `create` throw. + +## Keys and values + +| Entry | Key | Value | +| --- | --- | --- | +| Node | `{ id }` | `{ id, type, props }` (`type` defaults to the existing type, or `concept` for new nodes) | +| Edge | `{ from, to, type, provenance, page, section }` | `{ from, to, type, provenance, props }` | + +A key with `from` and `to` is an edge; a key with `id` is a node; anything else throws. For edges, `type` may be given in the key or the value; `provenance` may be given in either (key wins); `page` and `section` in the key are stored as `props.page` and `props.section`. + +```javascript +$ch("g").set({ id: "doc:a.md" }, { type: "document", props: { title: "A" } }); +$ch("g").set({ id: "doc:b.md" }, { type: "document", props: { title: "B" } }); +$ch("g").set({ id: "concept:x" }, { type: "concept" }); +$ch("g").set({ from: "doc:a.md", to: "doc:b.md", type: "LINKS_TO", provenance: "extracted" }, { props: { page: "a.md" } }); +$ch("g").set({ from: "doc:b.md", to: "concept:x", type: "MENTIONS" }, {}); + +$ch("g").get({ id: "doc:b.md" }); // { id: "doc:b.md", type: "document", props: { title: "B" } } +$ch("g").size(); // 5 (3 nodes + 2 edges) +``` + +Behavior to rely on: + +- `getKeys`, `getAll`, `forEach` and `size` cover nodes first, then edges. `getSortedKeys` sorts nodes by id followed by edges. +- Setting an edge that already exists (same identity) updates its `props` instead of adding a duplicate. +- Setting an edge whose `from` or `to` node does not exist throws. +- Unsetting a node also removes every edge connected to it. +- With `readOnly: true`, `set` and `unset` throw `graph channel is read-only`. +- `setAll`, `unsetAll` and `getSet` follow the other channel types. + +## Navigate with `ow.ch.utils.graph` + +Every helper takes the channel name first. Node ids without a `:` are treated as `doc:` ids, so `"a.md"` means `"doc:a.md"`. + +| Function | Returns | Description | +| --- | --- | --- | +| `neighbors(aName, anId, aOptions)` | Array | Edges connected to a node. Options: `direction` (`"out"`, `"in"` or `"both"`, default `"both"`), `type`, `provenance` (String or Array) | +| `degree(aName, anId, aOptions)` | Number | Number of edges connected to a node (same options as `neighbors`) | +| `path(aName, aFromId, aToId, aOptions)` | Array | Shortest path (BFS, ignoring edge direction) as node ids; `[]` if either node is missing or none exists. Options: `maxDepth`, `type` | +| `traverse(aName, anId, aOptions)` | Map | `{ nodes, edges }` reachable from a node. Options: `depth` (default `1`), `direction`, `type`, `provenance` | +| `find(aName, aOptions)` | Array | Nodes matching `type` and/or `text` (case-insensitive match on id and props) | +| `stats(aName)` | Map | `nodes`, `edges`, `nodeTypes`, `edgeTypes` and `provenance` counts | +| `save(aName)` | Boolean | Flush pending changes now (`false` if read-only) | + +```javascript +var g = ow.ch.utils.graph; + +g.path("g", "a.md", "concept:x"); // [ "doc:a.md", "doc:b.md", "concept:x" ] +g.path("g", "a.md", "concept:x", { maxDepth: 1 }); // [] +g.neighbors("g", "b.md", { direction: "out" }); // [ { from: "doc:b.md", to: "concept:x", type: "MENTIONS", ... } ] +g.traverse("g", "a.md", { depth: 2 }).nodes.length; // 3 +g.find("g", { type: "concept" }); // [ { id: "concept:x", ... } ] +g.stats("g"); // { nodes: 3, edges: 2, ... } +``` + +## Example: model a folder tree + +Directories and files become nodes and containment becomes a `CONTAINS` edge, so `path`, `neighbors` and `find` answer questions about the tree. Parents are set before children because an edge needs both of its nodes to exist. + +```javascript +// Build a small tree to model +var root = io.createTempDir("graphDemo"); +io.mkdir(root + "/src/app"); io.mkdir(root + "/docs"); +io.writeFileString(root + "/README.md", "demo"); +io.writeFileString(root + "/src/app/main.js", "//"); +io.writeFileString(root + "/docs/guide.md", "#"); + +$ch("tree").create(true, "graph", { file: root + "/tree-graph.json" }); + +var base = String(io.fileInfo(root).canonicalPath); +var rel = p => String(p).substring(base.length).replace(/^\//, ""); // path relative to root +var up = p => rel(p).replace(/\/?[^\/]*$/, ""); // relative path of the parent + +$ch("tree").set({ id: "dir:" }, { type: "directory", props: { name: "." } }); +listFilesRecursive(root) + .filter(f => f.filename != "tree-graph.json") + .sort((a, b) => a.filepath.length - b.filepath.length) // parents before children + .forEach(f => { + var id = (f.isDirectory ? "dir:" : "file:") + rel(f.canonicalPath); + $ch("tree").set({ id: id }, { type: f.isDirectory ? "directory" : "file", props: { name: f.filename, size: f.size } }); + $ch("tree").set({ from: "dir:" + up(f.canonicalPath), to: id, type: "CONTAINS" }, {}); + }); + +var g = ow.ch.utils.graph; +g.path("tree", "dir:", "file:src/app/main.js"); // [ "dir:", "dir:src", "dir:src/app", "file:src/app/main.js" ] +g.neighbors("tree", "dir:src", { direction: "out" }); // one edge: dir:src -> dir:src/app +g.find("tree", { type: "file", text: ".md" }); // README.md and docs/guide.md nodes +g.stats("tree"); // { nodes: 7, edges: 6, nodeTypes: { directory: 4, file: 3 }, edgeTypes: { CONTAINS: 6 }, ... } + +$ch("tree").destroy(); +io.rm(root); +``` + +Edges created without a `provenance` are counted as `AMBIGUOUS`; add `provenance: "EXTRACTED"` to the edge key when the relation is known for certain, as it is for containment here. + +## Working with Mini-A + +The format is the one Mini-A saves in `/graph.json` (default `./.mini-a-wiki-graph`), so you can inspect or edit a graph produced by Mini-A without loading its code: + +```javascript +$ch("wiki").create(true, "graph", { file: "./.mini-a-wiki-graph/graph.json", readOnly: true }); +ow.ch.utils.graph.stats("wiki"); +``` + +Limits to keep in mind: + +- The whole file is rewritten on save. Do not let Mini-A and a writable graph channel update the same file at the same time; use `readOnly: true` for shared access, or the `lock` option when both sides cooperate on a lock file. +- Semantic extraction, community detection and summaries stay in Mini-A. Changing nodes or edges through the channel can leave `communities` and `summaries` stale until Mini-A rebuilds them. +- Mini-A edge identity also includes `props.page` and `props.section`; use the same `page`/`section` values when updating or removing edges it created. +- Everything is held in memory, so very large graphs cost memory proportional to their size. + +## Source documentation + +The same contracts are documented in the source as Markdown ODoc entries (`ow.ch.types.graph` and `ow.ch.utils.graph`), available with `help ow.ch.types.graph` in the console or `openaf -helpscript ow.ch.types.graph`. See [odoc.md](./odoc.md#opt-in-to-markdown) for the format. diff --git a/docs/index.md b/docs/index.md index 6847ee4c..36462681 100644 --- a/docs/index.md +++ b/docs/index.md @@ -16,6 +16,7 @@ Core references and extended guides to build and operate oJobs and OpenAF soluti ## Core - [openaf.md](./openaf.md) – Core OpenAF runtime helpers & APIs (`$$`, `_$`, `$from`, `$path`, channels, etc.) - [openaf-dollar-functions.md](./openaf-dollar-functions.md) – Additional `$` helpers: formatting, caching, synchronization, argument conversion, JSON-RPC, processes, and output +- [channel-graph.md](./channel-graph.md) – `graph` channel type: operate and navigate a Mini-A knowledge graph (`graph.json`) with `ow.ch.utils.graph` - [ojob.md](./ojob.md) – Main oJob YAML reference (structure, jobs, built-ins, shortcuts, argument tables) - [ojob-all.yaml](./ojob-all.yaml) – Annotated syntax catalog (illustrative options, not a runnable definition) - [ojob-examples.md](./ojob-examples.md) – Patterns and source examples from the ojob.io YAML collection diff --git a/docs/openaf.md b/docs/openaf.md index 25a4c206..855fd5a4 100644 --- a/docs/openaf.md +++ b/docs/openaf.md @@ -333,6 +333,20 @@ ow.ch.new("sharedData", "big"); ow.ch.set("sharedData", "someKey", "someValue"); ``` +#### Graph Channels + +The `graph` channel type operates on a knowledge graph stored in a Mini-A style `graph.json` file (nodes and edges as keys) and `ow.ch.utils.graph` provides `neighbors`, `path`, `traverse`, `find` and `stats`. + +```javascript +$ch("g").create(true, "graph", { file: "graph.json" }); +$ch("g").set({ id: "doc:a.md" }, { type: "document", props: { title: "A" } }); +$ch("g").set({ id: "doc:b.md" }, { type: "document", props: { title: "B" } }); +$ch("g").set({ from: "doc:a.md", to: "doc:b.md", type: "LINKS_TO" }, {}); +print(ow.ch.utils.graph.path("g", "a.md", "b.md")); // ["doc:a.md", "doc:b.md"] +``` + +See [channel-graph.md](./channel-graph.md) for options, key formats and limits. + --- ## nLinq - Query Processing diff --git a/js/openafconsole.js b/js/openafconsole.js index 036627d2..43d7b908 100755 --- a/js/openafconsole.js +++ b/js/openafconsole.js @@ -585,7 +585,9 @@ function addAlias(aAssignment) { */ function __help(aTerm) { var __ores = "", c = __consoleFormat.helpLine; - var __o = (s, nomd) => __ores += (nomd ? s : ow.format.withMD(s))+ "\n"; + // the help frame uses 3 columns (side line + space + closing side line), so render markdown narrower + var __mdWidth = () => (__ansiflag && con.isAnsiSupported()) ? Math.max(1, con.getConsoleReader().getTerminal().getWidth() - 3) : __; + var __o = (s, nomd) => __ores += (nomd ? s : ow.format.withMD(s, __, __mdWidth()))+ "\n"; if(isUnDef(aTerm) || aTerm.length <= 0) { __o("__help__ Display this help text"); __o("__exit__ Exit this console"); diff --git a/js/owrap.ch.js b/js/owrap.ch.js index 204d6e37..b40a699c 100755 --- a/js/owrap.ch.js +++ b/js/owrap.ch.js @@ -1848,6 +1848,306 @@ OpenWrap.ch.prototype.__types = { } } }, + // Graph implementation + // + /** + * + * ow.ch.types.graph + * + * + */ + graph: { + __channels: Object.create(null), + __prov: [ "EXTRACTED", "INFERRED", "AMBIGUOUS" ], + __nprov: function(p) { + p = isString(p) ? p.toUpperCase().trim() : "AMBIGUOUS" + return this.__prov.indexOf(p) >= 0 ? p : "AMBIGUOUS" + }, + __ekey: function(f, t, ty, pr, pg, sc) { + return stringify([ f, t, ty, this.__nprov(pr), isString(pg) ? pg : "", isString(sc) ? sc : "" ], __, "") + }, + __isEdge: function(k) { return isMap(k) && isDef(k.from) && isDef(k.to) }, + __isNode: function(k) { return isMap(k) && isDef(k.id) && isUnDef(k.from) && isUnDef(k.to) }, + __edgeKeyOf: function(e) { + var p = isMap(e.props) ? e.props : {} + return this.__ekey(e.from, e.to, e.type, e.provenance, p.page, p.section) + }, + __keyOfEdge: function(e) { + var p = isMap(e.props) ? e.props : {} + return { from: e.from, to: e.to, type: e.type, provenance: this.__nprov(e.provenance), page: isString(p.page) ? p.page : "", section: isString(p.section) ? p.section : "" } + }, + __c: function(aName) { + var c = this.__channels[aName] + if (isUnDef(c)) throw "graph channel '" + aName + "' not found" + return c + }, + __l: (m) => (isString(m.lock) ? $flock(m.lock).lock() : __), + __ul: (m) => (isString(m.lock) ? $flock(m.lock).unlock() : __), + __empty: function() { + return { version: 2, created_at: new Date().toISOString(), nodes: {}, edges: [], summaries: { pages: {}, communities: {} }, semantic_cache: {}, communities: [], surprise: [] } + }, + // (re)build the edge index and adjacency from the state + __index: function(c) { + var s = c.state, self = this + if (!isMap(s.nodes)) s.nodes = {} + c.eidx = {} + c.adj = {} + var edges = [] + ;(isArray(s.edges) ? s.edges : []).forEach(function(e) { + if (!isMap(e) || e._deleted === true) return + e.provenance = self.__nprov(e.provenance) + if (!isMap(e.props)) e.props = {} + var k = self.__edgeKeyOf(e) + if (isDef(c.eidx[k])) return + c.eidx[k] = e + edges.push(e) + if (isUnDef(c.adj[e.from])) c.adj[e.from] = [] + if (isUnDef(c.adj[e.to])) c.adj[e.to] = [] + c.adj[e.from].push(e) + if (e.to !== e.from) c.adj[e.to].push(e) + }) + s.edges = edges + }, + __load: function(c) { + var o = c.opts + this.__l(o) + try { + if (io.fileExists(o.file)) { + var r = io.readFileJSON(o.file) + if (!isMap(r)) throw "graph file '" + o.file + "' doesn't contain a valid graph" + c.state = merge(this.__empty(), r) + } else { + if (o.create === false) throw "graph file '" + o.file + "' not found" + c.state = this.__empty() + } + } finally { + this.__ul(o) + } + this.__index(c) + c.dirty = false + c.lastSave = 0 + }, + __flush: function(c) { + var o = c.opts + if (o.readOnly) return false + this.__l(o) + try { + var dir = new java.io.File(o.file).getAbsoluteFile().getParentFile() + if (isDef(dir) && !dir.exists()) dir.mkdirs() + var tmp = o.file + "." + java.util.UUID.randomUUID() + ".tmp" + try { + io.writeFileString(tmp, stringify(c.state, __, "")) + java.nio.file.Files.move(new java.io.File(tmp).toPath(), new java.io.File(o.file).toPath(), java.nio.file.StandardCopyOption.ATOMIC_MOVE, java.nio.file.StandardCopyOption.REPLACE_EXISTING) + } finally { + java.nio.file.Files.deleteIfExists(new java.io.File(tmp).toPath()) + } + } finally { + this.__ul(o) + } + c.dirty = false + c.lastSave = now() + return true + }, + __changed: function(c) { + c.dirty = true + if (c.opts.autosave === "always") { + this.__flush(c) + } else if (c.opts.autosave === "debounced" && now() - c.lastSave >= c.opts.saveDebounceMs) { + this.__flush(c) + } + }, + __rw: function(c) { + if (c.opts.readOnly) throw "graph channel is read-only" + }, + __nodeVal: function(n) { return { id: n.id, type: n.type, props: isMap(n.props) ? clone(n.props) : {} } }, + __edgeVal: function(e) { return { from: e.from, to: e.to, type: e.type, provenance: e.provenance, props: isMap(e.props) ? clone(e.props) : {} } }, + __delEdge: function(c, e) { + var k = this.__edgeKeyOf(e) + delete c.eidx[k] + c.state.edges = c.state.edges.filter(x => x !== e) + ;[ e.from, e.to ].forEach(id => { if (isArray(c.adj[id])) c.adj[id] = c.adj[id].filter(x => x !== e) }) + }, + // public helpers used by ow.ch.utils.graph + __nodes: function(aName) { return this.__c(aName).state.nodes }, + __edgesOf: function(aName, id) { return (this.__c(aName).adj[id] || []) }, + __edges: function(aName) { return this.__c(aName).state.edges }, + __state: function(aName) { return this.__c(aName).state }, + __save: function(aName) { return this.__flush(this.__c(aName)) }, + create : function(aName, shouldCompress, options) { + options = _$(options, "options").isMap().default({}) + var o = { + file : _$(options.file, "options.file").isString().$_(), + readOnly : _$(options.readOnly, "options.readOnly").isBoolean().default(false), + create : _$(options.create, "options.create").isBoolean().default(true), + autosave : String(_$(options.autosave, "options.autosave").isString().default("always")).toLowerCase(), + saveDebounceMs: _$(options.saveDebounceMs, "options.saveDebounceMs").isNumber().default(5000), + lock : options.lock + } + if (["always", "debounced", "off"].indexOf(o.autosave) < 0) throw "options.autosave must be 'always', 'debounced' or 'off'" + var c = { opts: o } + this.__load(c) + this.__channels[aName] = c + }, + destroy : function(aName) { + var c = this.__channels[aName] + if (isUnDef(c)) return + try { if (c.dirty && !c.opts.readOnly) this.__flush(c) } finally { delete this.__channels[aName] } + }, + size : function(aName) { + var c = this.__c(aName) + return Object.keys(c.state.nodes).length + c.state.edges.length + }, + forEach : function(aName, aFunction) { + var ks = this.getKeys(aName) + for (var i = 0; i < ks.length; i++) { + try { aFunction(ks[i], this.get(aName, ks[i])) } catch(e) {} + } + }, + getAll : function(aName, full) { + var c = this.__c(aName), self = this + return Object.keys(c.state.nodes).map(id => self.__nodeVal(c.state.nodes[id])).concat(c.state.edges.map(e => self.__edgeVal(e))) + }, + getKeys : function(aName, full) { + var c = this.__c(aName), self = this + return Object.keys(c.state.nodes).map(id => ({ id: id })).concat(c.state.edges.map(e => self.__keyOfEdge(e))) + }, + getSortedKeys: function(aName, full) { + var c = this.__c(aName), self = this + var ns = Object.keys(c.state.nodes).sort().map(id => ({ id: id })) + var es = c.state.edges.map(e => self.__keyOfEdge(e)).sort((a, b) => stringify([a.from, a.to, a.type, a.provenance, a.page, a.section], __, "") < stringify([b.from, b.to, b.type, b.provenance, b.page, b.section], __, "") ? -1 : 1) + return ns.concat(es) + }, + getSet : function getSet(aName, aMatch, aK, aV, aTimestamp) { + var res = this.get(aName, aK) + if ($stream([res]).anyMatch(aMatch)) { + return this.set(aName, aK, aV, aTimestamp) + } + return __ + }, + set : function(aName, aK, aV, aTimestamp) { + var c = this.__c(aName) + this.__rw(c) + aV = isMap(aV) ? aV : {} + if (this.__isEdge(aK)) { + var from = String(aK.from), to = String(aK.to) + var type = isString(aK.type) ? aK.type : aV.type + if (!isString(type) || type.length == 0) throw "graph edge requires a type" + if (!isMap(c.state.nodes[from]) || !isMap(c.state.nodes[to])) throw "graph edge requires existing nodes (" + from + " -> " + to + ")" + var props = isMap(aV.props) ? clone(aV.props) : {} + if (isString(aK.page) && aK.page.length > 0) props.page = aK.page + if (isString(aK.section) && aK.section.length > 0) props.section = aK.section + var prov = this.__nprov(isDef(aK.provenance) ? aK.provenance : aV.provenance) + var e = { from: from, to: to, type: type, provenance: prov, props: props } + var k = this.__edgeKeyOf(e) + var old = c.eidx[k] + if (isDef(old)) { + old.props = props + } else { + c.eidx[k] = e + c.state.edges.push(e) + if (isUnDef(c.adj[from])) c.adj[from] = [] + if (isUnDef(c.adj[to])) c.adj[to] = [] + c.adj[from].push(e) + if (to !== from) c.adj[to].push(e) + } + } else if (this.__isNode(aK)) { + var id = String(aK.id) + if (id.length == 0) throw "graph node requires a non empty id" + var n = c.state.nodes[id] + var ntype = isString(aV.type) && aV.type.length > 0 ? aV.type : (isMap(n) ? n.type : "concept") + c.state.nodes[id] = { id: id, type: ntype, props: isMap(aV.props) ? clone(aV.props) : {} } + } else { + throw "graph key must have 'id' (node) or 'from' and 'to' (edge)" + } + this.__changed(c) + return aK + }, + setAll : function(aName, aKs, aVs, aTimestamp) { + ow.loadObj() + for(var i = 0; i < aVs.length; i++) { + this.set(aName, ow.obj.filterKeys(aKs, aVs[i]), aVs[i], aTimestamp) + } + }, + unsetAll : function(aName, aKs, aVs, aTimestamp) { + ow.loadObj() + for(var i = 0; i < aVs.length; i++) { + this.unset(aName, ow.obj.filterKeys(aKs, aVs[i]), aTimestamp) + } + }, + get : function(aName, aK) { + var c = this.__c(aName) + if (this.__isEdge(aK)) { + var e = c.eidx[this.__ekey(String(aK.from), String(aK.to), aK.type, aK.provenance, aK.page, aK.section)] + return isDef(e) ? this.__edgeVal(e) : __ + } + if (this.__isNode(aK)) { + var n = c.state.nodes[String(aK.id)] + return isMap(n) ? this.__nodeVal(n) : __ + } + return __ + }, + pop : function(aName) { + var elems = this.getSortedKeys(aName) + return elems[elems.length - 1] + }, + shift : function(aName) { + var elems = this.getSortedKeys(aName) + return elems[0] + }, + unset : function(aName, aK, aTimestamp) { + var c = this.__c(aName) + this.__rw(c) + if (this.__isEdge(aK)) { + var e = c.eidx[this.__ekey(String(aK.from), String(aK.to), aK.type, aK.provenance, aK.page, aK.section)] + if (isUnDef(e)) return + this.__delEdge(c, e) + } else if (this.__isNode(aK)) { + var id = String(aK.id) + if (!isMap(c.state.nodes[id])) return + var self = this + ;(c.adj[id] || []).slice().forEach(e => self.__delEdge(c, e)) + delete c.adj[id] + delete c.state.nodes[id] + } else { + return + } + this.__changed(c) + } + }, // Remote channel implementation // /** @@ -4864,6 +5164,119 @@ OpenWrap.ch.prototype.utils = { } }; +/** + * + * ow.ch.utils.graph + * + * + */ +OpenWrap.ch.prototype.utils.graph = { + __id: function(id) { + id = String(id) + return id.indexOf(":") > 0 ? id : "doc:" + id + }, + __arr: function(v) { return isUnDef(v) ? __ : (isArray(v) ? v : [ v ]) }, + __edges: function(aName, id, o) { + o = _$(o, "options").isMap().default({}) + var dir = isString(o.direction) ? o.direction : "both", types = this.__arr(o.type), provs = this.__arr(o.provenance) + if (isDef(provs)) provs = provs.map(p => String(p).toUpperCase()) + return ow.ch.__types.graph.__edgesOf(aName, id).filter(e => + (dir == "both" || (dir == "out" && e.from === id) || (dir == "in" && e.to === id)) && + (isUnDef(types) || types.indexOf(e.type) >= 0) && + (isUnDef(provs) || provs.indexOf(e.provenance) >= 0) + ) + }, + neighbors: function(aName, anId, aOptions) { + var id = this.__id(anId) + return this.__edges(aName, id, aOptions).map(e => ow.ch.__types.graph.__edgeVal(e)) + }, + degree: function(aName, anId, aOptions) { + return this.__edges(aName, this.__id(anId), aOptions).length + }, + path: function(aName, aFromId, aToId, aOptions) { + aOptions = _$(aOptions, "options").isMap().default({}) + var a = this.__id(aFromId), b = this.__id(aToId), nodes = ow.ch.__types.graph.__nodes(aName) + if (!isMap(nodes[a]) || !isMap(nodes[b])) return [] + var maxDepth = isNumber(aOptions.maxDepth) ? aOptions.maxDepth : __ + var q = [ a ], depth = {}, prev = {} + depth[a] = 0 + for (var qi = 0; qi < q.length && q[qi] !== b; qi++) { + var cur = q[qi] + if (isDef(maxDepth) && depth[cur] >= maxDepth) continue + var es = this.__edges(aName, cur, { type: aOptions.type }) + for (var i = 0; i < es.length; i++) { + var nx = es[i].from === cur ? es[i].to : es[i].from + if (isDef(depth[nx])) continue + depth[nx] = depth[cur] + 1 + prev[nx] = cur + q.push(nx) + } + } + if (isUnDef(depth[b])) return [] + var res = [ b ], p = b + while (p !== a) { p = prev[p]; res.unshift(p) } + return res + }, + traverse: function(aName, anId, aOptions) { + aOptions = _$(aOptions, "options").isMap().default({}) + var id = this.__id(anId), nodes = ow.ch.__types.graph.__nodes(aName), maxDepth = isNumber(aOptions.depth) ? aOptions.depth : 1 + if (!isMap(nodes[id])) return { nodes: [], edges: [] } + var depth = {}, seenE = {}, resE = [], q = [ id ] + depth[id] = 0 + for (var qi = 0; qi < q.length; qi++) { + var cur = q[qi] + if (depth[cur] >= maxDepth) continue + var es = this.__edges(aName, cur, aOptions) + for (var i = 0; i < es.length; i++) { + var e = es[i], k = ow.ch.__types.graph.__edgeKeyOf(e) + if (!seenE[k]) { seenE[k] = true; resE.push(ow.ch.__types.graph.__edgeVal(e)) } + var nx = e.from === cur ? e.to : e.from + if (isDef(depth[nx])) continue + depth[nx] = depth[cur] + 1 + q.push(nx) + } + } + return { nodes: q.map(n => ow.ch.__types.graph.__nodeVal(nodes[n])), edges: resE } + }, + find: function(aName, aOptions) { + aOptions = _$(aOptions, "options").isMap().default({}) + var types = this.__arr(aOptions.type), text = isString(aOptions.text) ? aOptions.text.toLowerCase() : __ + var nodes = ow.ch.__types.graph.__nodes(aName) + return Object.keys(nodes).filter(id => { + var n = nodes[id] + return (isUnDef(types) || types.indexOf(n.type) >= 0) && (isUnDef(text) || (id + " " + stringify(n.props, __, "")).toLowerCase().indexOf(text) >= 0) + }).map(id => ow.ch.__types.graph.__nodeVal(nodes[id])) + }, + stats: function(aName) { + var st = ow.ch.__types.graph.__state(aName), r = { nodes: Object.keys(st.nodes).length, edges: st.edges.length, nodeTypes: {}, edgeTypes: {}, provenance: {} } + Object.keys(st.nodes).forEach(id => { var t = st.nodes[id].type; r.nodeTypes[t] = (r.nodeTypes[t] || 0) + 1 }) + st.edges.forEach(e => { r.edgeTypes[e.type] = (r.edgeTypes[e.type] || 0) + 1; r.provenance[e.provenance] = (r.provenance[e.provenance] || 0) + 1 }) + return r + }, + save: function(aName) { + return ow.ch.__types.graph.__save(aName) + } +}; + OpenWrap.ch.prototype.comms = { __counter: {}, diff --git a/tests/autoTestAll.Channels.js b/tests/autoTestAll.Channels.js index ce8a8e4f..80021a10 100644 --- a/tests/autoTestAll.Channels.js +++ b/tests/autoTestAll.Channels.js @@ -386,6 +386,73 @@ $ch(this.chType + "HK").destroy(); }; + exports.testGraphChannel = function() { + var dir = String(java.nio.file.Files.createTempDirectory("openaf-graph-")); + var file = dir + "/graph.json"; + var name = "graph-test-" + genUUID(); + var g = ow.ch.utils.graph; + try { + // Seed with a mini-a style file holding an extra section that must survive saves + io.writeFileJSON(file, { version: 2, nodes: { "doc:a.md": { id: "doc:a.md", type: "document", props: { title: "A" } } }, edges: [], communities: [ { id: 1 } ] }); + $ch(name).create(false, "graph", { file: file }); + $ch(name).set({ id: "doc:b.md" }, { type: "document", props: { title: "B" } }); + $ch(name).set({ id: "concept:x" }, { type: "concept" }); + $ch(name).set({ id: "concept:island" }, { type: "concept" }); + $ch(name).set({ from: "doc:a.md", to: "doc:b.md", type: "LINKS_TO", provenance: "extracted" }, { props: { page: "a.md" } }); + $ch(name).set({ from: "doc:b.md", to: "concept:x", type: "MENTIONS" }, {}); + $ch(name).set({ from: "doc:b.md", to: "concept:x", type: "MENTIONS" }, {}); + ow.test.assert($ch(name).size(), 6, "Problem with graph size (edge dedupe)"); + ow.test.assert($ch(name).get({ id: "doc:b.md" }).props.title, "B", "Problem getting a graph node"); + ow.test.assert($ch(name).get({ from: "doc:a.md", to: "doc:b.md", type: "LINKS_TO", provenance: "EXTRACTED", page: "a.md" }).provenance, "EXTRACTED", "Problem getting a graph edge"); + + var failed = false; + try { $ch(name).set({ from: "doc:a.md", to: "doc:none", type: "T" }, {}); } catch(e) { failed = true; } + ow.test.assert(failed, true, "Graph edge to a missing node must fail"); + + ow.test.assert(g.path(name, "a.md", "concept:x"), [ "doc:a.md", "doc:b.md", "concept:x" ], "Problem with graph path"); + ow.test.assert(g.path(name, "a.md", "concept:island"), [], "Graph path to a disconnected node must be empty"); + ow.test.assert(g.path(name, "a.md", "concept:x", { maxDepth: 1 }), [], "Graph path must respect maxDepth"); + ow.test.assert(g.neighbors(name, "b.md", { direction: "out" }).length, 1, "Problem with graph neighbors (out)"); + ow.test.assert(g.neighbors(name, "b.md").length, 2, "Problem with graph neighbors (both)"); + ow.test.assert(g.degree(name, "b.md", { type: "MENTIONS" }), 1, "Problem with graph degree"); + ow.test.assert(g.traverse(name, "a.md", { depth: 1 }).nodes.length, 2, "Problem with graph traverse depth 1"); + ow.test.assert(g.traverse(name, "a.md", { depth: 2 }).nodes.length, 3, "Problem with graph traverse depth 2"); + ow.test.assert(g.find(name, { type: "concept" }).length, 2, "Problem with graph find by type"); + ow.test.assert(g.stats(name).edges, 2, "Problem with graph stats"); + + // Persistence, preserved sections and reload + var saved = io.readFileJSON(file); + ow.test.assert(saved.communities, [ { id: 1 } ], "Graph channel must preserve unknown sections"); + $ch(name).destroy(); + $ch(name).create(false, "graph", { file: file }); + ow.test.assert($ch(name).size(), 6, "Problem reloading a graph"); + + // Cascade delete + $ch(name).unset({ id: "doc:b.md" }); + ow.test.assert($ch(name).size(), 3, "Unsetting a graph node must remove its edges"); + $ch(name).destroy(); + + // Read-only + $ch(name).create(false, "graph", { file: file, readOnly: true }); + failed = false; + try { $ch(name).set({ id: "z" }, {}); } catch(e) { failed = true; } + ow.test.assert(failed, true, "Read-only graph must reject changes"); + ow.test.assert($ch(name).size(), 3, "Read-only graph must keep its content"); + $ch(name).destroy(); + + // Debounced autosave flushes on destroy + $ch(name).create(false, "graph", { file: file, autosave: "debounced", saveDebounceMs: 3600000 }); + $ch(name).set({ id: "doc:late" }, { type: "document" }); + $ch(name).set({ id: "doc:later" }, { type: "document" }); + ow.test.assert(isDef(io.readFileJSON(file).nodes["doc:later"]), false, "Debounced graph must not save immediately"); + $ch(name).destroy(); + ow.test.assert(isDef(io.readFileJSON(file).nodes["doc:later"]), true, "Debounced graph must save on destroy"); + } finally { + $ch(name).destroy(); + io.rm(dir); + } + }; + exports.testAll = function() { $ch("allA").destroy(); $ch("allB").destroy(); diff --git a/tests/autoTestAll.Channels.yaml b/tests/autoTestAll.Channels.yaml index 7bda95e1..1b7e7f87 100644 --- a/tests/autoTestAll.Channels.yaml +++ b/tests/autoTestAll.Channels.yaml @@ -122,6 +122,11 @@ jobs: to : oJob Test exec: args.func = args.tests.testMVSUtils; + - name: Channels::Graph channel + from: Channels::Init + to : oJob Test + exec: args.func = args.tests.testGraphChannel; + - name: Channels::Test channel (all) from: Channels::Init to : oJob Test @@ -207,6 +212,9 @@ todo: - Channels::Test channel (all) + # Graph + - Channels::Graph channel + # Ignite #- name: Channels::Creating a channel # args: From 145fa3c6c39cd9f7666e5bcb26fe09a521e363dd Mon Sep 17 00:00:00 2001 From: nmaguiar Date: Fri, 16 Oct 2026 05:18:29 +0100 Subject: [PATCH 02/19] docs(channel-graph): add channel graph documentation Add detailed documentation for the channel graph in docs/channel-graph.md. --- docs/channel-graph.md | 89 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) diff --git a/docs/channel-graph.md b/docs/channel-graph.md index ec1a7380..4e34f3a7 100644 --- a/docs/channel-graph.md +++ b/docs/channel-graph.md @@ -128,6 +128,95 @@ io.rm(root); Edges created without a `provenance` are counted as `AMBIGUOUS`; add `provenance: "EXTRACTED"` to the edge key when the relation is known for certain, as it is for containment here. +## More use cases + +The graph does not have to come from Mini-A: any set of things and relations fits. The snippets below assume the helpers `N` and `E` and the shortcut `G`: + +```javascript +$ch("g").create(true, "graph", { file: "infra.json" }); +var G = ow.ch.utils.graph; +var N = (id, type, props) => $ch("g").set({ id: id }, { type: type, props: props || {} }); +var E = (from, to, type, prov) => $ch("g").set({ from: from, to: to, type: type, provenance: prov || "EXTRACTED" }, {}); +``` + +### Service dependencies and impact analysis + +Model services and datastores with `DEPENDS_ON` edges, then ask what breaks when something goes down. Following edges backwards (`direction: "in"`) from a node lists everything that depends on it, directly or not. + +```javascript +N("svc:web", "service"); N("svc:api", "service"); N("svc:auth", "service"); +N("svc:queue", "service"); N("svc:db", "database"); +E("svc:web", "svc:api", "DEPENDS_ON"); E("svc:api", "svc:auth", "DEPENDS_ON"); +E("svc:api", "svc:db", "DEPENDS_ON"); E("svc:auth", "svc:db", "DEPENDS_ON"); +E("svc:queue", "svc:db", "DEPENDS_ON"); + +// Blast radius if the database fails (includes the database itself) +G.traverse("g", "svc:db", { depth: 5, direction: "in", type: "DEPENDS_ON" }).nodes.map(n => n.id); +// [ "svc:db", "svc:api", "svc:auth", "svc:queue", "svc:web" ] + +// Most critical component: how many services depend directly on it +G.degree("g", "svc:db", { direction: "in" }); // 3 + +// Entry points: services nothing else depends on +G.find("g", { type: "service" }).filter(n => G.degree("g", n.id, { direction: "in" }) == 0).map(n => n.id); +// [ "svc:web", "svc:queue" ] + +// How does the web tier reach the database? +G.path("g", "svc:web", "svc:db"); // [ "svc:web", "svc:api", "svc:db" ] +``` + +Rebuild the graph from your CMDB, Kubernetes manifests or Terraform state on a schedule, and keep the file in version control to diff the architecture between releases. + +### Review uncertain relations + +`provenance` records how sure you are about a relation. Edges created without it are `AMBIGUOUS`, and heuristics (naming conventions, log correlation, an LLM) can mark theirs `INFERRED`. Use that to build a review queue and promote the edges a person confirms. Provenance is part of the edge identity, so promoting means unsetting the old edge and setting a new one. + +```javascript +N("person:ana", "person"); +$ch("g").set({ from: "person:ana", to: "svc:api", type: "OWNS" }, {}); // AMBIGUOUS + +// Review queue: everything that is not certain +G.neighbors("g", "person:ana", { provenance: ["INFERRED", "AMBIGUOUS"] }); + +// Promote after confirmation +$ch("g").getKeys().filter(k => k.from).forEach(k => { + var e = $ch("g").get(k); + if (e.provenance == "AMBIGUOUS") { + $ch("g").unset(k); + $ch("g").set({ from: e.from, to: e.to, type: e.type, provenance: "EXTRACTED" }, { props: e.props }); + } +}); +G.stats("g").provenance; // only EXTRACTED edges left +``` + +### Ownership and "who do I ask?" + +Add people, teams and topics as nodes with `OWNS`, `MEMBER_OF` or `KNOWS` edges. `path` finds the shortest chain between any two nodes, which turns into an escalation route or an explanation of why two things are related. `find` with `text` searches ids and properties, so it works as a quick lookup. + +```javascript +G.find("g", { type: "person", text: "ana" }); // who is Ana? +G.neighbors("g", "svc:api", { direction: "in", type: "OWNS" }).map(e => e.from); // owners of the API +G.path("g", "person:ana", "svc:db"); // how is Ana connected to the database? +``` + +### Export for visualization + +Edge keys carry everything needed to draw the graph, so exporting to Mermaid (or Graphviz, or `oafp`) is a few lines. + +```javascript +var lines = [ "graph LR" ]; +$ch("g").getKeys().filter(k => k.from).forEach(k => + lines.push(" " + k.from.replace(/\W/g, "_") + " -->|" + k.type + "| " + k.to.replace(/\W/g, "_"))); +io.writeFileString("infra.mmd", lines.join("\n")); +``` + +### Other ideas + +- **Document and code maps**: `IMPORTS` or `LINKS_TO` edges between files, to find orphans (`degree` of 0) or circular references. +- **Data lineage**: `PRODUCES`/`CONSUMES` edges between jobs and datasets; `traverse` with `direction: "out"` shows everything downstream of a changed table. +- **Incident and runbook index**: link alerts, services and runbooks, then `neighbors` of an alert gives the runbook to follow. +- **Shared, read-only reference data**: ship a curated `graph.json` with an application and open it with `readOnly: true`. + ## Working with Mini-A The format is the one Mini-A saves in `/graph.json` (default `./.mini-a-wiki-graph`), so you can inspect or edit a graph produced by Mini-A without loading its code: From a7d3f3ff859bdc338f7299313539e86bd42ea8e6 Mon Sep 17 00:00:00 2001 From: nmaguiar Date: Fri, 16 Oct 2026 05:46:07 +0100 Subject: [PATCH 03/19] feat(channels): update channel wrapper logic and documentation - Update js/owrap.ch.js with significant logic changes - Expand docs/channel-graph.md - Add comprehensive tests in tests/autoTestAll.Channels.js and .yaml --- docs/channel-graph.md | 38 +++++ js/owrap.ch.js | 266 ++++++++++++++++++++++++-------- tests/autoTestAll.Channels.js | 91 +++++++++++ tests/autoTestAll.Channels.yaml | 6 + 4 files changed, 338 insertions(+), 63 deletions(-) diff --git a/docs/channel-graph.md b/docs/channel-graph.md index 4e34f3a7..0bb73375 100644 --- a/docs/channel-graph.md +++ b/docs/channel-graph.md @@ -31,9 +31,47 @@ $ch("g").create(true, "graph", { file: "graph.json" }); | `autosave` | String | `"always"` | `"always"` saves on every change, `"debounced"` saves at most every `saveDebounceMs`, `"off"` only saves on `save`/destroy | | `saveDebounceMs` | Number | `5000` | Minimum interval between saves when `autosave` is `"debounced"` | | `lock` | String | | Path to a dummy file used as a filesystem lock while loading and saving | +| `store` | Map | | Delegate nodes, edges and the remaining sections to another channel type instead of keeping them in memory (see [Keep the graph out of memory](#keep-the-graph-out-of-memory)). Can't be combined with `file` | Saves write a temporary file and atomically move it over `file`. Destroying the channel flushes pending changes (unless read-only). A missing file with `create: false`, or a file that is not a JSON map, makes `create` throw. +## Keep the graph out of memory + +By default the whole graph lives in memory and `file` is rewritten on every save. With `store` the graph is kept in other OpenAF channels (nodes, edges, adjacency and the remaining sections, one channel each, named `graph::::`), so the memory used is bounded by that channel's own caching instead of by the graph size. Memory stays the default. + +```javascript +$ch("g").create(true, "graph", { store: { type: "mvs", options: { file: "graph.db", compact: true } } }); +``` + +| `store` field | Type | Description | +| --- | --- | --- | +| `type` | String | Channel type used for every part (e.g. `mvs`) | +| `options` | Map or Function | Options for the underlying channels. A Map gets a distinct `map` name per part added (so one `mvs` file holds them all); a Function receives the part name (`nodes`, `edges`, `adj`, `meta`) and returns that part's options | +| `compress` | Boolean | Passed as `shouldCompress` when creating the underlying channels | +| `bulk` | Boolean | Use the underlying channel's `setAll` for `setAll` batches (defaults to `true` for `mvs`, `false` otherwise) | + +With a store, `file`, `autosave`, `saveDebounceMs` and `lock` don't apply: every change goes straight to the store channel. `create: false` throws if the store is empty, and `readOnly` works as usual. Destroying the graph channel destroys the underlying ones. The `ow.ch.utils.graph` helpers work unchanged. + +### When does it make sense? + +Measured with `mvs` (80,000 nodes with small props and 80,000 edges, JDK 21; indicative numbers): + +| | In memory (JSON file) | `mvs` store | +| --- | --- | --- | +| Open an existing graph | ~2.2 s | ~0.05 s | +| Heap after open + 2,000 `neighbors`, 200 `get`, one `path` | ~154 MB | ~12 MB | +| Writing 160,000 entries one `set` at a time | ~2.9 s | ~75 s (~25x slower) | +| Writing 160,000 entries in one `setAll` | ~1.4 s | ~6 s (~4x slower) | +| Point reads (`get`, `neighbors`) | baseline | ~3x slower | +| Whole-graph scans (`stats`, `find`) | baseline | ~10-50x slower | + +Use a store when the graph is large relative to the heap, is mostly read, or you can't afford to load it at start-up. Stay in memory for small graphs (a few thousand nodes), write-heavy loads or frequent `stats`/`find`. Notes: + +- Load in batches: `setAll` writes a whole batch to the store at once (one `mvs` commit, and each node's adjacency written once), about 12x faster than the same entries with `set`. It follows the same rules as sequential `set` calls (an edge needs its nodes to exist earlier in the batch or already stored). Batching is on for `mvs`; set `store.bulk` to `true` or `false` to force it for other types that implement `setAll` properly. +- `mvs` commits on every change and reuses space lazily, so a long write session can bloat the file badly (7 GB for the load above). Set `compact: true` in the `mvs` options so it is compacted when the channel is destroyed (61 MB for the same graph). +- `getKeys`, `getAll`, `find` and `stats` still materialize node ids and edges in memory while they run; avoid them on very large graphs. +- Unlike the JSON file, a store is not in the Mini-A `graph.json` format. + ## Keys and values | Entry | Key | Value | diff --git a/js/owrap.ch.js b/js/owrap.ch.js index b40a699c..6d0f7ed5 100755 --- a/js/owrap.ch.js +++ b/js/owrap.ch.js @@ -1867,6 +1867,7 @@ OpenWrap.ch.prototype.__types = { * | `autosave` | String | `"always"` | `"always"`, `"debounced"` or `"off"`; pending changes are always flushed on destroy | * | `saveDebounceMs` | Number | `5000` | Minimum ms between saves when `autosave` is `"debounced"` | * | `lock` | String | | Path to a dummy file used as a filesystem lock while loading/saving | + * | `store` | Map | | `{ type, options, compress, bulk }`: keep nodes, edges and other sections in channels of that type (e.g. `mvs`) instead of memory; can't be combined with `file` | * * Keys and values: * @@ -1944,6 +1945,8 @@ OpenWrap.ch.prototype.__types = { }, __load: function(c) { var o = c.opts + if (isDef(o.store)) return this.__loadStore(c) + c.st = this.__memStore(c) this.__l(o) try { if (io.fileExists(o.file)) { @@ -1964,6 +1967,7 @@ OpenWrap.ch.prototype.__types = { __flush: function(c) { var o = c.opts if (o.readOnly) return false + if (isDef(o.store)) return true this.__l(o) try { var dir = new java.io.File(o.file).getAbsoluteFile().getParentFile() @@ -1983,6 +1987,7 @@ OpenWrap.ch.prototype.__types = { return true }, __changed: function(c) { + if (isDef(c.opts.store)) return c.dirty = true if (c.opts.autosave === "always") { this.__flush(c) @@ -1995,22 +2000,156 @@ OpenWrap.ch.prototype.__types = { }, __nodeVal: function(n) { return { id: n.id, type: n.type, props: isMap(n.props) ? clone(n.props) : {} } }, __edgeVal: function(e) { return { from: e.from, to: e.to, type: e.type, provenance: e.provenance, props: isMap(e.props) ? clone(e.props) : {} } }, - __delEdge: function(c, e) { - var k = this.__edgeKeyOf(e) - delete c.eidx[k] - c.state.edges = c.state.edges.filter(x => x !== e) - ;[ e.from, e.to ].forEach(id => { if (isArray(c.adj[id])) c.adj[id] = c.adj[id].filter(x => x !== e) }) + // Storage backends: every access to nodes and edges goes through c.st + // memory: everything held in c.state (+ edge index and adjacency) + __memStore: function(c) { + var self = this + var delEdge = function(e) { + delete c.eidx[self.__edgeKeyOf(e)] + c.state.edges = c.state.edges.filter(x => x !== e) + ;[ e.from, e.to ].forEach(id => { if (isArray(c.adj[id])) c.adj[id] = c.adj[id].filter(x => x !== e) }) + } + return { + getNode : id => c.state.nodes[id], + putNode : n => { c.state.nodes[n.id] = n }, + nodeIds : () => Object.keys(c.state.nodes), + nodeCount: () => Object.keys(c.state.nodes).length, + getEdge : k => c.eidx[k], + putEdge : e => { + var k = self.__edgeKeyOf(e), old = c.eidx[k] + if (isDef(old)) { old.props = e.props; return } + c.eidx[k] = e + c.state.edges.push(e) + if (isUnDef(c.adj[e.from])) c.adj[e.from] = [] + if (isUnDef(c.adj[e.to])) c.adj[e.to] = [] + c.adj[e.from].push(e) + if (e.to !== e.from) c.adj[e.to].push(e) + }, + bulk : false, + delEdge : delEdge, + edgesOf : id => (c.adj[id] || []), + edges : () => c.state.edges, + edgeCount: () => c.state.edges.length, + delNode : id => { + (c.adj[id] || []).slice().forEach(delEdge) + delete c.adj[id] + delete c.state.nodes[id] + }, + close : () => {} + } + }, + // channel: nodes, edges, adjacency and the remaining sections are delegated to other channels (e.g. mvs) + __chStore: function(c, aName) { + var self = this, spec = c.opts.store, pre = "graph::" + aName + "::", n = {} + var parts = [ "nodes", "edges", "adj", "meta" ], made = [] + var close = () => made.forEach(x => { try { $ch(x).destroy() } catch(e) {} }) + try { + parts.forEach(p => { + n[p] = pre + p + var o = isFunction(spec.options) ? spec.options(p) : merge(isMap(spec.options) ? clone(spec.options) : {}, { map: pre + p }) + $ch(n[p]).create(spec.compress === true, spec.type, o) + made.push(n[p]) + }) + } catch(e) { close(); throw e } + var edgesOf = id => { + var a = $ch(n.adj).get({ id: id }) + return isMap(a) ? a.ks.map(k => $ch(n.edges).get({ k: k })).filter(isMap) : [] + } + var addAdj = (id, k) => { + var a = $ch(n.adj).get({ id: id }) + a = isMap(a) ? a : { id: id, ks: [] } + a.ks.push(k) + $ch(n.adj).set({ id: id }, a) + } + var delAdj = (id, k) => { + var a = $ch(n.adj).get({ id: id }) + if (!isMap(a)) return + a.ks = a.ks.filter(x => x !== k) + if (a.ks.length == 0) $ch(n.adj).unset({ id: id }); else $ch(n.adj).set({ id: id }, a) + } + var delEdge = e => { + var k = self.__edgeKeyOf(e) + $ch(n.edges).unset({ k: k }) + delAdj(e.from, k) + if (e.to !== e.from) delAdj(e.to, k) + } + return { + names : n, + getNode : id => $ch(n.nodes).get({ id: id }), + putNode : nd => { $ch(n.nodes).set({ id: nd.id }, nd) }, + nodeIds : () => $ch(n.nodes).getKeys().map(k => k.id), + nodeCount: () => $ch(n.nodes).size(), + getEdge : k => $ch(n.edges).get({ k: k }), + bulk : spec.bulk === true || (spec.bulk !== false && spec.type == "mvs"), + putBatch : (nodes, edges) => { + if (nodes.length > 0) $ch(n.nodes).setAll([ "id" ], nodes) + var uniq = {}, adds = {}, ids = [] + edges.forEach(e => { uniq[self.__edgeKeyOf(e)] = e }) + var vals = Object.keys(uniq).map(k => { + var e = uniq[k] + if (!isMap($ch(n.edges).get({ k: k }))) { + ;(e.to !== e.from ? [ e.from, e.to ] : [ e.from ]).forEach(id => { + if (isUnDef(adds[id])) { adds[id] = []; ids.push(id) } + adds[id].push(k) + }) + } + return merge(e, { k: k }) + }) + if (vals.length > 0) $ch(n.edges).setAll([ "k" ], vals) + var recs = ids.map(id => { + var a = $ch(n.adj).get({ id: id }) + return { id: id, ks: (isMap(a) ? a.ks : []).concat(adds[id]) } + }) + if (recs.length > 0) $ch(n.adj).setAll([ "id" ], recs) + }, + putEdge : e => { + var k = self.__edgeKeyOf(e), old = $ch(n.edges).get({ k: k }) + $ch(n.edges).set({ k: k }, merge(e, { k: k })) + if (!isMap(old)) { + addAdj(e.from, k) + if (e.to !== e.from) addAdj(e.to, k) + } + }, + delEdge : delEdge, + edgesOf : edgesOf, + edges : () => $ch(n.edges).getAll(), + edgeCount: () => $ch(n.edges).size(), + delNode : id => { + edgesOf(id).forEach(delEdge) + $ch(n.adj).unset({ id: id }) + $ch(n.nodes).unset({ id: id }) + }, + close : close + } + }, + __loadStore: function(c) { + var o = c.opts, st = this.__chStore(c, c.name) + c.st = st + try { + var meta = $ch(st.names.meta).get({ k: "meta" }) + if (!isMap(meta)) { + if (o.create === false) throw "graph store for '" + c.name + "' not found" + meta = { k: "meta", state: this.__empty() } + delete meta.state.nodes + delete meta.state.edges + if (!o.readOnly) $ch(st.names.meta).set({ k: "meta" }, meta) + } + c.state = meta.state + } catch(e) { st.close(); throw e } + c.dirty = false + c.lastSave = 0 }, // public helpers used by ow.ch.utils.graph - __nodes: function(aName) { return this.__c(aName).state.nodes }, - __edgesOf: function(aName, id) { return (this.__c(aName).adj[id] || []) }, - __edges: function(aName) { return this.__c(aName).state.edges }, - __state: function(aName) { return this.__c(aName).state }, + __getNode: function(aName, id) { return this.__c(aName).st.getNode(id) }, + __nodeIds: function(aName) { return this.__c(aName).st.nodeIds() }, + __edgesOf: function(aName, id) { return this.__c(aName).st.edgesOf(id) }, + __edges: function(aName) { return this.__c(aName).st.edges() }, __save: function(aName) { return this.__flush(this.__c(aName)) }, create : function(aName, shouldCompress, options) { options = _$(options, "options").isMap().default({}) var o = { - file : _$(options.file, "options.file").isString().$_(), + file : isDef(options.store) ? options.file : _$(options.file, "options.file").isString().$_(), + store : isDef(options.store) ? _$(options.store, "options.store").isMap().$_() : __, readOnly : _$(options.readOnly, "options.readOnly").isBoolean().default(false), create : _$(options.create, "options.create").isBoolean().default(true), autosave : String(_$(options.autosave, "options.autosave").isString().default("always")).toLowerCase(), @@ -2018,18 +2157,22 @@ OpenWrap.ch.prototype.__types = { lock : options.lock } if (["always", "debounced", "off"].indexOf(o.autosave) < 0) throw "options.autosave must be 'always', 'debounced' or 'off'" - var c = { opts: o } + if (isDef(o.store)) { + _$(o.store.type, "options.store.type").isString().$_() + if (isDef(o.file)) throw "options.file and options.store can't be used together" + } + var c = { opts: o, name: aName } this.__load(c) this.__channels[aName] = c }, destroy : function(aName) { var c = this.__channels[aName] if (isUnDef(c)) return - try { if (c.dirty && !c.opts.readOnly) this.__flush(c) } finally { delete this.__channels[aName] } + try { if (c.dirty && !c.opts.readOnly) this.__flush(c) } finally { delete this.__channels[aName]; c.st.close() } }, size : function(aName) { var c = this.__c(aName) - return Object.keys(c.state.nodes).length + c.state.edges.length + return c.st.nodeCount() + c.st.edgeCount() }, forEach : function(aName, aFunction) { var ks = this.getKeys(aName) @@ -2039,16 +2182,16 @@ OpenWrap.ch.prototype.__types = { }, getAll : function(aName, full) { var c = this.__c(aName), self = this - return Object.keys(c.state.nodes).map(id => self.__nodeVal(c.state.nodes[id])).concat(c.state.edges.map(e => self.__edgeVal(e))) + return c.st.nodeIds().map(id => self.__nodeVal(c.st.getNode(id))).concat(c.st.edges().map(e => self.__edgeVal(e))) }, getKeys : function(aName, full) { var c = this.__c(aName), self = this - return Object.keys(c.state.nodes).map(id => ({ id: id })).concat(c.state.edges.map(e => self.__keyOfEdge(e))) + return c.st.nodeIds().map(id => ({ id: id })).concat(c.st.edges().map(e => self.__keyOfEdge(e))) }, getSortedKeys: function(aName, full) { var c = this.__c(aName), self = this - var ns = Object.keys(c.state.nodes).sort().map(id => ({ id: id })) - var es = c.state.edges.map(e => self.__keyOfEdge(e)).sort((a, b) => stringify([a.from, a.to, a.type, a.provenance, a.page, a.section], __, "") < stringify([b.from, b.to, b.type, b.provenance, b.page, b.section], __, "") ? -1 : 1) + var ns = c.st.nodeIds().sort().map(id => ({ id: id })) + var es = c.st.edges().map(e => self.__keyOfEdge(e)).sort((a, b) => stringify([a.from, a.to, a.type, a.provenance, a.page, a.section], __, "") < stringify([b.from, b.to, b.type, b.provenance, b.page, b.section], __, "") ? -1 : 1) return ns.concat(es) }, getSet : function getSet(aName, aMatch, aK, aV, aTimestamp) { @@ -2058,49 +2201,50 @@ OpenWrap.ch.prototype.__types = { } return __ }, - set : function(aName, aK, aV, aTimestamp) { - var c = this.__c(aName) - this.__rw(c) + // validates a key/value and returns { node } or { edge }; pend (optional) holds nodes of the same batch not yet stored + __prep : function(c, aK, aV, pend) { aV = isMap(aV) ? aV : {} + var getN = id => (isDef(pend) && isMap(pend[id])) ? pend[id] : c.st.getNode(id) if (this.__isEdge(aK)) { var from = String(aK.from), to = String(aK.to) var type = isString(aK.type) ? aK.type : aV.type if (!isString(type) || type.length == 0) throw "graph edge requires a type" - if (!isMap(c.state.nodes[from]) || !isMap(c.state.nodes[to])) throw "graph edge requires existing nodes (" + from + " -> " + to + ")" + if (!isMap(getN(from)) || !isMap(getN(to))) throw "graph edge requires existing nodes (" + from + " -> " + to + ")" var props = isMap(aV.props) ? clone(aV.props) : {} if (isString(aK.page) && aK.page.length > 0) props.page = aK.page if (isString(aK.section) && aK.section.length > 0) props.section = aK.section var prov = this.__nprov(isDef(aK.provenance) ? aK.provenance : aV.provenance) - var e = { from: from, to: to, type: type, provenance: prov, props: props } - var k = this.__edgeKeyOf(e) - var old = c.eidx[k] - if (isDef(old)) { - old.props = props - } else { - c.eidx[k] = e - c.state.edges.push(e) - if (isUnDef(c.adj[from])) c.adj[from] = [] - if (isUnDef(c.adj[to])) c.adj[to] = [] - c.adj[from].push(e) - if (to !== from) c.adj[to].push(e) - } + return { edge: { from: from, to: to, type: type, provenance: prov, props: props } } } else if (this.__isNode(aK)) { var id = String(aK.id) if (id.length == 0) throw "graph node requires a non empty id" - var n = c.state.nodes[id] + var n = getN(id) var ntype = isString(aV.type) && aV.type.length > 0 ? aV.type : (isMap(n) ? n.type : "concept") - c.state.nodes[id] = { id: id, type: ntype, props: isMap(aV.props) ? clone(aV.props) : {} } - } else { - throw "graph key must have 'id' (node) or 'from' and 'to' (edge)" + return { node: { id: id, type: ntype, props: isMap(aV.props) ? clone(aV.props) : {} } } } + throw "graph key must have 'id' (node) or 'from' and 'to' (edge)" + }, + set : function(aName, aK, aV, aTimestamp) { + var c = this.__c(aName) + this.__rw(c) + var r = this.__prep(c, aK, aV) + if (isDef(r.edge)) c.st.putEdge(r.edge); else c.st.putNode(r.node) this.__changed(c) return aK }, + // one change notification (and, with a bulk capable store, one write) for the whole batch setAll : function(aName, aKs, aVs, aTimestamp) { ow.loadObj() + var c = this.__c(aName) + this.__rw(c) + var pend = {}, nodes = [], edges = [] for(var i = 0; i < aVs.length; i++) { - this.set(aName, ow.obj.filterKeys(aKs, aVs[i]), aVs[i], aTimestamp) + var r = this.__prep(c, ow.obj.filterKeys(aKs, aVs[i]), aVs[i], pend) + if (isDef(r.node)) { pend[r.node.id] = r.node; nodes.push(r.node) } else edges.push(r.edge) + if (!c.st.bulk) { if (isDef(r.node)) c.st.putNode(r.node); else c.st.putEdge(r.edge) } } + if (c.st.bulk) c.st.putBatch(nodes, edges) + this.__changed(c) }, unsetAll : function(aName, aKs, aVs, aTimestamp) { ow.loadObj() @@ -2111,11 +2255,11 @@ OpenWrap.ch.prototype.__types = { get : function(aName, aK) { var c = this.__c(aName) if (this.__isEdge(aK)) { - var e = c.eidx[this.__ekey(String(aK.from), String(aK.to), aK.type, aK.provenance, aK.page, aK.section)] - return isDef(e) ? this.__edgeVal(e) : __ + var e = c.st.getEdge(this.__ekey(String(aK.from), String(aK.to), aK.type, aK.provenance, aK.page, aK.section)) + return isMap(e) ? this.__edgeVal(e) : __ } if (this.__isNode(aK)) { - var n = c.state.nodes[String(aK.id)] + var n = c.st.getNode(String(aK.id)) return isMap(n) ? this.__nodeVal(n) : __ } return __ @@ -2132,16 +2276,13 @@ OpenWrap.ch.prototype.__types = { var c = this.__c(aName) this.__rw(c) if (this.__isEdge(aK)) { - var e = c.eidx[this.__ekey(String(aK.from), String(aK.to), aK.type, aK.provenance, aK.page, aK.section)] - if (isUnDef(e)) return - this.__delEdge(c, e) + var e = c.st.getEdge(this.__ekey(String(aK.from), String(aK.to), aK.type, aK.provenance, aK.page, aK.section)) + if (!isMap(e)) return + c.st.delEdge(e) } else if (this.__isNode(aK)) { var id = String(aK.id) - if (!isMap(c.state.nodes[id])) return - var self = this - ;(c.adj[id] || []).slice().forEach(e => self.__delEdge(c, e)) - delete c.adj[id] - delete c.state.nodes[id] + if (!isMap(c.st.getNode(id))) return + c.st.delNode(id) } else { return } @@ -5214,8 +5355,8 @@ OpenWrap.ch.prototype.utils.graph = { }, path: function(aName, aFromId, aToId, aOptions) { aOptions = _$(aOptions, "options").isMap().default({}) - var a = this.__id(aFromId), b = this.__id(aToId), nodes = ow.ch.__types.graph.__nodes(aName) - if (!isMap(nodes[a]) || !isMap(nodes[b])) return [] + var a = this.__id(aFromId), b = this.__id(aToId), gt = ow.ch.__types.graph + if (!isMap(gt.__getNode(aName, a)) || !isMap(gt.__getNode(aName, b))) return [] var maxDepth = isNumber(aOptions.maxDepth) ? aOptions.maxDepth : __ var q = [ a ], depth = {}, prev = {} depth[a] = 0 @@ -5238,8 +5379,8 @@ OpenWrap.ch.prototype.utils.graph = { }, traverse: function(aName, anId, aOptions) { aOptions = _$(aOptions, "options").isMap().default({}) - var id = this.__id(anId), nodes = ow.ch.__types.graph.__nodes(aName), maxDepth = isNumber(aOptions.depth) ? aOptions.depth : 1 - if (!isMap(nodes[id])) return { nodes: [], edges: [] } + var id = this.__id(anId), gt = ow.ch.__types.graph, maxDepth = isNumber(aOptions.depth) ? aOptions.depth : 1 + if (!isMap(gt.__getNode(aName, id))) return { nodes: [], edges: [] } var depth = {}, seenE = {}, resE = [], q = [ id ] depth[id] = 0 for (var qi = 0; qi < q.length; qi++) { @@ -5255,21 +5396,20 @@ OpenWrap.ch.prototype.utils.graph = { q.push(nx) } } - return { nodes: q.map(n => ow.ch.__types.graph.__nodeVal(nodes[n])), edges: resE } + return { nodes: q.map(n => gt.__nodeVal(gt.__getNode(aName, n))), edges: resE } }, find: function(aName, aOptions) { aOptions = _$(aOptions, "options").isMap().default({}) var types = this.__arr(aOptions.type), text = isString(aOptions.text) ? aOptions.text.toLowerCase() : __ - var nodes = ow.ch.__types.graph.__nodes(aName) - return Object.keys(nodes).filter(id => { - var n = nodes[id] - return (isUnDef(types) || types.indexOf(n.type) >= 0) && (isUnDef(text) || (id + " " + stringify(n.props, __, "")).toLowerCase().indexOf(text) >= 0) - }).map(id => ow.ch.__types.graph.__nodeVal(nodes[id])) + var gt = ow.ch.__types.graph + return gt.__nodeIds(aName).map(id => gt.__getNode(aName, id)).filter(n => + (isUnDef(types) || types.indexOf(n.type) >= 0) && (isUnDef(text) || (n.id + " " + stringify(n.props, __, "")).toLowerCase().indexOf(text) >= 0) + ).map(n => gt.__nodeVal(n)) }, stats: function(aName) { - var st = ow.ch.__types.graph.__state(aName), r = { nodes: Object.keys(st.nodes).length, edges: st.edges.length, nodeTypes: {}, edgeTypes: {}, provenance: {} } - Object.keys(st.nodes).forEach(id => { var t = st.nodes[id].type; r.nodeTypes[t] = (r.nodeTypes[t] || 0) + 1 }) - st.edges.forEach(e => { r.edgeTypes[e.type] = (r.edgeTypes[e.type] || 0) + 1; r.provenance[e.provenance] = (r.provenance[e.provenance] || 0) + 1 }) + var gt = ow.ch.__types.graph, ids = gt.__nodeIds(aName), es = gt.__edges(aName), r = { nodes: ids.length, edges: es.length, nodeTypes: {}, edgeTypes: {}, provenance: {} } + ids.forEach(id => { var t = gt.__getNode(aName, id).type; r.nodeTypes[t] = (r.nodeTypes[t] || 0) + 1 }) + es.forEach(e => { r.edgeTypes[e.type] = (r.edgeTypes[e.type] || 0) + 1; r.provenance[e.provenance] = (r.provenance[e.provenance] || 0) + 1 }) return r }, save: function(aName) { diff --git a/tests/autoTestAll.Channels.js b/tests/autoTestAll.Channels.js index 80021a10..429c470f 100644 --- a/tests/autoTestAll.Channels.js +++ b/tests/autoTestAll.Channels.js @@ -453,6 +453,97 @@ } }; + exports.testGraphChannelStore = function() { + var dir = String(java.nio.file.Files.createTempDirectory("openaf-graph-store-")); + var file = dir + "/graph.db"; + var name = "graph-store-test-" + genUUID(); + var g = ow.ch.utils.graph; + var opts = { store: { type: "mvs", options: { file: file } } }; + try { + $ch(name).create(true, "graph", opts); + $ch(name).set({ id: "doc:a.md" }, { type: "document", props: { title: "A" } }); + $ch(name).set({ id: "doc:b.md" }, { type: "document", props: { title: "B" } }); + $ch(name).set({ id: "concept:x" }, { type: "concept" }); + $ch(name).set({ id: "concept:island" }, { type: "concept" }); + $ch(name).set({ from: "doc:a.md", to: "doc:b.md", type: "LINKS_TO", provenance: "extracted" }, { props: { page: "a.md" } }); + $ch(name).set({ from: "doc:b.md", to: "concept:x", type: "MENTIONS" }, {}); + $ch(name).set({ from: "doc:b.md", to: "concept:x", type: "MENTIONS" }, { props: { w: 2 } }); + ow.test.assert($ch(name).size(), 6, "Problem with graph store size (edge dedupe)"); + ow.test.assert($ch(name).get({ from: "doc:b.md", to: "concept:x", type: "MENTIONS" }).props.w, 2, "Problem updating a graph store edge"); + ow.test.assert($ch(name).get({ id: "doc:b.md" }).props.title, "B", "Problem getting a graph store node"); + + var failed = false; + try { $ch(name).set({ from: "doc:a.md", to: "doc:none", type: "T" }, {}); } catch(e) { failed = true; } + ow.test.assert(failed, true, "Graph store edge to a missing node must fail"); + + ow.test.assert(g.path(name, "a.md", "concept:x"), [ "doc:a.md", "doc:b.md", "concept:x" ], "Problem with graph store path"); + ow.test.assert(g.path(name, "a.md", "concept:island"), [], "Graph store path to a disconnected node must be empty"); + ow.test.assert(g.neighbors(name, "b.md").length, 2, "Problem with graph store neighbors"); + ow.test.assert(g.degree(name, "b.md", { direction: "out" }), 1, "Problem with graph store degree"); + ow.test.assert(g.traverse(name, "a.md", { depth: 2 }).nodes.length, 3, "Problem with graph store traverse"); + ow.test.assert(g.find(name, { type: "concept" }).length, 2, "Problem with graph store find"); + ow.test.assert(g.stats(name).edges, 2, "Problem with graph store stats"); + ow.test.assert(g.stats(name).nodeTypes.concept, 2, "Problem with graph store stats (node types)"); + + // Persistence through the store channel + $ch(name).destroy(); + $ch(name).create(false, "graph", opts); + ow.test.assert($ch(name).size(), 6, "Problem reloading a graph from its store"); + ow.test.assert(g.path(name, "a.md", "concept:x").length, 3, "Problem navigating a reloaded graph store"); + + // Cascade delete + $ch(name).unset({ id: "doc:b.md" }); + ow.test.assert($ch(name).size(), 3, "Unsetting a graph store node must remove its edges"); + ow.test.assert(g.neighbors(name, "concept:x").length, 0, "Unsetting a graph store node must clean the adjacency"); + $ch(name).destroy(); + + // Read-only + missing store + $ch(name).create(false, "graph", { store: opts.store, readOnly: true }); + failed = false; + try { $ch(name).set({ id: "z" }, {}); } catch(e) { failed = true; } + ow.test.assert(failed, true, "Read-only graph store must reject changes"); + ow.test.assert($ch(name).size(), 3, "Read-only graph store must keep its content"); + $ch(name).destroy(); + // Batch writes (single commit on mvs) must behave like sequential sets + var bn = name + "-bulk"; + [ true, false ].forEach(function(bulk) { + var bfile = dir + "/bulk-" + bulk + ".db"; + var bopts = { store: { type: "mvs", bulk: bulk, options: { file: bfile } } }; + $ch(bn).create(true, "graph", bopts); + var ks = [ "id", "from", "to", "type", "provenance" ]; + $ch(bn).setAll(ks, [ + { id: "doc:a", type: "document" }, { id: "doc:b", type: "document" }, { id: "doc:c" }, + { from: "doc:a", to: "doc:b", type: "LINKS_TO", provenance: "EXTRACTED", props: { w: 1 } }, + { from: "doc:a", to: "doc:b", type: "LINKS_TO", provenance: "EXTRACTED", props: { w: 2 } }, + { from: "doc:b", to: "doc:c", type: "LINKS_TO" }, + { from: "doc:c", to: "doc:c", type: "SELF" } + ]); + ow.test.assert($ch(bn).size(), 6, "Problem with graph store batch size (bulk=" + bulk + ")"); + ow.test.assert($ch(bn).get({ from: "doc:a", to: "doc:b", type: "LINKS_TO", provenance: "EXTRACTED" }).props.w, 2, "Batch edge duplicates must keep the last (bulk=" + bulk + ")"); + ow.test.assert(g.degree(bn, "doc:b"), 2, "Problem with graph store batch adjacency (bulk=" + bulk + ")"); + ow.test.assert(g.degree(bn, "doc:c"), 2, "Problem with graph store batch adjacency, self edge (bulk=" + bulk + ")"); + ow.test.assert(g.path(bn, "doc:a", "doc:c"), [ "doc:a", "doc:b", "doc:c" ], "Problem with graph store batch path (bulk=" + bulk + ")"); + var bfailed = false; + try { $ch(bn).setAll([ "from", "to", "type" ], [ { from: "doc:a", to: "doc:zzz", type: "T" } ]); } catch(e) { bfailed = true; } + ow.test.assert(bfailed, true, "Batch edge to a missing node must fail (bulk=" + bulk + ")"); + // a second batch extends the existing adjacency + $ch(bn).setAll([ "id", "from", "to", "type" ], [ { id: "doc:d" }, { from: "doc:a", to: "doc:d", type: "LINKS_TO" } ]); + ow.test.assert(g.degree(bn, "doc:a"), 2, "Second batch must extend the adjacency (bulk=" + bulk + ")"); + $ch(bn).destroy(); + $ch(bn).create(true, "graph", bopts); + ow.test.assert($ch(bn).size(), 8, "Problem reloading a batch written graph store (bulk=" + bulk + ")"); + $ch(bn).destroy(); + }); + + failed = false; + try { $ch(name).create(false, "graph", { create: false, store: { type: "mvs", options: { file: dir + "/none.db" } } }); } catch(e) { failed = true; } + ow.test.assert(failed, true, "Missing graph store with create=false must fail"); + } finally { + $ch(name).destroy(); + io.rm(dir); + } + }; + exports.testAll = function() { $ch("allA").destroy(); $ch("allB").destroy(); diff --git a/tests/autoTestAll.Channels.yaml b/tests/autoTestAll.Channels.yaml index 1b7e7f87..8d19c888 100644 --- a/tests/autoTestAll.Channels.yaml +++ b/tests/autoTestAll.Channels.yaml @@ -127,6 +127,11 @@ jobs: to : oJob Test exec: args.func = args.tests.testGraphChannel; + - name: Channels::Graph channel store + from: Channels::Init + to : oJob Test + exec: args.func = args.tests.testGraphChannelStore; + - name: Channels::Test channel (all) from: Channels::Init to : oJob Test @@ -214,6 +219,7 @@ todo: # Graph - Channels::Graph channel + - Channels::Graph channel store # Ignite #- name: Channels::Creating a channel From 9da43287557abf5f148006ac6bcee5c659bfac93 Mon Sep 17 00:00:00 2001 From: nmaguiar Date: Fri, 16 Oct 2026 07:21:23 +0100 Subject: [PATCH 04/19] feat(channels): add time-series support to owrap.ch Extend the channels wrapper (js/owrap.ch.js) with time-series capabilities. Add a new channel-timeseries.md guide and update index.md, openaf.md and llm-guide.md to reference it. Broaden the Channels test suite (autoTestAll.Channels.js/.yaml) to cover the new behaviour. --- docs/channel-timeseries.md | 224 +++++ docs/index.md | 1 + docs/llm-guide.md | 8 + docs/openaf.md | 13 + js/owrap.ch.js | 1458 ++++++++++++++++++++++++++++--- tests/autoTestAll.Channels.js | 549 ++++++++++++ tests/autoTestAll.Channels.yaml | 64 +- 7 files changed, 2217 insertions(+), 100 deletions(-) create mode 100644 docs/channel-timeseries.md mode change 100755 => 100644 js/owrap.ch.js diff --git a/docs/channel-timeseries.md b/docs/channel-timeseries.md new file mode 100644 index 00000000..47593beb --- /dev/null +++ b/docs/channel-timeseries.md @@ -0,0 +1,224 @@ +# Time series channels: record, retain and query labelled metrics + +[Index](./index.md) | [Core](./openaf.md#ow-ch---channels) | [ODoc](./odoc.md) + +The `timeseries` channel type stores numeric points identified by a metric name, a set of labels and a timestamp (the Prometheus/OpenMetrics model) as a regular OpenAF channel. It keeps data in memory with an optional append-only NDJSON file, or delegates it to other channels (e.g. `mvs`), applies retention by age and/or count, maintains RRD-like rollups, and `ow.ch.utils.timeseries` adds queries (`range`, `latest`, ...), a collector for `ow.metrics`, a channel subscriber and OpenMetrics import/export. Contracts live in [owrap.ch.js](../js/owrap.ch.js); tests are in [autoTestAll.Channels.js](../tests/autoTestAll.Channels.js) (`testTimeSeriesChannel`, `testTimeSeriesChannelStore`, `testTimeSeriesIntegrations`). + +## Create a timeseries channel + +```javascript +$ch("m").create(true, "timeseries", { file: "metrics.ndjson", retention: "7d" }); +``` + +| Option | Type | Default | Description | +| --- | --- | --- | --- | +| `file` | String | | Path to the NDJSON log. Without `file` (and `store`) the channel only lives in memory | +| `readOnly` | Boolean | `false` | Reject changes and never write the file | +| `create` | Boolean | `true` | If `false` the file (or store) must already exist | +| `autosave` | String | `"always"` | `"always"` appends on every change, `"debounced"` appends at most every `saveDebounceMs`, `"off"` only appends on `save`/destroy | +| `saveDebounceMs` | Number | `5000` | Minimum interval between appends when `autosave` is `"debounced"` | +| `lock` | String | | Path to a dummy file used as a filesystem lock while loading and writing | +| `compactRatio` | Number | `0.5` | The file is rewritten when more than this ratio of its lines is dead (overwritten, deleted or expired) | +| `store` | Map | | Keep the data on channels of another type instead of memory (see [Keep the data out of memory](#keep-the-data-out-of-memory)). Can't be combined with `file` | +| `chunk` | String/Number | `"1h"` | Time span of each stored chunk (only with `store`) | +| `retention` | String/Number | | How long to keep raw points (`"7d"`, `"90m"`, or ms). Same as `raw.keep` | +| `maxPoints` | Number | | Maximum raw points per series. Same as `raw.maxPoints` | +| `raw` | Map | | `{ keep, maxPoints }` for the raw points | +| `rollups` | Array | | `[ { step: "1m", keep: "1d" }, { step: "1h", keep: "30d" } ]` (see [Rollups](#rollups)) | +| `retentionRef` | String | `"series"` | `"series"` measures `keep` back from the newest point of each series, `"now"` from the current time | +| `housekeepEveryMs` | Number | `0` | Minimum interval between retention runs triggered by writes (`0` runs on every write) | + +Durations accept numbers (ms) or abbreviations understood by `ow.format.fromTimeAbbreviation` (`500ms`, `30s`, `5m`, `1h`, `7d`, `1h30m`). The channel never starts threads of its own. + +## Keys and values + +| Entry | Key | Value | +| --- | --- | --- | +| Point | `{ metric, labels, ts }` | `{ v }` where `v` is a finite number | + +`labels` is optional and its values are stored as strings; `ts` is epoch ms, a Date or an ISO string (and defaults to now on `set`). A series is a metric plus its labels; a point is a series plus a `ts`. What `get` and `getAll` return is always `{ metric, labels, ts, v }`. + +```javascript +$ch("m").set({ metric: "cpu", labels: { host: "a" }, ts: 1000 }, { v: 0.4 }); +$ch("m").set({ metric: "cpu", labels: { host: "a" }, ts: 2000 }, { v: 0.6 }); +$ch("m").set({ metric: "cpu", labels: { host: "b" }, ts: "2026-01-01T00:00:00Z" }, { v: 0.1 }); +$ch("m").set({ metric: "uptime" }, { v: 3600 }); // ts defaults to now + +$ch("m").get({ metric: "cpu", labels: { host: "a" } }); // latest: { metric: "cpu", labels: { host: "a" }, ts: 2000, v: 0.6 } +$ch("m").get({ metric: "cpu", labels: { host: "a" }, ts: 1000 }); // exact point +$ch("m").size(); // 4 (points) +``` + +Behavior to rely on: + +- Setting the same series and `ts` again overwrites the point; out-of-order points are inserted in order. +- `get` without `ts` returns the latest point of the series. +- `unset` with `ts` removes that point; **without `ts` it drops the whole series** (points and rollups). `ow.ch.utils.timeseries.drop` removes many series by selector. +- `getKeys`, `getAll` and `forEach` go series by series (sorted by metric and labels), points in time order. `getKeys`/`getAll` accept a selector `{ metric, labels, from, to }`. `getSortedKeys` sorts all points by `ts`. +- `shift`/`pop` follow the other channel types: they return **and remove** the oldest/newest point of the channel. +- `setAll(aKeyFields, aValues)` takes flat values `{ metric, labels, ts, v }`; the key fields default to `metric`, `labels` and `ts`. It does one write per series (and one file append, or one `mvs` commit per chunk). +- With `readOnly: true`, changes throw `timeseries channel is read-only`. +- Channel subscribers run asynchronously (use `$ch(...).waitForJobs()` when you need to wait for them). + +## The NDJSON file + +The file is a log: each write appends one line (`{"o":"s",...}` series, `p` point, `r` rollup bucket, `d` delete, `x` drop series) so a write never rewrites the file. Loading replays the log (a truncated last line is ignored). Overwritten, deleted and expired entries become dead lines; when more than `compactRatio` of the lines are dead (checked every 1000 appended lines and on destroy) the file is rewritten with the live data using a temporary file and an atomic move. `ow.ch.utils.timeseries.compact` does it on request. + +Retention is not logged: it is applied again when the file is loaded and after writes. Use only one writer per file (or `lock`, for cooperating processes that reopen the channel between writes); use `readOnly: true` for shared reading. + +## Query with `ow.ch.utils.timeseries` + +Every function takes the channel name first. A **selector** is `{ metric, labels }` where `metric` and each label value can be a String, an Array (any of), a RegExp or a Function; an empty selector matches everything. + +| Function | Returns | Description | +| --- | --- | --- | +| `series(aName, aSelector)` | Array | `{ metric, labels }` of the matching series | +| `labels(aName, aMetric)` | Map | Label names with their sorted values (for a metric, or for all) | +| `latest(aName, aSelector)` | Array | The newest point of each matching series as flat rows | +| `range(aName, aSelector, aOptions)` | Array | Flat rows `{ ts, metric, ...labels, v }` (see below) | +| `drop(aName, aSelector)` | Number | Removes the matching series; returns how many | +| `stats(aName)` | Map | `series`, `points`, per `metrics` (series, points, first, last) and per `tiers` entries | +| `housekeep(aName, aOptions)` | Number | Applies retention now (`{ now }` measures from that time); returns the removed entries | +| `rebuildRollups(aName)` | | Recomputes every rollup from the raw points | +| `compact(aName)` | Boolean | Rewrites the NDJSON file with the live data only | +| `save(aName)` | Boolean | Flushes pending appends now (`false` if read-only) | + +`range` options: + +| Option | Description | +| --- | --- | +| `from`, `to` | Inclusive bounds (epoch ms, Date or ISO string). Default: everything | +| `step` | Bucket size (ms or `"5m"`). Buckets are aligned to the epoch and a row's `ts` is the start of its bucket. Without `step` the raw points are returned | +| `fn` | `avg` (default), `min`, `max`, `sum`, `count`, `first`, `last`, `rate` (per second) or `increase`. `rate` and `increase` treat a decrease of the value as a counter reset | +| `fill` | For buckets without data: `none` (default), `null`, `zero` or `prev` | +| `tier` | Force the raw tier (`"raw"`) or a rollup (its step, e.g. `"1h"`) | + +```javascript +var ts = ow.ch.utils.timeseries; + +ts.range("m", { metric: "cpu", labels: { host: /^web/ } }, { from: "2026-01-01", step: "5m", fn: "max" }); +// [ { ts: 1767225600000, metric: "cpu", host: "web1", v: 0.93 }, ... ] + +ts.range("m", { metric: "http_requests_total" }, { step: "1m", fn: "rate" }); // requests per second +ts.latest("m", { metric: "cpu" }); // one row per host +ts.labels("m", "cpu"); // { host: [ "a", "b" ] } +``` + +The rows are flat, so they go straight into `oafp`: + +```bash +openaf -c 'ow.loadCh(); $ch("m").create(true,"timeseries",{file:"cpu.ndjson"}); print(stringify(ow.ch.utils.timeseries.range("m",{metric:"cpu"},{step:"2m"}),__,""))' | oafp out=ctable +``` + +Label names that collide with `ts`, `metric` or `v` are prefixed with `_` in the rows. + +## Retention + +- `retention` / `raw.keep` drops raw points older than `keep` before the newest point of the same series (`retentionRef: "series"`, the default; it is deterministic and never empties a series that stopped reporting) or before the current time (`retentionRef: "now"`). +- `maxPoints` keeps the newest N raw points of each series. +- Both run on writes (every write by default, or every `housekeepEveryMs`), when the channel is created, and on demand with `ow.ch.utils.timeseries.housekeep(aName)`. Use `housekeep` (or a scheduled job) to trim series that stopped receiving writes with `retentionRef: "now"`. +- Points removed by retention do not notify channel subscribers. +- A point written older than the retention is accepted and removed right away. + +## Rollups + +Rollups keep aggregated buckets after the raw points are gone (the RRD idea), so long ranges stay cheap and history outlives the raw retention. + +```javascript +$ch("m").create(true, "timeseries", { + file: "metrics.ndjson", + raw : { keep: "6h" }, // raw points for 6 hours + rollups: [ { step: "1m", keep: "1d" }, // 1 minute buckets for a day + { step: "1h", keep: "30d" } ] // 1 hour buckets for 30 days +}); +``` + +Each bucket stores `count`, `sum`, `min`, `max`, `first` and `last` (so `avg`, `min`, `max`, `sum`, `count`, `first`, `last` and `increase` are all answered from it, and a coarser `step` can merge finer buckets exactly when it is a multiple of the bucket size). Buckets are updated on every write. + +- `range` with a `step` uses the finest tier whose buckets divide `step` and that still reaches back to `from` (raw points while they exist). Use `tier` to force one. +- A late point is always stored in the raw tier, and updates every rollup whose bucket has not expired. +- Overwriting or removing a point rebuilds its buckets from the raw points when those are complete. When raw data has been trimmed, the bucket is only adjusted (`count`, `sum`, `first`, `last` follow, but `min` and `max` can't narrow). +- On rollup tiers `rate`/`increase` are approximate: a reset inside a bucket is only detected when its `last` is below its `first`. +- Rollups only exist for the writes made after they were configured; `rebuildRollups` recomputes them from the raw points that are still there. + +## Keep the data out of memory + +By default everything is in memory (and optionally in the NDJSON file). With `store` the data is kept in other OpenAF channels (a `series` index and one chunk channel per tier, named `timeseries::::`) so memory is bounded by that channel's own caching. Points are stored as sorted chunks (key `{ s, b }` with the chunk's start time), so a `range` only reads the chunks it overlaps, and retention drops whole chunks. + +```javascript +$ch("m").create(true, "timeseries", { store: { type: "mvs", options: { file: "metrics.db", compact: true } }, chunk: "1h", retention: "30d" }); +``` + +| `store` field | Type | Description | +| --- | --- | --- | +| `type` | String | Channel type used for every part (e.g. `mvs`) | +| `options` | Map or Function | Options for the underlying channels. A Map gets a distinct `map` name per part (so one `mvs` file holds all); a Function receives the part name (`series`, `raw`, `r`) and returns its options | +| `compress` | Boolean | Passed as `shouldCompress` when creating the underlying channels | +| `bulk` | Boolean | Use the underlying channel's `setAll` to write several chunks at once (defaults to `true` for `mvs`) | + +With a store, `file`, `autosave`, `saveDebounceMs`, `lock` and `compactRatio` don't apply: every change goes straight to the store. `create: false` throws if the store is empty. Destroying the timeseries channel destroys the underlying ones. + +Measured with `mvs` (100,000 points over 20 series, 1 s spacing; indicative numbers on a laptop): + +| | Memory | NDJSON file | `mvs` store | +| --- | --- | --- | --- | +| Load 100,000 points in one `setAll` | ~1.1 s | ~1.0 s | ~1.1 s (1.7 s with two rollups) | +| 2,000 points one `set` at a time | ~0.1 s | ~0.2 s | ~2.5 s (4.7 s with two rollups) | +| Reopen with 100,000 points | | ~0.3 s | no replay | +| `range` of one series, all points | ~10 ms | ~10 ms | ~15 ms | +| `latest` over 20 series | ~2 ms | ~2 ms | ~30 ms | + +Use a store when the data is large relative to the heap or you can't afford to replay a log at start-up; write in batches (`setAll`) because every `mvs` write is a commit. Notes: `getKeys`, `getAll`, `stats` and `size` on a store walk the series index or all chunks of the selected series; set `compact: true` in the `mvs` options so the file is compacted on destroy. + +## Record data + +### From `ow.metrics` + +`collect(aEvery, aName, aSource, aOptions)` records a source periodically, in the style of `ow.ch.utils.keepHistory`: `aEvery` is a number of ms or a cron expression; `aSource` is omitted (all of `ow.metrics.getAll()`), an Array of metric names, or a function returning a map or an array of `{ metric, labels, v, ts }`. Nested maps extend the metric name with `_` (leading underscores of a key are dropped), array items get an `idx` label, booleans become 0/1 and non numeric values are skipped. Options: `labels` (added to every point), `prefix`, `onError`. It returns `{ once, stop }`; **call `stop()` before the process ends or the channel is destroyed** (the collector also stops itself if the channel no longer exists). + +```javascript +$ch("m").create(true, "timeseries", { file: "oaf.ndjson", retention: "2d", retentionRef: "now" }); +var col = ow.ch.utils.timeseries.collect(5000, "m", __, { prefix: "oaf_", labels: { app: "demo" } }); +// ... +col.stop(); +``` + +### From another channel + +`getTimeSeriesSubscriber(aTargetCh, aOptions)` returns a subscriber that records the `set`/`setall` of another channel as points. Options: `metric` (String or `function(key, value)`), `labels` (Array of field names, or a function), `value` (field name or function; defaults to `v`) **or** `values` (Array of field names, one metric `_` each), `ts` (field name or function; defaults to now) and `ops`. + +```javascript +$ch("events").subscribe(ow.ch.utils.timeseries.getTimeSeriesSubscriber("m", { metric: "temp", labels: [ "room" ], value: "t", ts: "when" })); +$ch("events").set({ id: 1 }, { id: 1, room: "kitchen", t: 21.5, when: 1000 }); +``` + +### OpenMetrics (Prometheus text) + +`toOpenMetrics(aName, aSelector, aOptions)` returns the latest point of each series (or every point between `from` and `to`) as text: `name{label="value"} value timestamp`. Metric and label names are sanitized, label values escaped. Options: `help` (`{ metric: { help, type } }` for `# HELP`/`# TYPE` lines), `timestamps: false`, `tsUnit` (`"s"` default, `"ms"`). `fromOpenMetrics(aName, aText, aOptions)` records such text as points (options: `labels` added to all, `ts` default timestamp, `tsUnit`) and returns how many. + +```javascript +var txt = ow.ch.utils.timeseries.toOpenMetrics("m", { metric: /^cpu/ }); +// cpu{host="a"} 0.6 2 +// POST txt to a Prometheus push gateway: +$rest().post("http://pushgw:9091/metrics/job/demo", txt); +$ch("copy").create(true, "timeseries", {}); +ow.ch.utils.timeseries.fromOpenMetrics("copy", txt); +``` + +## More use cases + +- **Host and JVM history**: `collect` with `retentionRef: "now"` and a rollup per hour gives a cheap local history of `ow.metrics` without an external system. +- **Job durations and SLOs**: record one point per run (`{ metric: "job_seconds", labels: { job: "etl" } }`) and ask `range(..., { step: "1d", fn: "max" })`. +- **Counters**: store the raw counter and use `rate`/`increase`, resets are handled. +- **Sensor and IoT feeds**: a channel subscriber on the receiving channel keeps a bounded history with `maxPoints`. +- **Export**: `range` rows feed `oafp` (`out=ctable`, `out=csv`, `out=md`...); `toOpenMetrics` feeds Prometheus tooling. + +## Limits + +- One writer per NDJSON file; the whole file content is replayed in memory at load. +- Values are numbers only; for several fields record several metrics or add a `field` label. +- Retention deletes made by the channel do not notify subscribers. + +## Source documentation + +The same contracts are documented in the source as Markdown ODoc entries (`ow.ch.types.timeseries` and `ow.ch.utils.timeseries`), available with `help ow.ch.types.timeseries` in the console or `openaf -helpscript ow.ch.types.timeseries`. See [odoc.md](./odoc.md#opt-in-to-markdown) for the format. diff --git a/docs/index.md b/docs/index.md index 36462681..53bcaf29 100644 --- a/docs/index.md +++ b/docs/index.md @@ -17,6 +17,7 @@ Core references and extended guides to build and operate oJobs and OpenAF soluti - [openaf.md](./openaf.md) – Core OpenAF runtime helpers & APIs (`$$`, `_$`, `$from`, `$path`, channels, etc.) - [openaf-dollar-functions.md](./openaf-dollar-functions.md) – Additional `$` helpers: formatting, caching, synchronization, argument conversion, JSON-RPC, processes, and output - [channel-graph.md](./channel-graph.md) – `graph` channel type: operate and navigate a Mini-A knowledge graph (`graph.json`) with `ow.ch.utils.graph` +- [channel-timeseries.md](./channel-timeseries.md) – `timeseries` channel type: labelled numeric series with retention, rollups, `range` queries, `ow.metrics` collector and OpenMetrics import/export (`ow.ch.utils.timeseries`) - [ojob.md](./ojob.md) – Main oJob YAML reference (structure, jobs, built-ins, shortcuts, argument tables) - [ojob-all.yaml](./ojob-all.yaml) – Annotated syntax catalog (illustrative options, not a runnable definition) - [ojob-examples.md](./ojob-examples.md) – Patterns and source examples from the ojob.io YAML collection diff --git a/docs/llm-guide.md b/docs/llm-guide.md index 129b1bd0..a496c0a5 100644 --- a/docs/llm-guide.md +++ b/docs/llm-guide.md @@ -395,7 +395,13 @@ ow.ch.new("myChannel", "mem"); // In-memory ow.ch.new("persistent", "mvs", { // Persistent file: "data.db" }); +ow.ch.new("sharded", "mvs", { // Persistent, keys hashed across 4 files + multifile: true, path: "data", files: 4 +}); ow.ch.new("distributed", "big"); // Peer-to-peer +ow.ch.new("metrics", "timeseries", { // Labelled numeric series (see docs/channel-timeseries.md) + file: "metrics.ndjson", retention: "7d" +}); // Operations ow.ch.set("myChannel", "key1", {data: "value"}); @@ -404,6 +410,8 @@ var all = ow.ch.getAll("myChannel"); ow.ch.destroy("myChannel"); ``` +Time series: `$ch("m").set({ metric: "cpu", labels: { host: "a" }, ts: 1000 }, { v: 0.4 })` records a point (the key is metric + labels + `ts`, the value a number `v`), and `ow.ch.utils.timeseries.range("m", { metric: "cpu" }, { step: "5m", fn: "avg" })` returns flat rows `{ ts, metric, ...labels, v }`. `unset` without `ts` drops the series; `shift`/`pop` remove points. See [channel-timeseries.md](./channel-timeseries.md). + #### ow.ai - Artificial Intelligence ```javascript diff --git a/docs/openaf.md b/docs/openaf.md index 855fd5a4..30087a2f 100644 --- a/docs/openaf.md +++ b/docs/openaf.md @@ -347,6 +347,19 @@ print(ow.ch.utils.graph.path("g", "a.md", "b.md")); // ["doc:a.md", "doc:b.md"] See [channel-graph.md](./channel-graph.md) for options, key formats and limits. +#### Time Series Channels + +The `timeseries` channel type stores numeric points identified by a metric, labels and a timestamp, in memory with an optional append-only NDJSON file or on top of another channel type (`store`). It supports retention (age and count) and stored rollups, and `ow.ch.utils.timeseries` provides `range`, `latest`, `series`, `labels`, `stats`, a collector for `ow.metrics`, a channel subscriber and OpenMetrics import/export. + +```javascript +$ch("m").create(true, "timeseries", { file: "metrics.ndjson", retention: "7d", rollups: [ { step: "1h" } ] }); +$ch("m").set({ metric: "cpu", labels: { host: "a" }, ts: 1000 }, { v: 0.4 }); +$ch("m").set({ metric: "cpu", labels: { host: "a" }, ts: 2000 }, { v: 0.6 }); +sprint(ow.ch.utils.timeseries.range("m", { metric: "cpu" }, { step: "1s", fn: "avg" })); // [ { ts: 1000, metric: "cpu", host: "a", v: 0.4 }, { ts: 2000, metric: "cpu", host: "a", v: 0.6 } ] +``` + +See [channel-timeseries.md](./channel-timeseries.md) for options, retention, rollups, the `store` mode and the integrations. + --- ## nLinq - Query Processing diff --git a/js/owrap.ch.js b/js/owrap.ch.js old mode 100755 new mode 100644 index 6d0f7ed5..35e06311 --- a/js/owrap.ch.js +++ b/js/owrap.ch.js @@ -2289,6 +2289,779 @@ OpenWrap.ch.prototype.__types = { this.__changed(c) } }, + // Time series implementation + // + /** + * + * ow.ch.types.timeseries + * + * + */ + timeseries: { + __channels: Object.create(null), + __c: function(aName) { + var c = this.__channels[aName] + if (isUnDef(c)) throw "timeseries channel '" + aName + "' not found" + return c + }, + __l: (m) => (isString(m.lock) ? $flock(m.lock).lock() : __), + __ul: (m) => (isString(m.lock) ? $flock(m.lock).unlock() : __), + __rw: function(c) { + if (c.opts.readOnly) throw "timeseries channel is read-only" + }, + // ---- value helpers + __ts: function(t) { + if (isUnDef(t) || t === null) return now() + var r + if (isNumber(t)) r = t + else if (isDate(t)) r = t.getTime() + else if (isString(t)) r = /^\s*-?\d+(\.\d+)?\s*$/.test(t) ? Number(t) : (new Date(t)).getTime() + if (!isNumber(r) || isNaN(r) || !isFinite(r)) throw "timeseries: invalid timestamp '" + t + "'" + return Math.floor(r) + }, + __dur: function(d, what) { + if (isUnDef(d) || d === null) return __ + var r = isString(d) ? ow.loadFormat().fromTimeAbbreviation(d) : d + if (!isNumber(r) || isNaN(r) || !(r > 0)) throw "timeseries: invalid duration for " + what + " (" + d + ")" + return r + }, + __val: function(v) { + if (isMap(v)) v = v.v + if (isString(v) && v.trim().length > 0) v = Number(v) + if (!isNumber(v) || isNaN(v) || !isFinite(v)) throw "timeseries value 'v' must be a finite number" + return v + }, + __labels: function(l) { + if (isUnDef(l) || l === null) return {} + if (!isMap(l)) throw "timeseries labels must be a map" + var r = {} + Object.keys(l).sort().forEach(k => { if (isDef(l[k]) && l[k] !== null) r[k] = String(l[k]) }) + return r + }, + __sid: function(metric, labels) { + return stringify([ metric, labels ], __, "") + }, + // validates a key; ts is only normalized when present + __key: function(k) { + if (!isMap(k) || !isString(k.metric) || k.metric.length == 0) throw "timeseries key must have a 'metric' and optionally 'labels' and 'ts'" + var labels = this.__labels(k.labels) + return { id: this.__sid(k.metric, labels), metric: k.metric, labels: labels, ts: isDef(k.ts) ? this.__ts(k.ts) : __ } + }, + __pt: function(m, t, v) { + return { metric: m.metric, labels: clone(m.labels), ts: t, v: v } + }, + // selector matching: String/Number, Array (any), RegExp or Function + __m: function(want, have) { + if (isUnDef(want)) return true + if (isArray(want)) return want.some(w => this.__m(w, have)) + if (Object.prototype.toString.call(want) == "[object RegExp]") return isDef(have) && want.test(String(have)) + if (isFunction(want)) return want(have) === true + return isDef(have) && String(want) === String(have) + }, + __match: function(m, sel) { + if (!this.__m(sel.metric, m.metric)) return false + if (isMap(sel.labels)) { + for (var k in sel.labels) if (!this.__m(sel.labels[k], m.labels[k])) return false + } + return true + }, + __select: function(c, sel) { + var self = this + sel = isMap(sel) ? sel : {} + return c.st.ids().filter(id => self.__match(c.st.meta(id), sel)).sort() + }, + // ---- sorted array helpers over { ts: [], v: [] } + __lb: function(a, x) { var lo = 0, hi = a.length; while (lo < hi) { var m = (lo + hi) >>> 1; if (a[m] < x) lo = m + 1; else hi = m } return lo }, + __ub: function(a, x) { var lo = 0, hi = a.length; while (lo < hi) { var m = (lo + hi) >>> 1; if (a[m] <= x) lo = m + 1; else hi = m } return lo }, + // upserts into o; returns how many entries were added (not replaced) + __put: function(o, ts, v) { + var added = 0 + for (var i = 0; i < ts.length; i++) { + var t = ts[i], n = o.ts.length + if (n == 0 || t > o.ts[n - 1]) { + o.ts.push(t); o.v.push(v[i]); added++ + } else { + var p = this.__lb(o.ts, t) + if (o.ts[p] === t) o.v[p] = v[i] + else { o.ts.splice(p, 0, t); o.v.splice(p, 0, v[i]); added++ } + } + } + return added + }, + __rm: function(o, ts) { + var removed = 0 + for (var i = 0; i < ts.length; i++) { + var p = this.__lb(o.ts, ts[i]) + if (o.ts[p] === ts[i]) { o.ts.splice(p, 1); o.v.splice(p, 1); removed++ } + } + return removed + }, + // ---- aggregates (rollup buckets) + __a0: function(t, v) { return { count: 1, sum: v, min: v, max: v, first: v, last: v, ft: t, lt: t } }, + __am: function(a, b) { + if (isUnDef(a)) return b + if (isUnDef(b)) return a + return { + count: a.count + b.count, sum: a.sum + b.sum, min: Math.min(a.min, b.min), max: Math.max(a.max, b.max), + first: b.ft < a.ft ? b.first : a.first, ft: Math.min(a.ft, b.ft), + last : b.lt >= a.lt ? b.last : a.last, lt: Math.max(a.lt, b.lt) + } + }, + __ae: function(a) { return [ a.count, a.sum, a.min, a.max, a.first, a.last, a.ft, a.lt ] }, + __ad: function(a) { return { count: a[0], sum: a[1], min: a[2], max: a[3], first: a[4], last: a[5], ft: a[6], lt: a[7] } }, + // ---- storage backends: every access goes through c.st + // memory + __memStore: function(c) { + var self = this, S = {} + var T = (id, k) => { var s = S[id]; if (isUnDef(s.t[k])) s.t[k] = { ts: [], v: [] }; return s.t[k] } + return { + bulk : false, + ids : () => Object.keys(S), + has : id => isDef(S[id]), + meta : id => isDef(S[id]) ? S[id].meta : __, + putMeta : m => { if (isUnDef(S[m.id])) S[m.id] = { meta: m, t: {} } }, + read : (id, k, from, to) => { + var o = T(id, k), a = self.__lb(o.ts, from), b = self.__ub(o.ts, to) + return { ts: o.ts.slice(a, b), v: o.v.slice(a, b) } + }, + upsert : (id, k, ts, v) => self.__put(T(id, k), ts, v), + del : (id, k, ts) => self.__rm(T(id, k), ts), + trimBefore: (id, k, cut) => { + var o = T(id, k), p = self.__lb(o.ts, cut) + if (p > 0) { o.ts.splice(0, p); o.v.splice(0, p) } + return p + }, + trimCount : (id, k, max) => { + var o = T(id, k), n = o.ts.length - max + if (n > 0) { o.ts.splice(0, n); o.v.splice(0, n) } + return Math.max(n, 0) + }, + bounds : (id, k) => { var o = T(id, k), n = o.ts.length; return n == 0 ? __ : { first: o.ts[0], last: o.ts[n - 1], count: n } }, + drop : id => { delete S[id] }, + rawCount : () => Object.keys(S).reduce((a, id) => a + (isDef(S[id].t.raw) ? S[id].t.raw.ts.length : 0), 0), + liveCount : () => Object.keys(S).reduce((a, id) => a + 1 + Object.keys(S[id].t).reduce((b, k) => b + S[id].t[k].ts.length, 0), 0), + close : () => {} + } + }, + // channel: series index + one chunk channel per tier, delegated to other channels (e.g. mvs) + __chStore: function(c, aName) { + var self = this, spec = c.opts.store, pre = "timeseries::" + aName + "::", n = {}, made = [] + var close = () => made.forEach(x => { try { $ch(x).destroy() } catch(e) {} }) + try { + ;[ "series" ].concat(c.tiers.map(t => t.key)).forEach(p => { + n[p] = pre + p + var o = isFunction(spec.options) ? spec.options(p) : merge(isMap(spec.options) ? clone(spec.options) : {}, { map: pre + p }) + $ch(n[p]).create(spec.compress === true, spec.type, o) + made.push(n[p]) + }) + } catch(e) { close(); throw e } + var bulk = spec.bulk === true || (spec.bulk !== false && spec.type == "mvs") + var cs = k => c.tierBy[k].chunk + var bk = (k, t) => Math.floor(t / cs(k)) * cs(k) + var rec = id => $ch(n.series).get({ s: id }) + var tinfo = (r, k) => { if (isUnDef(r.t[k])) r.t[k] = { first: __, last: __, count: 0, chunks: [] }; return r.t[k] } + var chunk = (id, k, b) => $ch(n[k]).get({ s: id, b: b }) + var wr = (k, list) => { + if (list.length == 0) return + if (bulk && list.length > 1) $ch(n[k]).setAll([ "s", "b" ], list) + else list.forEach(x => $ch(n[k]).set({ s: x.s, b: x.b }, x)) + } + var refresh = (id, r, k) => { + var info = tinfo(r, k) + info.chunks.sort((a, b) => a - b) + if (info.chunks.length == 0 || info.count <= 0) { info.first = __; info.last = __; info.count = 0; info.chunks = []; return } + var f = chunk(id, k, info.chunks[0]), l = info.chunks.length == 1 ? f : chunk(id, k, info.chunks[info.chunks.length - 1]) + info.first = f.ts[0] + info.last = l.ts[l.ts.length - 1] + } + var save = r => $ch(n.series).set({ s: r.s }, r) + // groups ts (+ optional v) by chunk start + var group = (k, ts, v) => { + var g = {}, order = [] + ts.forEach((t, i) => { + var b = bk(k, t) + if (isUnDef(g[b])) { g[b] = { ts: [], v: [] }; order.push(b) } + g[b].ts.push(t); if (isDef(v)) g[b].v.push(v[i]) + }) + return { g: g, order: order } + } + return { + names : n, + bulk : bulk, + ids : () => $ch(n.series).getKeys().map(k => k.s), + has : id => isMap(rec(id)), + meta : id => { var r = rec(id); return isMap(r) ? { id: id, metric: r.metric, labels: r.labels } : __ }, + putMeta : m => { if (!isMap(rec(m.id))) save({ s: m.id, metric: m.metric, labels: m.labels, t: {} }) }, + read : (id, k, from, to) => { + var r = rec(id), res = { ts: [], v: [] } + if (!isMap(r) || isUnDef(r.t[k])) return res + r.t[k].chunks.forEach(b => { + if (b > to || b + cs(k) - 1 < from) return + var ch = chunk(id, k, b) + if (!isMap(ch)) return + var a = self.__lb(ch.ts, from), z = self.__ub(ch.ts, to) + res.ts = res.ts.concat(ch.ts.slice(a, z)); res.v = res.v.concat(ch.v.slice(a, z)) + }) + return res + }, + upsert : (id, k, ts, v) => { + var r = rec(id), info = tinfo(r, k), gr = group(k, ts, v), added = 0, out = [] + gr.order.forEach(b => { + var ch = chunk(id, k, b) + if (!isMap(ch)) { ch = { s: id, b: b, ts: [], v: [] }; if (info.chunks.indexOf(b) < 0) info.chunks.push(b) } + added += self.__put(ch, gr.g[b].ts, gr.g[b].v) + out.push(ch) + }) + wr(k, out) + info.count += added + refresh(id, r, k) + save(r) + return added + }, + del : (id, k, ts) => { + var r = rec(id) + if (!isMap(r) || isUnDef(r.t[k])) return 0 + var info = r.t[k], gr = group(k, ts), removed = 0, out = [] + gr.order.forEach(b => { + var ch = chunk(id, k, b) + if (!isMap(ch)) return + removed += self.__rm(ch, gr.g[b].ts) + if (ch.ts.length == 0) { $ch(n[k]).unset({ s: id, b: b }); info.chunks = info.chunks.filter(x => x !== b) } else out.push(ch) + }) + wr(k, out) + info.count -= removed + refresh(id, r, k) + save(r) + return removed + }, + trimBefore: (id, k, cut) => { + var r = rec(id) + if (!isMap(r) || isUnDef(r.t[k]) || isUnDef(r.t[k].first) || r.t[k].first >= cut) return 0 + var info = r.t[k], removed = 0, keep = [] + info.chunks.sort((a, b) => a - b).forEach(b => { + if (b > cut) { keep.push(b); return } + var ch = chunk(id, k, b) + if (!isMap(ch)) return + var p = self.__lb(ch.ts, cut) + removed += p + if (p >= ch.ts.length) { $ch(n[k]).unset({ s: id, b: b }); return } + if (p > 0) { ch.ts.splice(0, p); ch.v.splice(0, p); $ch(n[k]).set({ s: id, b: b }, ch) } + keep.push(b) + }) + info.chunks = keep + info.count -= removed + refresh(id, r, k) + save(r) + return removed + }, + trimCount : (id, k, max) => { + var r = rec(id) + if (!isMap(r) || isUnDef(r.t[k]) || r.t[k].count <= max) return 0 + var info = r.t[k], left = info.count - max, removed = 0, keep = [] + info.chunks.sort((a, b) => a - b).forEach(b => { + if (left <= 0) { keep.push(b); return } + var ch = chunk(id, k, b) + if (!isMap(ch)) return + var p = Math.min(left, ch.ts.length) + left -= p; removed += p + if (p >= ch.ts.length) { $ch(n[k]).unset({ s: id, b: b }); return } + ch.ts.splice(0, p); ch.v.splice(0, p); $ch(n[k]).set({ s: id, b: b }, ch) + keep.push(b) + }) + info.chunks = keep + info.count -= removed + refresh(id, r, k) + save(r) + return removed + }, + bounds : (id, k) => { + var r = rec(id) + if (!isMap(r) || isUnDef(r.t[k]) || r.t[k].count <= 0) return __ + return { first: r.t[k].first, last: r.t[k].last, count: r.t[k].count } + }, + drop : id => { + var r = rec(id) + if (!isMap(r)) return + Object.keys(r.t).forEach(k => r.t[k].chunks.forEach(b => $ch(n[k]).unset({ s: id, b: b }))) + $ch(n.series).unset({ s: id }) + }, + rawCount : () => $ch(n.series).getAll().reduce((a, r) => a + (isDef(r.t.raw) ? r.t.raw.count : 0), 0), + liveCount : () => 0, + close : close + } + }, + // ---- engine + // reference timestamp used by retention for a series + __ref: function(c, id, aNow) { + if (isDef(aNow)) return aNow + if (c.opts.retentionRef == "now") return now() + var b = c.st.bounds(id, "raw") + return isDef(b) ? b.last : __ + }, + __retain: function(c, id, aNow) { + var ref = this.__ref(c, id, aNow), removed = 0 + if (isUnDef(ref)) return 0 + c.tiers.forEach(t => { + if (isDef(t.keep)) removed += c.st.trimBefore(id, t.key, ref - t.keep - (t.step > 0 ? t.step - 1 : 0)) + if (isDef(t.maxPoints)) removed += c.st.trimCount(id, t.key, t.maxPoints) + }) + return removed + }, + __hk: function(c, id) { + if (!c.hasRetention) return + var ids = [ id ] + if (c.opts.housekeepEveryMs > 0) { + c.touched[id] = true + if (now() - c.lastHK < c.opts.housekeepEveryMs) return + ids = Object.keys(c.touched); c.touched = {}; c.lastHK = now() + } + for (var i = 0; i < ids.length; i++) if (c.st.has(ids[i])) this.__retain(c, ids[i]) + }, + __lg: function(c, o) { + if (isDef(c.buf)) c.buf.push(stringify(o, __, "")) + }, + // is the raw data of the bucket starting at b intact (no raw trimming configured or nothing older than b was lost)? + __intact: function(c, id, b) { + if (!c.rawTrims) return true + var bd = c.st.bounds(id, "raw") + return isDef(bd) && bd.first <= b + }, + __rebuild: function(c, id, t, b) { + var r = c.st.read(id, "raw", b, b + t.step - 1), agg + for (var i = 0; i < r.ts.length; i++) agg = this.__am(agg, this.__a0(r.ts[i], r.v[i])) + return agg + }, + // updates the rollup tiers after raw points ts/vs were written (old holds the previous values of overwritten points) + __roll: function(c, id, ts, vs, old) { + var self = this + c.tiers.forEach(t => { + if (t.step <= 0) return + var ref = self.__ref(c, id), cut = (isDef(t.keep) && isDef(ref)) ? ref - t.keep : __ + var bl = {}, order = [] + ts.forEach((x, i) => { + var b = Math.floor(x / t.step) * t.step + if (isDef(cut) && b + t.step <= cut) return + if (isUnDef(bl[b])) { bl[b] = []; order.push(b) } + bl[b].push(i) + }) + var ub = [], uv = [] + order.forEach(b => { + var cur = c.st.read(id, t.key, b, b), agg = cur.ts.length > 0 ? cur.v[0] : __ + var over = bl[b].some(i => isDef(old[ts[i]])) + if (over && self.__intact(c, id, b)) { + agg = self.__rebuild(c, id, t, b) + } else { + bl[b].forEach(i => { + var x = ts[i], v = vs[i] + if (isDef(old[x]) && isDef(agg)) { + agg = { count: agg.count, sum: agg.sum - old[x] + v, min: Math.min(agg.min, v), max: Math.max(agg.max, v), + first: agg.ft === x ? v : agg.first, ft: agg.ft, last: agg.lt === x ? v : agg.last, lt: agg.lt } + } else { + agg = self.__am(agg, self.__a0(x, v)) + } + }) + } + if (isDef(agg)) { ub.push(b); uv.push(agg); self.__lg(c, { o: "r", s: id, k: t.key, t: b, a: self.__ae(agg) }) } + }) + if (ub.length > 0) c.st.upsert(id, t.key, ub, uv) + }) + }, + // fixes rollup buckets after raw points were removed + __fix: function(c, id, ts) { + var self = this + c.tiers.forEach(t => { + if (t.step <= 0) return + var seen = {} + ts.forEach(x => { + var b = Math.floor(x / t.step) * t.step + if (seen[b] || !self.__intact(c, id, b)) return + seen[b] = true + var agg = self.__rebuild(c, id, t, b) + if (isDef(agg)) { + c.st.upsert(id, t.key, [ b ], [ agg ]); self.__lg(c, { o: "r", s: id, k: t.key, t: b, a: self.__ae(agg) }) + } else { + c.st.del(id, t.key, [ b ]); self.__lg(c, { o: "d", s: id, k: t.key, t: b }) + } + }) + }) + }, + // writes points [ { t, v } ] of one series + __write: function(c, m, pts) { + var st = c.st, id = m.id + if (!st.has(id)) { + st.putMeta({ id: id, metric: m.metric, labels: m.labels }) + this.__lg(c, { o: "s", s: id, m: m.metric, l: m.labels }) + } + var map = {} + pts.forEach(p => { map[p.t] = p.v }) + var ts = Object.keys(map).map(Number).sort((a, b) => a - b), vs = ts.map(t => map[t]) + if (ts.length == 0) return + var old = {} + if (c.tiers.length > 1) { + var ex = st.read(id, "raw", ts[0], ts[ts.length - 1]) + ex.ts.forEach((t, i) => { old[t] = ex.v[i] }) + } + st.upsert(id, "raw", ts, vs) + for (var i = 0; i < ts.length; i++) this.__lg(c, { o: "p", s: id, t: ts[i], v: vs[i] }) + if (c.tiers.length > 1) this.__roll(c, id, ts, vs, old) + this.__hk(c, id) + }, + __changed: function(c) { + if (isUnDef(c.opts.file)) return + c.dirty = true + if (c.opts.autosave === "always" || (c.opts.autosave === "debounced" && now() - c.lastSave >= c.opts.saveDebounceMs)) this.__flush(c) + }, + __flush: function(c) { + var o = c.opts + if (o.readOnly) return false + if (isUnDef(o.file)) return true + if (c.buf.length > 0) { + this.__l(o) + try { + var dir = new java.io.File(o.file).getAbsoluteFile().getParentFile() + if (isDef(dir) && !dir.exists()) dir.mkdirs() + io.writeFileString(o.file, (c.needNL ? "\n" : "") + c.buf.join("\n") + "\n", __, true) + c.needNL = false + c.lines += c.buf.length + c.buf = [] + } finally { + this.__ul(o) + } + } + c.dirty = false + c.lastSave = now() + if (c.lines - c.lastCompact >= 1000) this.__compactIf(c) + return true + }, + __compactIf: function(c) { + var live = c.st.liveCount() + if (c.lines > 0 && (c.lines - live) / c.lines > c.opts.compactRatio) this.__compact(c); else c.lastCompact = c.lines + }, + // rewrites the file with the live state only + __compact: function(c) { + var o = c.opts, self = this + if (o.readOnly || isUnDef(o.file)) return false + var lines = [] + c.st.ids().sort().forEach(id => { + var m = c.st.meta(id) + lines.push(stringify({ o: "s", s: id, m: m.metric, l: m.labels }, __, "")) + c.tiers.forEach(t => { + var r = c.st.read(id, t.key, -Infinity, Infinity) + for (var i = 0; i < r.ts.length; i++) + lines.push(t.step > 0 ? stringify({ o: "r", s: id, k: t.key, t: r.ts[i], a: self.__ae(r.v[i]) }, __, "") : stringify({ o: "p", s: id, t: r.ts[i], v: r.v[i] }, __, "")) + }) + }) + this.__l(o) + try { + var dir = new java.io.File(o.file).getAbsoluteFile().getParentFile() + if (isDef(dir) && !dir.exists()) dir.mkdirs() + var tmp = o.file + "." + java.util.UUID.randomUUID() + ".tmp" + try { + io.writeFileString(tmp, lines.length > 0 ? lines.join("\n") + "\n" : "") + java.nio.file.Files.move(new java.io.File(tmp).toPath(), new java.io.File(o.file).toPath(), java.nio.file.StandardCopyOption.ATOMIC_MOVE, java.nio.file.StandardCopyOption.REPLACE_EXISTING) + } finally { + java.nio.file.Files.deleteIfExists(new java.io.File(tmp).toPath()) + } + } finally { + this.__ul(o) + } + c.buf = [] + c.needNL = false + c.lines = lines.length + c.lastCompact = c.lines + c.dirty = false + c.lastSave = now() + return true + }, + __load: function(c) { + var o = c.opts, st = c.st + this.__l(o) + try { + if (!io.fileExists(o.file)) { + if (o.create === false) throw "timeseries file '" + o.file + "' not found" + return + } + var txt = io.readFileString(o.file) + c.needNL = txt.length > 0 && txt.charAt(txt.length - 1) != "\n" + var self = this + txt.split("\n").forEach(line => { + if (line.length == 0) return + c.lines++ + var r + try { r = JSON.parse(line) } catch(e) { c.bad++; return } + if (!isMap(r)) { c.bad++; return } + var k = isString(r.k) ? r.k : "raw" + switch(r.o) { + case "s": st.putMeta({ id: r.s, metric: r.m, labels: isMap(r.l) ? r.l : {} }); break + case "p": if (st.has(r.s)) st.upsert(r.s, "raw", [ r.t ], [ r.v ]); break + case "r": if (st.has(r.s) && isDef(c.tierBy[k])) st.upsert(r.s, k, [ r.t ], [ self.__ad(r.a) ]); break + case "d": if (st.has(r.s) && isDef(c.tierBy[k])) st.del(r.s, k, [ r.t ]); break + case "x": st.drop(r.s); break + default: c.bad++ + } + }) + } finally { + this.__ul(o) + } + }, + // applies retention to every series + __housekeep: function(c, aNow) { + var self = this, removed = 0 + if (!c.hasRetention) return 0 + c.st.ids().forEach(id => { removed += self.__retain(c, id, aNow) }) + return removed + }, + // rebuilds every rollup tier of every series from the raw points + __rebuildAll: function(c) { + var self = this + c.st.ids().forEach(id => { + c.tiers.forEach(t => { + if (t.step <= 0) return + c.st.trimBefore(id, t.key, Infinity) + }) + var r = c.st.read(id, "raw", -Infinity, Infinity) + c.tiers.forEach(t => { + if (t.step <= 0) return + var map = {}, bs = [] + r.ts.forEach((x, i) => { + var b = Math.floor(x / t.step) * t.step + if (isUnDef(map[b])) bs.push(b) + map[b] = self.__am(map[b], self.__a0(x, r.v[i])) + }) + if (bs.length > 0) c.st.upsert(id, t.key, bs, bs.map(b => map[b])) + }) + }) + this.__housekeep(c) + }, + // public helpers used by ow.ch.utils.timeseries + __save: function(aName) { var c = this.__c(aName); return this.__flush(c) }, + create : function(aName, shouldCompress, options) { + options = _$(options, "options").isMap().default({}) + var o = { + file : _$(options.file, "options.file").isString().default(__), + store : isDef(options.store) ? _$(options.store, "options.store").isMap().$_() : __, + readOnly : _$(options.readOnly, "options.readOnly").isBoolean().default(false), + create : _$(options.create, "options.create").isBoolean().default(true), + autosave : String(_$(options.autosave, "options.autosave").isString().default("always")).toLowerCase(), + saveDebounceMs : _$(options.saveDebounceMs, "options.saveDebounceMs").isNumber().default(5000), + lock : options.lock, + compactRatio : _$(options.compactRatio, "options.compactRatio").isNumber().default(0.5), + retentionRef : String(_$(options.retentionRef, "options.retentionRef").isString().default("series")).toLowerCase(), + housekeepEveryMs: _$(options.housekeepEveryMs, "options.housekeepEveryMs").isNumber().default(0) + } + if ([ "always", "debounced", "off" ].indexOf(o.autosave) < 0) throw "options.autosave must be 'always', 'debounced' or 'off'" + if ([ "series", "now" ].indexOf(o.retentionRef) < 0) throw "options.retentionRef must be 'series' or 'now'" + if (isDef(o.store)) { + _$(o.store.type, "options.store.type").isString().$_() + if (isDef(o.file)) throw "options.file and options.store can't be used together" + } + var chunk = this.__dur(options.chunk, "options.chunk") || 3600000 + var raw = _$(options.raw, "options.raw").isMap().default({}) + var tiers = [ { + key: "raw", step: 0, chunk: chunk, + keep: this.__dur(isDef(raw.keep) ? raw.keep : options.retention, "raw.keep"), + maxPoints: isDef(raw.maxPoints) ? raw.maxPoints : options.maxPoints + } ] + if (isDef(tiers[0].maxPoints) && (!isNumber(tiers[0].maxPoints) || tiers[0].maxPoints < 1)) throw "options.maxPoints must be a number >= 1" + var rolls = _$(options.rollups, "options.rollups").isArray().default([]) + var steps = {} + rolls.map(r => { + _$(r, "options.rollups[]").isMap().$_() + var step = this.__dur(r.step, "rollup step") + if (isUnDef(step)) throw "each rollup needs a step" + if (isDef(steps[step])) throw "duplicated rollup step " + step + steps[step] = true + return { key: "r" + step, step: step, chunk: Math.max(chunk, step * 60), keep: this.__dur(r.keep, "rollup keep") } + }).sort((a, b) => a.step - b.step).forEach(t => tiers.push(t)) + var c = { + mx: new java.util.concurrent.locks.ReentrantLock(), opts: o, name: aName, tiers: tiers, tierBy: {}, buf: isDef(o.file) ? [] : __, lines: 0, bad: 0, lastCompact: 0, + lastHK: 0, touched: {}, lastSave: 0, dirty: false, needNL: false + } + tiers.forEach(t => { c.tierBy[t.key] = t }) + c.hasRetention = tiers.some(t => isDef(t.keep) || isDef(t.maxPoints)) + c.rawTrims = isDef(tiers[0].keep) || isDef(tiers[0].maxPoints) + if (isDef(o.store)) { + c.st = this.__chStore(c, aName) + if (o.create === false && c.st.ids().length == 0) { c.st.close(); throw "timeseries store for '" + aName + "' not found" } + } else { + c.st = this.__memStore(c) + if (isDef(o.file)) this.__load(c) + } + this.__channels[aName] = c + if (!o.readOnly) this.__housekeep(c) + }, + destroy : function(aName) { + var c = this.__channels[aName] + if (isUnDef(c)) return + try { + if (!c.opts.readOnly && isDef(c.opts.file)) { + this.__flush(c) + this.__compactIf(c) + } + } finally { delete this.__channels[aName]; c.st.close() } + }, + size : function(aName) { + return this.__c(aName).st.rawCount() + }, + forEach : function(aName, aFunction) { + var ks = this.getKeys(aName) + for (var i = 0; i < ks.length; i++) { + try { aFunction(ks[i], this.get(aName, ks[i])) } catch(e) {} + } + }, + // full: optional selector { metric, labels, from, to } + __points: function(c, full, withV) { + var self = this, sel = isMap(full) ? full : {}, res = [] + var from = isDef(sel.from) ? this.__ts(sel.from) : -Infinity, to = isDef(sel.to) ? this.__ts(sel.to) : Infinity + this.__select(c, sel).forEach(id => { + var m = c.st.meta(id), r = c.st.read(id, "raw", from, to) + for (var i = 0; i < r.ts.length; i++) + res.push(withV ? self.__pt(m, r.ts[i], r.v[i]) : { metric: m.metric, labels: clone(m.labels), ts: r.ts[i] }) + }) + return res + }, + getAll : function(aName, full) { + return this.__points(this.__c(aName), full, true) + }, + getKeys : function(aName, full) { + return this.__points(this.__c(aName), full, false) + }, + getSortedKeys: function(aName, full) { + var self = this + return this.getKeys(aName, full).sort((a, b) => a.ts != b.ts ? a.ts - b.ts : (self.__sid(a.metric, a.labels) < self.__sid(b.metric, b.labels) ? -1 : 1)) + }, + getSet : function getSet(aName, aMatch, aK, aV, aTimestamp) { + var res = this.get(aName, aK) + if ($stream([res]).anyMatch(aMatch)) { + return this.set(aName, aK, aV, aTimestamp) + } + return __ + }, + set : function(aName, aK, aV, aTimestamp) { + var c = this.__c(aName) + this.__rw(c) + var k = this.__key(aK) + var t = isDef(k.ts) ? k.ts : (isMap(aV) && isDef(aV.ts) ? this.__ts(aV.ts) : now()) + this.__write(c, k, [ { t: t, v: this.__val(aV) } ]) + this.__changed(c) + return aK + }, + // aKs are the key fields in each value (defaults to metric, labels, ts) + setAll : function(aName, aKs, aVs, aTimestamp) { + ow.loadObj() + var c = this.__c(aName) + this.__rw(c) + if (!isArray(aKs) || aKs.length == 0) aKs = [ "metric", "labels", "ts" ] + var groups = {}, order = [] + for (var i = 0; i < aVs.length; i++) { + var k = this.__key(ow.obj.filterKeys(aKs, aVs[i])) + var t = isDef(k.ts) ? k.ts : (isDef(aVs[i].ts) ? this.__ts(aVs[i].ts) : now()) + if (isUnDef(groups[k.id])) { groups[k.id] = { k: k, pts: [] }; order.push(k.id) } + groups[k.id].pts.push({ t: t, v: this.__val(aVs[i]) }) + } + order.forEach(id => this.__write(c, groups[id].k, groups[id].pts)) + this.__changed(c) + }, + unsetAll : function(aName, aKs, aVs, aTimestamp) { + ow.loadObj() + if (!isArray(aKs) || aKs.length == 0) aKs = [ "metric", "labels", "ts" ] + for (var i = 0; i < aVs.length; i++) { + this.unset(aName, ow.obj.filterKeys(aKs, aVs[i]), aTimestamp) + } + }, + get : function(aName, aK) { + var c = this.__c(aName) + if (!isMap(aK) || !isString(aK.metric)) return __ + var k = this.__key(aK) + if (!c.st.has(k.id)) return __ + var t = k.ts + if (isUnDef(t)) { + var b = c.st.bounds(k.id, "raw") + if (isUnDef(b)) return __ + t = b.last + } + var r = c.st.read(k.id, "raw", t, t) + return r.ts.length > 0 ? this.__pt(k, t, r.v[0]) : __ + }, + pop : function(aName) { + var c = this.__c(aName), best + c.st.ids().sort().forEach(id => { + var b = c.st.bounds(id, "raw") + if (isDef(b) && (isUnDef(best) || b.last > best.t)) best = { id: id, t: b.last } + }) + if (isUnDef(best)) return __ + var m = c.st.meta(best.id) + return { metric: m.metric, labels: clone(m.labels), ts: best.t } + }, + shift : function(aName) { + var c = this.__c(aName), best + c.st.ids().sort().forEach(id => { + var b = c.st.bounds(id, "raw") + if (isDef(b) && (isUnDef(best) || b.first < best.t)) best = { id: id, t: b.first } + }) + if (isUnDef(best)) return __ + var m = c.st.meta(best.id) + return { metric: m.metric, labels: clone(m.labels), ts: best.t } + }, + unset : function(aName, aK, aTimestamp) { + var c = this.__c(aName) + this.__rw(c) + if (!isMap(aK) || !isString(aK.metric)) return + var k = this.__key(aK) + if (!c.st.has(k.id)) return + if (isDef(k.ts)) { + if (c.st.del(k.id, "raw", [ k.ts ]) == 0) return + this.__lg(c, { o: "d", s: k.id, t: k.ts }) + this.__fix(c, k.id, [ k.ts ]) + } else { + c.st.drop(k.id) + this.__lg(c, { o: "x", s: k.id }) + } + this.__changed(c) + } + }, // Remote channel implementation // /** @@ -3025,46 +3798,121 @@ OpenWrap.ch.prototype.__types = { * - compact (Boolean) Upon channel create/destroy it will try to run a compact operation over the file to save space.\ * - map (String/Function) The map name (defaults to 'default'). If defined as a function it will receive the key as argument if possible (only for get/set/unset/setall) * for sharding proposes.\ + * - multifile (Boolean) If true keys will be spread (by key hash) across several MVStore files on "path" (defaults to false) (*)\ + * - path (String) The folder where to keep the MVStore files (required if multifile is true)\ + * - files (Number) The number of MVStore files to use if multifile is true (defaults to 4 or to the number already in use on "path")\ * \ * The map will be created if it doesn't exist. Operations getKeys/getAll can be paginated with the extra map argument containing start and end\ + * \ + * (*) - With multifile the files are named mvs-NN.db and a mvs-multifile.json manifest records the number of files (to change it use ow.ch.utils.mvs.reshard). + * All other options apply to each file. getKeys/getAll (including start/end pagination) and pop/shift go through the files in order (no global key ordering).\ + * Each file can be handled individually with ow.ch.utils.mvs.\ * */ mvs: { + __open: function(absFile, shouldCompress) { + if (isUnDef(this.__f[absFile])) { + var s = Packages.org.h2.mvstore.MVStore.Builder().autoCommitDisabled(); + if (absFile != "memory") s = s.fileName(absFile); + if (shouldCompress) s = s.compress(); + this.__f[absFile] = s.open(); + } + if (absFile != "memory") this.__f[absFile].setReuseSpace(true); + return this.__f[absFile]; + }, + // Shard file names for a multifile path (mvs-00.db, mvs-01.db, ...) + __fnames: function(aPath, aN) { + var width = Math.max(2, String(aN - 1).length); + var res = []; + for(var i = 0; i < aN; i++) { + var n = String(i); + while(n.length < width) n = "0" + n; + res.push(aPath + "/mvs-" + n + ".db"); + } + return res; + }, + // Shard index for an already canonical (untrimmed) key string + __hidx: function(aCanonicalKey, aN) { + if (aN <= 1) return 0; + return Number(java.lang.Math.floorMod(new java.lang.String(aCanonicalKey).hashCode(), aN)); + }, + __idx: function(aName, aKey) { + // Always hash the untrimmed canonical key so internalTrim can't move keys between files + return this.__hidx(stringify(sortMapKeys(aKey), __, ""), this.__o[aName].files); + }, + __st: function(aName, aKey) { + return this.__ss[aName][this.__idx(aName, aKey)]; + }, + __k: function(aName, aKey) { + return stringify(sortMapKeys(aKey), __, this.__o[aName].stry); + }, + __page: function(aName, full, fn) { + var res = []; + full = _$(full, "full").isMap().default({}); + + var mapName = this.__m[aName](full); + var maps = this.__ss[aName].map(s => s.openMap(mapName)); + + if (isDef(full.start) || isDef(full.end)) { + var start = _$(full.start, "full.start").isNumber().default(0); + var limit = _$(full.end, "full.end").isNumber().default(Infinity); + + var offset = 0; + for(var m = 0; m < maps.length && offset < limit; m++) { + var size = Number(maps[m].sizeAsLong()); + for(var i = Math.max(start - offset, 0); i < size && offset + i < limit; i++) { + res.push(fn(maps[m], maps[m].getKey(i))); + } + offset += size; + } + } else { + maps.forEach(map => { + af.fromJavaArray(map.keyList()).forEach(k => res.push(fn(map, k))); + }); + } + + return res; + }, create : function(aName, shouldCompress, options) { if (isUnDef(options)) options = {}; if (isUnDef(options.file)) options.file = undefined; if (isUnDef(this.__s)) this.__s = {}; + if (isUnDef(this.__ss)) this.__ss = {}; if (isUnDef(this.__f)) this.__f = {}; if (isUnDef(this.__m)) this.__m = {}; if (isUnDef(this.__o)) this.__o = {}; var shut = _$(options.closeOnShutdown).isBoolean().default(true); + options.multifile = _$(options.multifile, "options.multifile").isBoolean().default(false); + + var absFiles = []; + if (options.multifile) { + _$(options.path, "options.path").isString().$_(); + var absPath = String((new java.io.File(options.path)).getAbsoluteFile()); + io.mkdir(absPath); + + var manifestFile = absPath + "/mvs-multifile.json"; + var manifest = io.fileExists(manifestFile) ? io.readFileJSON(manifestFile) : __; + if (isMap(manifest) && isNumber(manifest.files)) { + if (isDef(options.files) && options.files != manifest.files) throw "mvs multifile: options.files (" + options.files + ") doesn't match the " + manifest.files + " files already in use on " + absPath; + options.files = manifest.files; + } else { + options.files = _$(options.files, "options.files").isNumber().default(4); + if (options.files < 1 || options.files != Math.floor(options.files)) throw "mvs multifile: options.files needs to be an integer >= 1"; + io.writeFileJSON(manifestFile, { files: options.files, hash: "javaStringHashCode", version: 1 }, ""); + } - var existing = false, absFile; - if (isDef(options.file)) { - absFile = String((new java.io.File(options.file)).getAbsoluteFile()); - } else { - absFile = "memory"; - } - options.absFile = absFile; - - if (isUnDef(this.__f[absFile])) { - this.__s[aName] = Packages.org.h2.mvstore.MVStore.Builder(); - this.__s[aName] = this.__s[aName].autoCommitDisabled(); - - if (absFile != "memory") this.__s[aName] = this.__s[aName].fileName(absFile); - if (shouldCompress) this.__s[aName] = this.__s[aName].compress(); - this.__s[aName] = this.__s[aName].open(); - - this.__f[absFile] = this.__s[aName]; + absFiles = this.__fnames(absPath, options.files); } else { - existing = true; - this.__s[aName] = this.__f[absFile]; + options.files = 1; + absFiles.push(isDef(options.file) ? String((new java.io.File(options.file)).getAbsoluteFile()) : "memory"); } + options.absFile = absFiles[0]; + options.absFiles = absFiles; - //this.__s[aName].setVersionsToKeep(2); - if (isDef(options.file)) this.__s[aName].setReuseSpace(true) + this.__ss[aName] = absFiles.map(f => this.__open(f, shouldCompress)); + this.__s[aName] = this.__ss[aName][0]; if (isUnDef(options.map)) { options.map = function() { return "default"; }; @@ -3076,8 +3924,8 @@ OpenWrap.ch.prototype.__types = { this.__m[aName] = options.map; - if (isDef(options.file) && isDef(options.compact) && options.compact) { - this.__s[aName].compactFile(2500) + if ((isDef(options.file) || options.multifile) && isDef(options.compact) && options.compact) { + this.__ss[aName].forEach(s => s.compactFile(2500)); } this.__o[aName] = options; @@ -3089,28 +3937,26 @@ OpenWrap.ch.prototype.__types = { } }, destroy : function(aName) { - if (isDef(this.__o[aName].absFile) && isDef(this.__o[aName].compact) && this.__o[aName].compact) { - this.__s[aName].compactFile(2500) - } + var o = this.__o[aName], sts = this.__ss[aName]; - var found = false; - for(var i in this.__s) { - if (i != aName && - this.__s[i] == this.__f[this.__o[aName].absFile]) - found = true; - } + o.absFiles.forEach((f, i) => { + if (f != "memory" && isDef(o.compact) && o.compact) sts[i].compactFile(2500); - if (!found) { - this.__s[aName].close(); - delete this.__f[this.__o[aName].absFile]; - } + var found = Object.keys(this.__o).some(n => n != aName && this.__o[n].absFiles.indexOf(f) >= 0); + if (!found) { + sts[i].close(); + delete this.__f[f]; + } + }); delete this.__s[aName]; + delete this.__ss[aName]; delete this.__o[aName]; }, size : function(aName) { - var map = this.__s[aName].openMap(this.__m[aName]()); - return map.sizeAsLong(); + var mapName = this.__m[aName](), size = 0; + this.__ss[aName].forEach(s => size += Number(s.openMap(mapName).sizeAsLong())); + return size; }, forEach : function(aName, aFunction, x) { var aKs = this.getKeys(aName, x); @@ -3120,44 +3966,10 @@ OpenWrap.ch.prototype.__types = { } }, getKeys : function(aName, full) { - var res = []; - full = _$(full, "full").isMap().default({}); - - var map = this.__s[aName].openMap(this.__m[aName](full)); - - if (isDef(full.start) || isDef(full.end)) { - var start = _$(full.start, "full.start").isNumber().default(0); - var max = this.size(aName); - var limit = _$(full.end, "full.end").isNumber().default(max); - - for(var i = start; i < limit && i < max; i++) { - res.push(jsonParse(map.getKey(i), true)); - } - } else { - res = af.fromJavaArray(map.keyList()).map(r => jsonParse(r, true)) - } - - return res; + return this.__page(aName, full, (map, k) => jsonParse(k, true)); }, getAll : function(aName, full) { - var res = []; - full = _$(full, "full").isMap().default({}); - - var map = this.__s[aName].openMap(this.__m[aName](full)); - - if (isDef(full.start) || isDef(full.end)) { - var start = _$(full.start, "full.start").isNumber().default(0); - var max = this.size(aName); - var limit = _$(full.end, "full.end").isNumber().default(max); - - for(var i = start; i < limit && i < max; i++) { - res.push(jsonParse(map.get(map.getKey(i)), true)); - } - } else { - res = af.fromJavaArray(map.keyList()).map(k => jsonParse(map.get(k), true)) - } - - return res; + return this.__page(aName, full, (map, k) => jsonParse(map.get(k), true)); }, getSortedKeys: function(aName, full) { return this.getKeys(aName, full); @@ -3168,57 +3980,74 @@ OpenWrap.ch.prototype.__types = { if ($stream([res]).anyMatch(aMatch)) { return this.set(aName, aK, aV, aTimestamp); } - return undefined; + return undefined; }, set : function(aName, ak, av, aTimestamp) { - var map = this.__s[aName].openMap(this.__m[aName](ak)); + var s = this.__st(aName, ak); + var map = s.openMap(this.__m[aName](ak)); - map.put(stringify(sortMapKeys(ak), __, this.__o[aName].stry), stringify(av, __, this.__o[aName].stry)); - this.__s[aName].commit(); + map.put(this.__k(aName, ak), stringify(av, __, this.__o[aName].stry)); + s.commit(); return ak; }, setAll : function(aName, anArrayOfKeys, anArrayOfMapData, aTimestamp) { ow.loadObj(); + var touched = {}; for(var iii in anArrayOfMapData) { var ak = ow.obj.filterKeys(anArrayOfKeys, anArrayOfMapData[iii]); var av = anArrayOfMapData[iii]; - var map = this.__s[aName].openMap(this.__m[aName](ak)); - map.put(stringify(sortMapKeys(ak), __, this.__o[aName].stry), stringify(av, __, this.__o[aName].stry)); - } - this.__s[aName].sync(); - this.__s[aName].commit(); + var idx = this.__idx(aName, ak); + var map = this.__ss[aName][idx].openMap(this.__m[aName](ak)); + map.put(this.__k(aName, ak), stringify(av, __, this.__o[aName].stry)); + touched[idx] = true; + } + Object.keys(touched).forEach(i => { + this.__ss[aName][i].sync(); + this.__ss[aName][i].commit(); + }); }, unsetAll : function(aName, anArrayOfKeys, anArrayOfMapData, aTimestamp) { ow.loadObj(); + var touched = {}; for(var iii in anArrayOfMapData) { var ak = ow.obj.filterKeys(anArrayOfKeys, anArrayOfMapData[iii]); - //var av = anArrayOfMapData[iii]; - var map = this.__s[aName].openMap(this.__m[aName](ak)); - map.remove(stringify(sortMapKeys(ak), __, this.__o[aName].stry)); - } - this.__s[aName].sync(); - this.__s[aName].commit(); + var idx = this.__idx(aName, ak); + var map = this.__ss[aName][idx].openMap(this.__m[aName](ak)); + map.remove(this.__k(aName, ak)); + touched[idx] = true; + } + Object.keys(touched).forEach(i => { + this.__ss[aName][i].sync(); + this.__ss[aName][i].commit(); + }); }, get : function(aName, aKey) { - var map = this.__s[aName].openMap(this.__m[aName](aKey)); + var map = this.__st(aName, aKey).openMap(this.__m[aName](aKey)); - var r = map.get(stringify(sortMapKeys(aKey), __, this.__o[aName].stry)); + var r = map.get(this.__k(aName, aKey)); if (r == null || isUnDef(r)) return __; else return jsonParse(r, true); }, + // pop/shift return the key; ow.ch.pop/shift then get and unset it pop : function(aName) { - var map = this.__s[aName].openMap(this.__m[aName]()); - var aKey = map.lastKey(); - return jsonParse(map.remove(aKey), true); + var mapName = this.__m[aName](), sts = this.__ss[aName]; + for(var i = sts.length - 1; i >= 0; i--) { + var map = sts[i].openMap(mapName); + if (map.sizeAsLong() > 0) return jsonParse(map.lastKey(), true); + } + return __; }, shift : function(aName) { - var map = this.__s[aName].openMap(this.__m[aName]()); - var aKey = map.firstKey(); - return jsonParse(map.remove(aKey), true); + var mapName = this.__m[aName](), sts = this.__ss[aName]; + for(var i = 0; i < sts.length; i++) { + var map = sts[i].openMap(mapName); + if (map.sizeAsLong() > 0) return jsonParse(map.firstKey(), true); + } + return __; }, unset : function(aName, aKey) { - var map = this.__s[aName].openMap(this.__m[aName](aKey)); + var map = this.__st(aName, aKey).openMap(this.__m[aName](aKey)); - return jsonParse(map.remove(stringify(sortMapKeys(aKey), __, this.__o[aName].stry)), true); + return jsonParse(map.remove(this.__k(aName, aKey)), true); } }, /** @@ -5301,6 +6130,86 @@ OpenWrap.ch.prototype.utils = { } return true; + }, + + /** + * + * ow.ch.utils.mvs.reshard(aPath, aNewFiles, shouldCompress) : Number + * Changes the number of files of an "mvs" channel created with multifile = true on aPath to aNewFiles, redistributing + * all the entries of all the maps. Returns the number of entries moved. The channel(s) using aPath need to be destroyed first. + * The new files are built on a temporary folder inside aPath and only replace the current ones (and the mvs-multifile.json manifest) after + * all the entries are copied and counted. Use shouldCompress = true if the channel was created with it. + * + */ + reshard: function(aPath, aNewFiles, shouldCompress) { + _$(aPath, "aPath").isString().$_(); + _$(aNewFiles, "aNewFiles").isNumber().$_(); + if (aNewFiles < 1 || aNewFiles != Math.floor(aNewFiles)) throw "aNewFiles needs to be an integer >= 1"; + + var t = ow.ch.__types.mvs; + var absPath = String((new java.io.File(aPath)).getAbsoluteFile()); + var manifestFile = absPath + "/mvs-multifile.json"; + var manifest = io.fileExists(manifestFile) ? io.readFileJSON(manifestFile) : __; + if (!isMap(manifest) || !isNumber(manifest.files)) throw "No mvs multifile manifest found on " + absPath; + + var oldFiles = t.__fnames(absPath, manifest.files); + var inUse = isMap(t.__o) && Object.keys(t.__o).some(n => t.__o[n].absFiles.some(f => oldFiles.indexOf(f) >= 0)); + if (inUse) throw "Channels using " + absPath + " need to be destroyed before resharding"; + if (manifest.files == aNewFiles) return 0; + + var tmpPath = absPath + "/.reshard-tmp"; + if (io.fileExists(tmpPath)) io.rm(tmpPath); + io.mkdir(tmpPath); + + var newFiles = t.__fnames(tmpPath, aNewFiles); + var sources = [], targets = [], moved = 0, expected = 0; + var open = (f, comp) => { + var b = Packages.org.h2.mvstore.MVStore.Builder().autoCommitDisabled().fileName(f); + if (comp) b = b.compress(); + return b.open(); + }; + + try { + sources = oldFiles.map(f => open(f, shouldCompress)); + targets = newFiles.map(f => open(f, shouldCompress)); + + sources.forEach(src => { + af.fromJavaArray(src.getMapNames().toArray()).forEach(mapName => { + mapName = String(mapName); + var smap = src.openMap(mapName); + af.fromJavaArray(smap.keyList()).forEach(k => { + // Same placement rule as the channel: hash of the untrimmed canonical key + var idx = t.__hidx(stringify(sortMapKeys(jsonParse(String(k), true)), __, ""), aNewFiles); + targets[idx].openMap(mapName).put(k, smap.get(k)); + moved++; + }); + expected += Number(smap.sizeAsLong()); + }); + }); + + var total = 0; + targets.forEach(tgt => { + tgt.commit(); + af.fromJavaArray(tgt.getMapNames().toArray()).forEach(mapName => total += Number(tgt.openMap(String(mapName)).sizeAsLong())); + }); + if (total != expected) throw "reshard failed: expected " + expected + " entries but copied " + total; + } catch(e) { + targets.forEach(tgt => { try { tgt.close() } catch(ee) {} }); + sources.forEach(src => { try { src.close() } catch(ee) {} }); + io.rm(tmpPath); + throw e; + } + + targets.forEach(tgt => tgt.close()); + sources.forEach(src => src.close()); + + // Swap: remove the old files, move the new ones and finally update the manifest + oldFiles.forEach(f => { if (io.fileExists(f)) io.rm(f); }); + t.__fnames(absPath, aNewFiles).forEach((f, i) => io.mv(newFiles[i], f)); + io.writeFileJSON(manifestFile, { files: aNewFiles, hash: "javaStringHashCode", version: 1 }, ""); + io.rm(tmpPath); + + return moved; } } }; @@ -5417,6 +6326,357 @@ OpenWrap.ch.prototype.utils.graph = { } }; +/** + * + * ow.ch.utils.timeseries + * + * + */ +OpenWrap.ch.prototype.utils.timeseries = { + __t: () => ow.ch.__types.timeseries, + __FNS: [ "avg", "min", "max", "sum", "count", "first", "last", "rate", "increase" ], + // finest tier that reaches back to 'from' and whose buckets divide 'step' + __pick: function(c, id, from, step, force) { + var tt = this.__t() + if (isDef(force)) { + if (String(force) == "raw") return c.tiers[0] + var s = tt.__dur(force, "tier"), f = c.tiers.filter(t => t.step == s) + if (f.length == 0) throw "timeseries: tier '" + force + "' not found" + return f[0] + } + if (isUnDef(step)) return c.tiers[0] + var cand = c.tiers.filter(t => t.step == 0 || (t.step <= step && step % t.step == 0)), best, bestFirst + cand.forEach(t => { + var b = c.st.bounds(id, t.key) + if (isUnDef(b)) return + if (isUnDef(best) || b.first <= bestFirst - t.step) { best = t; bestFirst = b.first } + }) + if (isUnDef(best)) return c.tiers[0] + for (var i = 0; i < cand.length; i++) { + var bd = c.st.bounds(id, cand[i].key) + if (isUnDef(bd)) continue + if (cand[i] === best || bd.first <= from) return cand[i] + } + return best + }, + __row: function(m, t, v) { + var r = { ts: t, metric: m.metric } + Object.keys(m.labels).forEach(k => { r[(k == "ts" || k == "metric" || k == "v") ? "_" + k : k] = m.labels[k] }) + r.v = v + return r + }, + series: function(aName, aSelector) { + var tt = this.__t(), c = tt.__c(aName) + return tt.__select(c, aSelector).map(id => { var m = c.st.meta(id); return { metric: m.metric, labels: clone(m.labels) } }) + }, + labels: function(aName, aMetric) { + var res = {} + this.series(aName, isDef(aMetric) ? { metric: aMetric } : {}).forEach(s => { + Object.keys(s.labels).forEach(k => { + if (isUnDef(res[k])) res[k] = [] + if (res[k].indexOf(s.labels[k]) < 0) res[k].push(s.labels[k]) + }) + }) + Object.keys(res).forEach(k => res[k].sort()) + return res + }, + latest: function(aName, aSelector) { + var tt = this.__t(), c = tt.__c(aName), res = [] + tt.__select(c, aSelector).forEach(id => { + var b = c.st.bounds(id, "raw") + if (isUnDef(b)) return + var m = c.st.meta(id), r = c.st.read(id, "raw", b.last, b.last) + res.push(this.__row(m, b.last, r.v[0])) + }) + return res + }, + range: function(aName, aSelector, aOptions) { + var tt = this.__t(), c = tt.__c(aName), self = this + aOptions = _$(aOptions, "options").isMap().default({}) + var from = isDef(aOptions.from) ? tt.__ts(aOptions.from) : -Infinity, to = isDef(aOptions.to) ? tt.__ts(aOptions.to) : Infinity + var step = tt.__dur(aOptions.step, "step"), fn = String(isDef(aOptions.fn) ? aOptions.fn : "avg"), fill = String(isDef(aOptions.fill) ? aOptions.fill : "none") + if (this.__FNS.indexOf(fn) < 0) throw "timeseries: unknown fn '" + fn + "' (use " + this.__FNS.join(", ") + ")" + if ([ "none", "null", "zero", "prev" ].indexOf(fill) < 0) throw "timeseries: unknown fill '" + fill + "'" + var res = [] + tt.__select(c, aSelector).forEach(id => { + var m = c.st.meta(id), tier = self.__pick(c, id, from, step, aOptions.tier) + var s = step + if (isUnDef(s) && tier.step > 0) s = tier.step + var rd = c.st.read(id, tier.key, tier.step > 0 ? from - tier.step + 1 : from, to) + if (tier.step > 0 && isDef(from) && isFinite(from)) { + // drop rollup buckets that end before 'from' + var keep = rd.ts.map((b, i) => b + tier.step > from) + rd = { ts: rd.ts.filter((b, i) => keep[i]), v: rd.v.filter((b, i) => keep[i]) } + } + // no step on the raw tier: the points as they are + if (isUnDef(s)) { + rd.ts.forEach((t, i) => res.push(self.__row(m, t, rd.v[i]))) + return + } + var rows = {}, order = [], bucket = t => Math.floor(t / s) * s + if (fn == "rate" || fn == "increase") { + var prevLast + rd.ts.forEach((t, i) => { + var inc = 0, b = bucket(t) + if (tier.step == 0) { + var v = rd.v[i] + if (isDef(prevLast)) inc = v >= prevLast ? v - prevLast : v + prevLast = v + } else { + var a = rd.v[i] + inc = a.last >= a.first ? a.last - a.first : a.last + if (isDef(prevLast)) inc += a.first >= prevLast ? a.first - prevLast : a.first + prevLast = a.last + } + if (isUnDef(rows[b])) { rows[b] = 0; order.push(b) } + rows[b] += inc + }) + order.forEach(b => { rows[b] = fn == "rate" ? rows[b] / (s / 1000) : rows[b] }) + } else { + rd.ts.forEach((t, i) => { + var b = bucket(t), a = tier.step == 0 ? tt.__a0(t, rd.v[i]) : rd.v[i] + if (isUnDef(rows[b])) order.push(b) + rows[b] = tt.__am(rows[b], a) + }) + order.forEach(b => { + var a = rows[b] + rows[b] = fn == "avg" ? a.sum / a.count : a[fn] + }) + } + order.sort((a, b) => a - b) + var out = order.map(b => ({ b: b, v: rows[b] })) + if (fill != "none" && out.length > 0) { + var f0 = isFinite(from) ? bucket(from) : out[0].b, f1 = isFinite(to) ? bucket(to) : out[out.length - 1].b + if ((f1 - f0) / s > 1000000) throw "timeseries: too many buckets to fill" + var byB = {}, last + out.forEach(o => { byB[o.b] = o.v }) + out = [] + for (var b = f0; b <= f1; b += s) { + if (isDef(byB[b])) { last = byB[b]; out.push({ b: b, v: last }) } + else out.push({ b: b, v: fill == "null" ? null : (fill == "zero" ? 0 : (isDef(last) ? last : null)) }) + } + } + out.forEach(o => res.push(self.__row(m, o.b, o.v))) + }) + return res + }, + drop: function(aName, aSelector) { + var tt = this.__t(), c = tt.__c(aName) + tt.__rw(c) + var ids = tt.__select(c, aSelector) + ids.forEach(id => { c.st.drop(id); tt.__lg(c, { o: "x", s: id }) }) + if (ids.length > 0) tt.__changed(c) + return ids.length + }, + stats: function(aName) { + var tt = this.__t(), c = tt.__c(aName), res = { series: 0, points: 0, metrics: {}, tiers: {} } + c.tiers.forEach(t => { res.tiers[t.key] = { step: t.step, entries: 0 } }) + c.st.ids().forEach(id => { + var m = c.st.meta(id) + res.series++ + if (isUnDef(res.metrics[m.metric])) res.metrics[m.metric] = { series: 0, points: 0, first: __, last: __ } + var ms = res.metrics[m.metric] + ms.series++ + c.tiers.forEach(t => { + var b = c.st.bounds(id, t.key) + if (isUnDef(b)) return + res.tiers[t.key].entries += b.count + if (t.step == 0) { + res.points += b.count; ms.points += b.count + ms.first = isUnDef(ms.first) ? b.first : Math.min(ms.first, b.first) + ms.last = isUnDef(ms.last) ? b.last : Math.max(ms.last, b.last) + } + }) + }) + return res + }, + housekeep: function(aName, aOptions) { + var tt = this.__t(), c = tt.__c(aName) + aOptions = _$(aOptions, "options").isMap().default({}) + tt.__rw(c) + var n = tt.__housekeep(c, isDef(aOptions.now) ? tt.__ts(aOptions.now) : __) + if (isDef(c.opts.file)) { tt.__flush(c); tt.__compactIf(c) } + return n + }, + rebuildRollups: function(aName) { + var tt = this.__t(), c = tt.__c(aName) + tt.__rw(c) + tt.__rebuildAll(c) + if (isDef(c.opts.file)) tt.__compact(c) + }, + compact: function(aName) { + var tt = this.__t() + return tt.__compact(tt.__c(aName)) + }, + save: function(aName) { + return this.__t().__save(aName) + }, + // flattens a map of metrics into [ { metric, labels, v } ]; nested maps extend the metric name, arrays add an 'idx' label + __flat: function(aObj, aPrefix, aLabels, aOut) { + var self = this + if (isNumber(aObj) && isFinite(aObj)) aOut.push({ metric: aPrefix, labels: aLabels, v: aObj }) + else if (isBoolean(aObj)) aOut.push({ metric: aPrefix, labels: aLabels, v: aObj ? 1 : 0 }) + else if (isArray(aObj)) aObj.forEach((o, i) => self.__flat(o, aPrefix, merge(aLabels, { idx: String(i) }), aOut)) + else if (isMap(aObj)) Object.keys(aObj).forEach(k => { var n = k.replace(/^_+/, ""); self.__flat(aObj[k], isDef(aPrefix) && aPrefix.length > 0 ? aPrefix + "_" + n : n, aLabels, aOut) }) + return aOut + }, + __sane: s => String(s).replace(/[^a-zA-Z0-9_:]/g, "_").replace(/^(\d)/, "_$1"), + collect: function(aEvery, aName, aSource, aOptions) { + var self = this, tt = this.__t() + aOptions = _$(aOptions, "options").isMap().default({}) + var labels = tt.__labels(aOptions.labels), prefix = isString(aOptions.prefix) ? aOptions.prefix : "" + tt.__c(aName) + var once = function() { + var raw = isFunction(aSource) ? aSource() : (isArray(aSource) ? ow.loadMetrics().getSome(aSource) : ow.loadMetrics().getAll()) + var pts = [] + if (isArray(raw) && raw.length > 0 && isString(raw[0].metric)) { + pts = raw.map(r => ({ metric: r.metric, labels: merge(labels, tt.__labels(r.labels)), v: r.v, ts: r.ts })) + } else { + pts = self.__flat(raw, "", labels, []) + } + var t = now() + pts = pts.filter(p => isString(p.metric) && p.metric.length > 0).map(p => ({ metric: self.__sane(prefix + p.metric), labels: p.labels, ts: isDef(p.ts) ? p.ts : t, v: p.v })) + if (pts.length > 0) $ch(aName).setAll([ "metric", "labels", "ts" ], pts) + return pts.length + } + var __f = function() { + if ($ch().list().indexOf(aName) < 0) { res.stop(); return } + try { once() } catch(e) { if (isFunction(aOptions.onError)) aOptions.onError(e); else logErr("timeseries collect: " + e) } + } + var res + if (isString(aEvery)) { + ow.loadServer() + var sch = new ow.server.scheduler() + sch.addEntry(aEvery, __f) + res = { once: once, stop: () => sch.stop() } + } else if (isNumber(aEvery)) { + var th = new Threads() + th.addScheduleThreadWithFixedDelay(__f, aEvery) + res = { once: once, stop: () => th.stop() } + } else { + res = { once: once, stop: () => {} } + } + return res + }, + // records changes of a channel into aTargetCh. aOptions: metric (String|Function(key, value)), labels (Array of field names | Function), values (Array of field names; one metric per field named metric_field) or value (field name | Function), ts (field name | Function) + getTimeSeriesSubscriber: function(aTargetCh, aOptions) { + var self = this + ow.loadObj() + _$(aTargetCh, "aTargetCh").isString().$_() + aOptions = _$(aOptions, "options").isMap().$_() + if (isUnDef(aOptions.metric)) throw "options.metric is required" + var ops = isArray(aOptions.ops) ? aOptions.ops : [ "set", "setall" ] + var rec = function(k, v, out) { + var all = merge(isMap(k) ? k : {}, isMap(v) ? v : {}) + var metric = isFunction(aOptions.metric) ? aOptions.metric(k, v) : aOptions.metric + var lbs = {} + if (isFunction(aOptions.labels)) lbs = aOptions.labels(k, v) + else if (isArray(aOptions.labels)) aOptions.labels.forEach(f => { if (isDef(all[f])) lbs[f] = all[f] }) + var t = isFunction(aOptions.ts) ? aOptions.ts(k, v) : (isString(aOptions.ts) ? all[aOptions.ts] : __) + var add = (m, x) => { if (isNumber(x) && isFinite(x)) out.push({ metric: self.__sane(m), labels: lbs, ts: t, v: x }) } + if (isArray(aOptions.values)) aOptions.values.forEach(f => add(metric + "_" + f, all[f])) + else add(metric, isFunction(aOptions.value) ? aOptions.value(k, v) : (isString(aOptions.value) ? all[aOptions.value] : all.v)) + } + return function(aC, aO, aK, aV) { + if (ops.indexOf(aO) < 0 || aC == aTargetCh) return + var out = [] + if (aO == "set") rec(aK, aV, out) + else if (aO == "setall") { for (var i in aV) rec(ow.obj.filterKeys(isArray(aK) ? aK : [], aV[i]), aV[i], out) } + if (out.length > 0) $ch(aTargetCh).setAll([ "metric", "labels", "ts" ], out) + } + }, + toOpenMetrics: function(aName, aSelector, aOptions) { + var tt = this.__t(), c = tt.__c(aName), self = this + aOptions = _$(aOptions, "options").isMap().default({}) + var unit = aOptions.tsUnit == "ms" ? 1 : 1000, help = _$(aOptions.help, "options.help").isMap().default({}) + var all = isDef(aOptions.from) || isDef(aOptions.to) + var from = isDef(aOptions.from) ? tt.__ts(aOptions.from) : -Infinity, to = isDef(aOptions.to) ? tt.__ts(aOptions.to) : Infinity + var esc = s => String(s).replace(/\\/g, "\\\\").replace(/"/g, "\\\"").replace(/\n/g, "\\n") + var lines = [], seen = {} + tt.__select(c, aSelector).forEach(id => { + var m = c.st.meta(id), name = self.__sane(m.metric), ls = Object.keys(m.labels).map(k => self.__sane(k) + "=\"" + esc(m.labels[k]) + "\"").join(",") + var r + if (all) r = c.st.read(id, "raw", from, to) + else { var b = c.st.bounds(id, "raw"); if (isUnDef(b)) return; r = c.st.read(id, "raw", b.last, b.last) } + if (!seen[name] && isMap(help[m.metric])) { + seen[name] = true + if (isDef(help[m.metric].help)) lines.push("# HELP " + name + " " + help[m.metric].help) + if (isDef(help[m.metric].type)) lines.push("# TYPE " + name + " " + help[m.metric].type) + } + for (var i = 0; i < r.ts.length; i++) + lines.push(name + (ls.length > 0 ? "{" + ls + "}" : "") + " " + r.v[i] + (aOptions.timestamps === false ? "" : " " + (r.ts[i] / unit))) + }) + return lines.join("\n") + (lines.length > 0 ? "\n" : "") + }, + fromOpenMetrics: function(aName, aText, aOptions) { + var tt = this.__t(), c = tt.__c(aName), self = this + aOptions = _$(aOptions, "options").isMap().default({}) + var unit = aOptions.tsUnit == "ms" ? 1 : 1000, extra = tt.__labels(aOptions.labels) + var re = /^([a-zA-Z_:][a-zA-Z0-9_:]*)(?:\{(.*)\})?\s+(\S+)(?:\s+(\S+))?\s*$/ + var pts = [] + String(aText).split("\n").forEach(line => { + line = line.trim() + if (line.length == 0 || line.charAt(0) == "#") return + var mm = line.match(re) + if (!isArray(mm)) return + var v = Number(mm[3]) + if (!isFinite(v)) return + var lbs = {}, lre = /([a-zA-Z_][a-zA-Z0-9_]*)="((?:[^"\\]|\\.)*)"/g, x + while ((x = lre.exec(mm[2] || "")) !== null) lbs[x[1]] = x[2].replace(/\\n/g, "\n").replace(/\\"/g, "\"").replace(/\\\\/g, "\\") + var t = isDef(mm[4]) && isFinite(Number(mm[4])) ? Math.round(Number(mm[4]) * unit) : (isDef(aOptions.ts) ? tt.__ts(aOptions.ts) : now()) + pts.push({ metric: mm[1], labels: merge(lbs, extra), ts: t, v: v }) + }) + if (pts.length > 0) $ch(aName).setAll([ "metric", "labels", "ts" ], pts) + return pts.length + } +}; + +// Serializes the access to each timeseries channel: collectors and subscribers write from other threads while queries read the same arrays +;(function() { + var wrap = function(o, names) { + names.forEach(function(n) { + var orig = o[n] + o[n] = function(aName) { + var c = ow.ch.__types.timeseries.__channels[aName] + if (isUnDef(c)) return orig.apply(this, arguments) + c.mx.lock() + try { return orig.apply(this, arguments) } finally { c.mx.unlock() } + } + }) + } + wrap(OpenWrap.ch.prototype.__types.timeseries, [ "destroy", "size", "forEach", "getAll", "getKeys", "getSortedKeys", "getSet", "set", "setAll", "unset", "unsetAll", "get", "pop", "shift" ]) + wrap(OpenWrap.ch.prototype.utils.timeseries, [ "series", "labels", "latest", "range", "drop", "stats", "housekeep", "rebuildRollups", "compact", "save", "toOpenMetrics" ]) +})(); + OpenWrap.ch.prototype.comms = { __counter: {}, diff --git a/tests/autoTestAll.Channels.js b/tests/autoTestAll.Channels.js index 429c470f..c10163cb 100644 --- a/tests/autoTestAll.Channels.js +++ b/tests/autoTestAll.Channels.js @@ -260,6 +260,138 @@ io.rm("testMVS.db"); }; + var withMVSMultifile = function(fn) { + var path = String(java.nio.file.Files.createTempDirectory("openaf-mvs-")); + try { + fn(path); + } finally { + io.rm(path); + } + }; + + exports.testMVSMultifile = function() { + withMVSMultifile(function(path) { + var name = "mvs-multifile-" + genUUID(); + $ch(name).create(1, "mvs", { multifile: true, path: path, files: 3 }); + for(var i = 0; i < 50; i++) $ch(name).set({ id: i }, { id: i, v: i * 2 }); + + ow.test.assert($ch(name).size(), 50, "Problem with mvs multifile size."); + ow.test.assert($ch(name).get({ id: 7 }), { id: 7, v: 14 }, "Problem with mvs multifile get."); + ow.test.assert($ch(name).getKeys().length, 50, "Problem with mvs multifile getKeys."); + ow.test.assert($ch(name).getAll().length, 50, "Problem with mvs multifile getAll."); + + var files = io.listFilenames(path).filter(f => f.endsWith(".db")); + ow.test.assert(files.length, 3, "Problem with the number of mvs multifile files."); + ow.test.assert(io.readFileJSON(path + "/mvs-multifile.json").files, 3, "Problem with mvs multifile manifest."); + + $ch(name).unset({ id: 7 }); + ow.test.assert(isUnDef($ch(name).get({ id: 7 })), true, "Problem with mvs multifile unset."); + $ch(name).unsetAll(["id"], [{ id: 1 }, { id: 2 }, { id: 3 }]); + ow.test.assert($ch(name).size(), 46, "Problem with mvs multifile unsetAll."); + $ch(name).setAll(["id"], [{ id: 100, v: 1 }, { id: 101, v: 2 }]); + ow.test.assert($ch(name).size(), 48, "Problem with mvs multifile setAll."); + $ch(name).destroy(); + + // Re-open without files: the manifest must be honored + $ch(name).create(1, "mvs", { multifile: true, path: path }); + ow.test.assert($ch(name).size(), 48, "Problem with mvs multifile reopen."); + ow.test.assert($ch(name).get({ id: 100 }), { id: 100, v: 1 }, "Problem with mvs multifile reopen get."); + $ch(name).destroy(); + }); + }; + + exports.testMVSMultifileManifestMismatch = function() { + withMVSMultifile(function(path) { + var name = "mvs-multifile-" + genUUID(); + $ch(name).create(1, "mvs", { multifile: true, path: path, files: 2 }); + $ch(name).destroy(); + + var failed = false; + try { $ch(name).create(1, "mvs", { multifile: true, path: path, files: 5 }); } catch(e) { failed = true; } + ow.test.assert(failed, true, "mvs multifile should fail on a different number of files."); + try { $ch(name).destroy(); } catch(e) {} + }); + }; + + exports.testMVSMultifilePagination = function() { + withMVSMultifile(function(path) { + var name = "mvs-multifile-" + genUUID(); + $ch(name).create(1, "mvs", { multifile: true, path: path, files: 4 }); + try { + for(var i = 0; i < 40; i++) $ch(name).set({ id: i }, { id: i }); + var all = $ch(name).getKeys(); + [[0, 5], [3, 17], [9, 31], [35, 40], [0, 40], [38, 100]].forEach(r => { + ow.test.assert($ch(name).getKeys({ start: r[0], end: r[1] }), all.slice(r[0], r[1]), "Problem with mvs multifile getKeys pagination " + r); + }); + ow.test.assert($ch(name).getAll({ start: 10, end: 20 }).map(r => r.id), all.slice(10, 20).map(r => r.id), "Problem with mvs multifile getAll pagination."); + + ow.test.assert($ch(name).shift(), all[0], "Problem with mvs multifile shift."); + ow.test.assert($ch(name).pop(), all[all.length - 1], "Problem with mvs multifile pop."); + ow.test.assert($ch(name).size(), 38, "Problem with mvs multifile size after shift/pop."); + } finally { + $ch(name).destroy(); + } + }); + }; + + exports.testMVSShiftPop = function() { + var name = "mvs-shiftpop-" + genUUID(); + $ch(name).create(1, "mvs", { map: name }); + try { + $ch(name).set({ id: 1 }, { id: 1, v: "a" }); + $ch(name).set({ id: 2 }, { id: 2, v: "b" }); + $ch(name).set({ id: 3 }, { id: 3, v: "c" }); + ow.test.assert($ch(name).shift(), { id: 1, v: "a" }, "Problem with mvs shift."); + ow.test.assert($ch(name).pop(), { id: 3, v: "c" }, "Problem with mvs pop."); + ow.test.assert($ch(name).getKeys(), [{ id: 2 }], "Problem with mvs keys after shift/pop."); + } finally { + $ch(name).destroy(); + } + }; + + exports.testMVSMultifileReshard = function() { + withMVSMultifile(function(path) { + var name = "mvs-multifile-" + genUUID(); + $ch(name).create(1, "mvs", { multifile: true, path: path, files: 2, map: "m1" }); + for(var i = 0; i < 60; i++) $ch(name).set({ id: i }, { id: i, v: i * 3 }); + + var failed = false; + try { ow.ch.utils.mvs.reshard(path, 5); } catch(e) { failed = true; } + ow.test.assert(failed, true, "mvs reshard should fail while a channel is using the path."); + $ch(name).destroy(); + + ow.test.assert(ow.ch.utils.mvs.reshard(path, 5), 60, "Problem with the number of entries resharded."); + ow.test.assert(io.readFileJSON(path + "/mvs-multifile.json").files, 5, "Problem with the manifest after reshard."); + ow.test.assert(io.listFilenames(path).filter(f => f.endsWith(".db")).length, 5, "Problem with the number of files after reshard."); + ow.test.assert(io.fileExists(path + "/.reshard-tmp"), false, "Reshard should remove its temporary folder."); + + $ch(name).create(1, "mvs", { multifile: true, path: path, map: "m1" }); + ow.test.assert($ch(name).size(), 60, "Problem with size after reshard."); + for(var i = 0; i < 60; i++) ow.test.assert($ch(name).get({ id: i }), { id: i, v: i * 3 }, "Problem with get after reshard for " + i); + $ch(name).destroy(); + + // Shrinking down to a single file + ow.test.assert(ow.ch.utils.mvs.reshard(path, 1), 60, "Problem resharding down to one file."); + $ch(name).create(1, "mvs", { multifile: true, path: path, map: "m1" }); + ow.test.assert($ch(name).get({ id: 59 }), { id: 59, v: 177 }, "Problem with get after reshard to one file."); + $ch(name).destroy(); + }); + }; + + exports.testMVSMultifileSharing = function() { + withMVSMultifile(function(path) { + var n1 = "mvs-multifile-" + genUUID(), n2 = "mvs-multifile-" + genUUID(); + $ch(n1).create(1, "mvs", { multifile: true, path: path, files: 2 }); + $ch(n2).create(1, "mvs", { multifile: true, path: path, files: 2 }); + $ch(n1).set({ id: 1 }, { id: 1 }); + ow.test.assert($ch(n2).get({ id: 1 }), { id: 1 }, "Problem with mvs multifile sharing."); + $ch(n1).destroy(); + $ch(n2).set({ id: 2 }, { id: 2 }); + ow.test.assert($ch(n2).size(), 2, "Problem with mvs multifile after destroying a sharing channel."); + $ch(n2).destroy(); + }); + }; + exports.testCreateChannel = function() { var ops; if (this.chType == "cache") { @@ -544,6 +676,423 @@ } }; + // Battery shared by the memory/file and the store flavours of the timeseries channel + var __tsCommon = function(name) { + var u = ow.ch.utils.timeseries; + var K = (h, ts) => ({ metric: "cpu", labels: { host: h }, ts: ts }); + for (var i = 1; i <= 5; i++) $ch(name).set(K("a", i * 1000), { v: i }); + $ch(name).set(K("b", 1000), { v: 10 }); + $ch(name).set({ metric: "mem", ts: 1000 }, { v: 7 }); + ow.test.assert($ch(name).size(), 7, "Problem with timeseries size"); + ow.test.assert($ch(name).get({ metric: "cpu", labels: { host: "a" } }).v, 5, "Problem getting the latest timeseries point"); + ow.test.assert($ch(name).get({ metric: "cpu", labels: { host: "a" } }).ts, 5000, "Problem getting the latest timeseries ts"); + ow.test.assert($ch(name).get(K("a", 2000)).v, 2, "Problem getting an exact timeseries point"); + ow.test.assert($ch(name).get(K("a", 2500)), __, "Missing timeseries point must be undefined"); + ow.test.assert($ch(name).get({ metric: "nope" }), __, "Missing timeseries must be undefined"); + + // Overwrite, out-of-order, timestamp formats + $ch(name).set(K("a", 3000), { v: 30 }); + ow.test.assert($ch(name).size(), 7, "Overwriting a timeseries point must not add one"); + ow.test.assert($ch(name).get(K("a", 3000)).v, 30, "Problem overwriting a timeseries point"); + $ch(name).set(K("a", 500), { v: 0.5 }); + ow.test.assert($ch(name).get(K("a", 500)).v, 0.5, "Problem with an out-of-order timeseries point"); + ow.test.assert($ch(name).getKeys({ metric: "cpu", labels: { host: "a" } })[0].ts, 500, "An out-of-order timeseries point must be sorted"); + // shift and pop return and remove the oldest and newest points + ow.test.assert($ch(name).shift().ts, 500, "Problem with timeseries shift"); + ow.test.assert($ch(name).get(K("a", 500)), __, "Shift must remove the oldest timeseries point"); + ow.test.assert($ch(name).pop().ts, 5000, "Problem with timeseries pop"); + $ch(name).set(K("a", 5000), { v: 5 }); + ow.test.assert($ch(name).size(), 7, "Problem with the timeseries size after shift and pop"); + $ch(name).set({ metric: "iso", ts: "1970-01-01T00:00:10.000Z" }, { v: 1 }); + ow.test.assert($ch(name).get({ metric: "iso" }).ts, 10000, "Problem with an ISO timeseries timestamp"); + $ch(name).set({ metric: "iso", ts: new Date(20000) }, { v: 2 }); + ow.test.assert($ch(name).get({ metric: "iso" }).ts, 20000, "Problem with a Date timeseries timestamp"); + $ch(name).unset({ metric: "iso" }); + ow.test.assert($ch(name).get({ metric: "iso" }), __, "Unsetting a timeseries without ts must drop it"); + var failed = false; + try { $ch(name).set(K("a", 6000), { v: "x" }); } catch(e) { failed = true; } + ow.test.assert(failed, true, "A non numeric timeseries value must fail"); + failed = false; + try { $ch(name).set({ labels: {} }, { v: 1 }); } catch(e) { failed = true; } + ow.test.assert(failed, true, "A timeseries key without metric must fail"); + + // Keys, selectors + var ks = $ch(name).getKeys(); + ow.test.assert(ks.length, 7, "Problem with timeseries getKeys"); + ow.test.assert(ks[0].metric, "cpu", "Problem with the timeseries getKeys order"); + ow.test.assert($ch(name).getKeys({ metric: "cpu", labels: { host: "a" }, from: 2000, to: 4000 }).map(k => k.ts), [ 2000, 3000, 4000 ], "Problem with the timeseries getKeys selector"); + ow.test.assert($ch(name).getAll({ metric: "mem" })[0].v, 7, "Problem with the timeseries getAll selector"); + ow.test.assert($ch(name).getSortedKeys()[0].ts, 1000, "Problem with timeseries getSortedKeys"); + ow.test.assert(u.series(name, { metric: "cpu" }).length, 2, "Problem with timeseries series"); + ow.test.assert(u.series(name, { metric: /^c/, labels: { host: [ "a", "z" ] } }).length, 1, "Problem with timeseries series regexp/array selector"); + ow.test.assert(u.labels(name, "cpu"), { host: [ "a", "b" ] }, "Problem with timeseries labels"); + + // Queries + var r = (sel, o) => u.range(name, sel, o); + var a = { metric: "cpu", labels: { host: "a" } }; + ow.test.assert(r(a, { from: 2000, to: 4000 }).map(x => x.v), [ 2, 30, 4 ], "Problem with raw timeseries range"); + ow.test.assert(r(a)[0], { ts: 1000, metric: "cpu", host: "a", v: 1 }, "Problem with the flat timeseries range row"); + ow.test.assert(r(a, { step: "2s", fn: "avg" }).map(x => [ x.ts, x.v ]), [ [ 0, 1 ], [ 2000, 16 ], [ 4000, 4.5 ] ], "Problem with timeseries range avg"); + ow.test.assert(r(a, { step: 2000, fn: "sum" }).map(x => x.v), [ 1, 32, 9 ], "Problem with timeseries range sum"); + ow.test.assert(r(a, { step: 2000, fn: "min" }).map(x => x.v), [ 1, 2, 4 ], "Problem with timeseries range min"); + ow.test.assert(r(a, { step: 2000, fn: "max" }).map(x => x.v), [ 1, 30, 5 ], "Problem with timeseries range max"); + ow.test.assert(r(a, { step: 2000, fn: "count" }).map(x => x.v), [ 1, 2, 2 ], "Problem with timeseries range count"); + ow.test.assert(r(a, { step: 2000, fn: "first" }).map(x => x.v), [ 1, 2, 4 ], "Problem with timeseries range first"); + ow.test.assert(r(a, { step: 2000, fn: "last" }).map(x => x.v), [ 1, 30, 5 ], "Problem with timeseries range last"); + ow.test.assert(r({ metric: "cpu" }, { step: 10000, fn: "count" }).length, 2, "A timeseries range must return a row per series"); + ow.test.assert(r(a, { from: 0, to: 6000, step: 1000, fn: "last", fill: "zero" }).map(x => x.v), [ 0, 1, 2, 30, 4, 5, 0 ], "Problem with timeseries range fill zero"); + ow.test.assert(r({ metric: "cpu", labels: { host: "b" } }, { from: 0, to: 3000, step: 1000, fn: "last", fill: "prev" }).map(x => x.v), [ null, 10, 10, 10 ], "Problem with timeseries range fill prev"); + failed = false; + try { r(a, { step: 1000, fn: "median" }); } catch(e) { failed = true; } + ow.test.assert(failed, true, "An unknown timeseries fn must fail"); + ow.test.assert(u.latest(name, { metric: "cpu" }).map(x => x.v), [ 5, 10 ], "Problem with timeseries latest"); + + // Counters: a decrease is a reset + [ [ 0, 0 ], [ 1000, 10 ], [ 2000, 20 ], [ 3000, 5 ], [ 4000, 15 ], [ 5000, 25 ] ].forEach(p => $ch(name).set({ metric: "reqs", ts: p[0] }, { v: p[1] })); + ow.test.assert(r({ metric: "reqs" }, { step: 2000, fn: "increase" }).map(x => x.v), [ 10, 15, 20 ], "Problem with timeseries increase"); + ow.test.assert(r({ metric: "reqs" }, { step: 2000, fn: "rate" }).map(x => x.v), [ 5, 7.5, 10 ], "Problem with timeseries rate"); + + // setAll, stats, drop + $ch(name).setAll([ "metric", "labels", "ts" ], [ { metric: "disk", labels: { d: "x" }, ts: 1000, v: 1 }, { metric: "disk", labels: { d: "x" }, ts: 2000, v: 2 }, { metric: "disk", labels: { d: "y" }, ts: 1000, v: 3 } ]); + var st = u.stats(name); + ow.test.assert(st.series, 6, "Problem with timeseries stats series"); + ow.test.assert(st.metrics.disk.points, 3, "Problem with timeseries stats points"); + ow.test.assert(st.metrics.disk.last, 2000, "Problem with timeseries stats last"); + ow.test.assert(u.drop(name, { metric: "disk" }), 2, "Problem with timeseries drop"); + $ch(name).unsetAll([ "metric", "labels", "ts" ], [ { metric: "reqs", ts: 0 }, { metric: "reqs", ts: 1000 } ]); + ow.test.assert($ch(name).getKeys({ metric: "reqs" }).length, 4, "Problem with timeseries unsetAll"); + $ch(name).unset({ metric: "reqs" }); + $ch(name).unset({ metric: "mem" }); + ow.test.assert($ch(name).size(), 6, "Problem with timeseries size after drops"); + }; + + var __tsRetention = function(name, opts) { + // retention is measured from the newest point of the series + $ch(name).create(true, "timeseries", merge({ retention: "2s", housekeepEveryMs: 0 }, opts)); + for (var i = 1; i <= 10; i++) $ch(name).set({ metric: "m", ts: i * 1000 }, { v: i }); + ow.test.assert($ch(name).getKeys().map(k => k.ts), [ 8000, 9000, 10000 ], "Problem with timeseries retention"); + $ch(name).set({ metric: "m", ts: 1000 }, { v: 1 }); + ow.test.assert($ch(name).size(), 3, "A point older than the retention must be dropped"); + if (isDef(opts.file)) { + $ch(name).destroy(); + $ch(name).create(false, "timeseries", merge({ retention: "2s" }, opts)); + ow.test.assert($ch(name).getKeys().map(k => k.ts), [ 8000, 9000, 10000 ], "Retention must hold after reloading a timeseries file"); + } + $ch(name).destroy(); + }; + + var __tsMaxPoints = function(name, opts) { + $ch(name).create(true, "timeseries", merge({ maxPoints: 2 }, opts)); + for (var i = 1; i <= 5; i++) $ch(name).set({ metric: "m", ts: i * 1000 }, { v: i }); + $ch(name).set({ metric: "n", ts: 1000 }, { v: 1 }); + ow.test.assert($ch(name).getKeys({ metric: "m" }).map(k => k.ts), [ 4000, 5000 ], "Problem with timeseries maxPoints"); + ow.test.assert($ch(name).size(), 3, "maxPoints must apply per series"); + $ch(name).destroy(); + }; + + var __tsRollups = function(name, opts) { + var u = ow.ch.utils.timeseries; + // raw: 3s, 1s buckets: 5s, 10s buckets: forever + $ch(name).create(true, "timeseries", merge({ raw: { keep: "3s" }, rollups: [ { step: "10s" }, { step: "1s", keep: "5s" } ], housekeepEveryMs: 0 }, opts)); + for (var i = 0; i < 20; i++) $ch(name).set({ metric: "m", labels: { h: "a" }, ts: i * 1000 }, { v: i }); + var s = { metric: "m" }; + ow.test.assert($ch(name).getKeys().map(k => k.ts), [ 16000, 17000, 18000, 19000 ], "Problem with the raw retention of a timeseries with rollups"); + var st = u.stats(name); + ow.test.assert(st.tiers.r1000.entries, 6, "Problem with the rollup retention"); + ow.test.assert(st.tiers.r10000.entries, 2, "Problem counting the rollup buckets"); + ow.test.assert(u.range(name, s, { step: "10s", fn: "avg" }).map(x => [ x.ts, x.v ]), [ [ 0, 4.5 ], [ 10000, 14.5 ] ], "Problem reading rollups (avg)"); + ow.test.assert(u.range(name, s, { step: "10s", fn: "min", tier: "10s" }).map(x => x.v), [ 0, 10 ], "Problem reading rollups (min)"); + ow.test.assert(u.range(name, s, { step: "10s", fn: "max", tier: "10s" }).map(x => x.v), [ 9, 19 ], "Problem reading rollups (max)"); + ow.test.assert(u.range(name, s, { step: "10s", fn: "sum", tier: "10s" }).map(x => x.v), [ 45, 145 ], "Problem reading rollups (sum)"); + ow.test.assert(u.range(name, s, { step: "10s", fn: "first", tier: "10s" }).map(x => x.v), [ 0, 10 ], "Problem reading rollups (first)"); + ow.test.assert(u.range(name, s, { step: "10s", fn: "last", tier: "10s" }).map(x => x.v), [ 9, 19 ], "Problem reading rollups (last)"); + ow.test.assert(u.range(name, s, { step: "10s", fn: "increase", tier: "10s" }).map(x => x.v), [ 9, 10 ], "Problem reading rollups (increase)"); + ow.test.assert(u.range(name, s, { fn: "count", tier: "10s" }).map(x => [ x.ts, x.v ]), [ [ 0, 10 ], [ 10000, 10 ] ], "Problem reading a rollup tier without step"); + ow.test.assert(u.range(name, s, { from: 16000, step: "1s", fn: "last" }).map(x => x.v), [ 16, 17, 18, 19 ], "The finest tier must be used when it covers 'from'"); + ow.test.assert(u.range(name, s, { from: 14000, step: "1s", fn: "last" }).map(x => x.v), [ 14, 15, 16, 17, 18, 19 ], "The 1s rollup must cover what raw dropped"); + // a late point inside an expired 1s bucket only reaches the 10s tier + $ch(name).set({ metric: "m", labels: { h: "a" }, ts: 1000 }, { v: 100 }); + ow.test.assert(u.range(name, s, { step: "10s", fn: "count", tier: "10s" }).map(x => x.v), [ 11, 10 ], "Problem with a late point on rollups"); + ow.test.assert(u.stats(name).tiers.r1000.entries, 6, "A late point must not recreate an expired rollup bucket"); + ow.test.assert($ch(name).size(), 4, "A late point must not stay in the raw tier"); + $ch(name).destroy(); + + // Overwriting or removing a point fixes the buckets when raw is complete + $ch(name).create(true, "timeseries", merge({ rollups: [ { step: "10s" } ] }, opts)); + s = { metric: "o" }; + [ [ 0, 1 ], [ 1000, 5 ], [ 2000, 3 ] ].forEach(p => $ch(name).set({ metric: "o", ts: p[0] }, { v: p[1] })); + var q = f => u.range(name, s, { step: "10s", fn: f, tier: "10s" })[0].v; + ow.test.assert([ q("count"), q("sum"), q("min"), q("max") ], [ 3, 9, 1, 5 ], "Problem with rollup aggregates"); + $ch(name).set({ metric: "o", ts: 1000 }, { v: 2 }); + ow.test.assert([ q("count"), q("sum"), q("min"), q("max") ], [ 3, 6, 1, 3 ], "Overwriting a point must fix the rollup bucket"); + $ch(name).unset({ metric: "o", ts: 2000 }); + ow.test.assert([ q("count"), q("sum"), q("min"), q("max"), q("last") ], [ 2, 3, 1, 2, 2 ], "Removing a point must fix the rollup bucket"); + $ch(name).unset({ metric: "o", ts: 0 }); + $ch(name).unset({ metric: "o", ts: 1000 }); + ow.test.assert(u.range(name, s, { step: "10s", tier: "10s" }), [], "An empty rollup bucket must be removed"); + // rebuild after rollups were not there + [ [ 0, 1 ], [ 1000, 5 ] ].forEach(p => $ch(name).set({ metric: "o", ts: p[0] }, { v: p[1] })); + u.rebuildRollups(name); + ow.test.assert(q("sum"), 6, "Problem rebuilding rollups"); + $ch(name).destroy(); + }; + + exports.testTimeSeriesChannel = function() { + var dir = String(java.nio.file.Files.createTempDirectory("openaf-ts-")); + var file = dir + "/ts.ndjson"; + var name = "ts-test-" + genUUID(); + try { + // memory only + $ch(name).create(true, "timeseries", {}); + __tsCommon(name); + $ch(name).destroy(); + ow.test.assert(io.fileExists(file), false, "A memory timeseries must not write files"); + + // file + $ch(name).create(true, "timeseries", { file: file }); + __tsCommon(name); + var before = $ch(name).getAll(); + $ch(name).destroy(); + $ch(name).create(false, "timeseries", { file: file }); + ow.test.assert($ch(name).getAll(), before, "Problem reloading a timeseries file"); + ow.test.assert($ch(name).size(), 6, "Problem with the size of a reloaded timeseries"); + + // a truncated last line must not break loading or the next append + $ch(name).destroy(); + io.writeFileString(file, "{\"o\":\"p\",\"s\":", __, true); + $ch(name).create(false, "timeseries", { file: file }); + ow.test.assert($ch(name).size(), 6, "A truncated timeseries line must be ignored"); + $ch(name).set({ metric: "after", ts: 1 }, { v: 1 }); + $ch(name).destroy(); + $ch(name).create(false, "timeseries", { file: file }); + ow.test.assert($ch(name).get({ metric: "after" }).v, 1, "Problem appending after a truncated timeseries line"); + $ch(name).destroy(); + + // compaction: overwrites and deletes are dead lines + io.rm(file); + $ch(name).create(true, "timeseries", { file: file, compactRatio: 0.3 }); + for (var i = 0; i < 20; i++) $ch(name).set({ metric: "c", ts: 1000 }, { v: i }); + for (var i = 0; i < 10; i++) { $ch(name).set({ metric: "d", ts: i }, { v: i }); $ch(name).unset({ metric: "d", ts: i }); } + var lines = () => io.readFileString(file).split("\n").filter(l => l.length > 0).length; + ow.test.assert(lines() > 30, true, "The timeseries file must be append-only"); + ow.test.assert($ch(name).size(), 1, "Problem with a timeseries with dead lines"); + $ch(name).destroy(); + ow.test.assert(lines() <= 3, true, "Destroying a timeseries channel must compact a mostly dead file"); + $ch(name).create(false, "timeseries", { file: file }); + ow.test.assert($ch(name).get({ metric: "c" }).v, 19, "Problem reading a compacted timeseries file"); + ow.test.assert(ow.ch.utils.timeseries.compact(name), true, "Problem compacting a timeseries on request"); + $ch(name).destroy(); + + // read-only + $ch(name).create(false, "timeseries", { file: file, readOnly: true }); + var failed = false; + try { $ch(name).set({ metric: "z", ts: 1 }, { v: 1 }); } catch(e) { failed = true; } + ow.test.assert(failed, true, "Read-only timeseries must reject changes"); + ow.test.assert($ch(name).size(), 1, "Read-only timeseries must keep its content"); + $ch(name).destroy(); + failed = false; + try { $ch(name).create(true, "timeseries", { file: dir + "/none.ndjson", create: false }); } catch(e) { failed = true; } + ow.test.assert(failed, true, "A missing timeseries file with create=false must fail"); + + // debounced autosave flushes on destroy + io.rm(file); + $ch(name).create(true, "timeseries", { file: file, autosave: "debounced", saveDebounceMs: 3600000 }); + $ch(name).set({ metric: "late", ts: 1 }, { v: 1 }); + $ch(name).set({ metric: "later", ts: 1 }, { v: 1 }); + ow.test.assert(io.fileExists(file) && io.readFileString(file).indexOf("later") >= 0, false, "Debounced timeseries must not append immediately"); + $ch(name).destroy(); + ow.test.assert(io.readFileString(file).indexOf("later") >= 0, true, "Debounced timeseries must append on destroy"); + + // retention, count limit and rollups (also persisted and reloaded) + io.rm(file); + __tsRetention(name, {}); + __tsMaxPoints(name, {}); + __tsRollups(name, {}); + io.rm(file); + __tsRetention(name, { file: file }); + io.rm(file); + __tsRollups(name, { file: file }); + io.rm(file); + var ro = { file: file, rollups: [ { step: "10s" } ] }; + $ch(name).create(true, "timeseries", ro); + [ [ 0, 1 ], [ 1000, 5 ] ].forEach(p => $ch(name).set({ metric: "m", ts: p[0] }, { v: p[1] })); + $ch(name).destroy(); + $ch(name).create(false, "timeseries", ro); + var sum10 = () => ow.ch.utils.timeseries.range(name, { metric: "m" }, { step: "10s", fn: "sum", tier: "10s" }).map(x => x.v); + ow.test.assert(sum10(), [ 6 ], "Problem persisting rollups"); + $ch(name).unset({ metric: "m", ts: 1000 }); + $ch(name).destroy(); + $ch(name).create(false, "timeseries", ro); + ow.test.assert(sum10(), [ 1 ], "Problem persisting rollup changes"); + $ch(name).destroy(); + + // retention measured from now and housekeep on demand + $ch(name).create(true, "timeseries", { retention: "1h", retentionRef: "now", housekeepEveryMs: 3600000 }); + $ch(name).set({ metric: "new", ts: now() }, { v: 1 }); + $ch(name).set({ metric: "old", ts: 1000 }, { v: 1 }); + ow.test.assert(ow.ch.utils.timeseries.housekeep(name), 1, "Problem with timeseries housekeep"); + ow.test.assert($ch(name).getKeys().map(k => k.metric), [ "new" ], "Housekeep must drop what is older than the retention from now"); + $ch(name).destroy(); + } finally { + $ch(name).destroy(); + io.rm(dir); + } + }; + + exports.testTimeSeriesChannelStore = function() { + var dir = String(java.nio.file.Files.createTempDirectory("openaf-ts-store-")); + var name = "ts-store-test-" + genUUID(); + var u = ow.ch.utils.timeseries; + var so = f => ({ store: { type: "mvs", options: { file: dir + "/" + f + ".db" } }, chunk: "2s" }); + var opts = so("ts"); + try { + $ch(name).create(true, "timeseries", opts); + __tsCommon(name); + var before = $ch(name).getAll(); + $ch(name).destroy(); + $ch(name).create(false, "timeseries", opts); + ow.test.assert($ch(name).getAll(), before, "Problem reopening a timeseries store"); + $ch(name).destroy(); + + var failed = false; + try { $ch(name).create(true, "timeseries", { create: false, store: { type: "mvs", options: { file: dir + "/empty.db" } } }); } catch(e) { failed = true; } + ow.test.assert(failed, true, "A missing timeseries store with create=false must fail"); + failed = false; + try { $ch(name).create(true, "timeseries", { file: dir + "/x.ndjson", store: { type: "mvs" } }); } catch(e) { failed = true; } + ow.test.assert(failed, true, "A timeseries can't use file and store together"); + + // retention, count limit and rollups over chunks + __tsRetention(name, so("ret")); + __tsMaxPoints(name, so("max")); + __tsRollups(name, so("roll")); + + // many points across many chunks in one setAll + $ch(name).create(true, "timeseries", { store: { type: "mvs", options: { file: dir + "/b.db" } }, chunk: "1s" }); + var vals = []; + for (var i = 0; i < 500; i++) vals.push({ metric: "big", labels: { n: "1" }, ts: i * 100, v: i }); + $ch(name).setAll([ "metric", "labels", "ts" ], vals); + ow.test.assert($ch(name).size(), 500, "Problem with a timeseries store setAll"); + ow.test.assert(u.range(name, { metric: "big" }, { from: 9500, to: 10499 }).map(x => x.v), [ 95, 96, 97, 98, 99, 100, 101, 102, 103, 104 ], "Problem reading a timeseries store across chunks"); + ow.test.assert(u.range(name, { metric: "big" }, { from: 9500, to: 10499, step: "1s", fn: "count" }).map(x => x.v), [ 5, 5 ], "Problem with a timeseries store range"); + $ch(name).destroy(); + } finally { + try { $ch(name).destroy(); } catch(e) {} + io.rm(dir); + } + }; + + exports.testTimeSeriesIntegrations = function() { + var u = ow.ch.utils.timeseries; + var name = "ts-int-" + genUUID(), src = "ts-src-" + genUUID(), name2 = "ts-int2-" + genUUID(); + try { + $ch(name).create(true, "timeseries", {}); + + // channel subscriber + $ch(src).create(); + $ch(src).subscribe(u.getTimeSeriesSubscriber(name, { metric: "temp", labels: [ "room" ], value: "t", ts: "when" })); + $ch(src).set({ id: 1 }, { id: 1, room: "kitchen", t: 21.5, when: 1000 }); + $ch(src).set({ id: 2 }, { id: 2, room: "kitchen", t: 22, when: 2000 }); + $ch(src).setAll([ "id" ], [ { id: 3, room: "hall", t: 18, when: 2000 } ]); + $ch(src).waitForJobs(15000); + ow.test.assert($ch(name).get({ metric: "temp", labels: { room: "kitchen" } }).v, 22, "Problem with the timeseries subscriber (set)"); + ow.test.assert($ch(name).get({ metric: "temp", labels: { room: "hall" }, ts: 2000 }).v, 18, "Problem with the timeseries subscriber (setAll)"); + ow.test.assert($ch(name).size(), 3, "Problem with the timeseries subscriber size"); + $ch(src).destroy(); + u.drop(name, {}); + + // multiple value fields + $ch(src).create(); + $ch(src).subscribe(u.getTimeSeriesSubscriber(name, { metric: "cpu", labels: [ "host" ], values: [ "user", "sys" ], ts: "when" })); + $ch(src).set({ id: 1 }, { id: 1, host: "a", user: 0.3, sys: 0.1, when: 5000 }); + $ch(src).waitForJobs(15000); + ow.test.assert(u.series(name, {}).map(s => s.metric).sort(), [ "cpu_sys", "cpu_user" ], "Problem with the timeseries subscriber (values)"); + $ch(src).destroy(); + u.drop(name, {}); + + // collector + var col = u.collect(__, name, () => ({ jvm: { mem: { used: 10, free: 5 }, name: "x" }, up: true, cpus: [ 1, 2 ] }), { labels: { app: "t" }, prefix: "oaf_" }); + ow.test.assert(col.once(), 5, "Problem with the timeseries collector"); + ow.test.assert(u.series(name, { metric: "oaf_jvm_mem_used" }).length, 1, "Problem with the timeseries collector metric names"); + ow.test.assert(u.latest(name, { metric: "oaf_up" })[0].v, 1, "Problem with the timeseries collector booleans"); + ow.test.assert(u.latest(name, { metric: "oaf_cpus", labels: { idx: "1" } })[0].v, 2, "Problem with the timeseries collector arrays"); + ow.test.assert(u.latest(name, { metric: "oaf_up" })[0].app, "t", "Problem with the timeseries collector labels"); + u.drop(name, {}); + var n = 0; + var col2 = u.collect(50, name, () => [ { metric: "tick", labels: { k: "v" }, v: ++n } ]); + var tries = 0; + while($ch(name).size() < 2 && tries++ < 100) sleep(50); + col2.stop(); + ow.test.assert($ch(name).size() >= 2, true, "The periodic timeseries collector must record points"); + var sz = $ch(name).size(); + sleep(200); + ow.test.assert($ch(name).size() <= sz + 1, true, "A stopped timeseries collector must stop recording"); + u.drop(name, {}); + + // OpenMetrics + $ch(name).set({ metric: "http_requests_total", labels: { code: "200", path: "/a,b \"q\"" }, ts: 1500 }, { v: 10 }); + $ch(name).set({ metric: "http_requests_total", labels: { code: "500", path: "/x" }, ts: 1500 }, { v: 2 }); + $ch(name).set({ metric: "http_requests_total", labels: { code: "500", path: "/x" }, ts: 2500 }, { v: 3 }); + $ch(name).set({ metric: "temp.c", ts: 1500 }, { v: 1.5 }); + var txt = u.toOpenMetrics(name, {}, { help: { http_requests_total: { help: "Requests", type: "counter" } } }); + ow.test.assert(txt.indexOf("# TYPE http_requests_total counter") >= 0, true, "Problem with the OpenMetrics type line"); + ow.test.assert(txt.indexOf("http_requests_total{code=\"500\",path=\"/x\"} 3 2.5") >= 0, true, "Problem with the OpenMetrics latest line"); + ow.test.assert(txt.indexOf("temp_c 1.5 1.5") >= 0, true, "Problem with the OpenMetrics metric name"); + ow.test.assert(u.toOpenMetrics(name, { metric: "temp.c" }, { timestamps: false }), "temp_c 1.5\n", "Problem with OpenMetrics without timestamps"); + ow.test.assert(u.toOpenMetrics(name, { metric: "http_requests_total", labels: { code: "500" } }, { from: 0 }).split("\n").filter(l => l.length > 0).length, 2, "Problem exporting all points as OpenMetrics"); + $ch(name2).create(true, "timeseries", {}); + ow.test.assert(u.fromOpenMetrics(name2, u.toOpenMetrics(name, {}, { from: 0 }), { labels: { src: "t" } }), 4, "Problem importing OpenMetrics"); + ow.test.assert($ch(name2).get({ metric: "http_requests_total", labels: { code: "200", path: "/a,b \"q\"", src: "t" }, ts: 1500 }).v, 10, "Problem importing OpenMetrics labels"); + ow.test.assert($ch(name2).get({ metric: "http_requests_total", labels: { code: "500", path: "/x", src: "t" } }).ts, 2500, "Problem importing OpenMetrics timestamps"); + ow.test.assert(u.fromOpenMetrics(name2, "# comment\nfoo 4\nbar{a=\"1\"} 5 7", { ts: 100 }), 2, "Problem importing OpenMetrics without timestamps"); + ow.test.assert($ch(name2).get({ metric: "foo" }).ts, 100, "Problem with the default OpenMetrics timestamp"); + ow.test.assert($ch(name2).get({ metric: "bar", labels: { a: "1" } }).ts, 7000, "Problem with the OpenMetrics timestamp unit"); + } finally { + [ name, name2, src ].forEach(n => { try { $ch(n).destroy(); } catch(e) {} }); + } + }; + + exports.testTimeSeriesConcurrency = function() { + var u = ow.ch.utils.timeseries; + var name = "ts-conc-" + genUUID(); + var errors = []; + try { + $ch(name).create(true, "timeseries", { rollups: [ { step: "1s" } ] }); + plugin("Threads"); + var N = 1500; + var th = new Threads(); + [ 0, 1 ].forEach(function(w) { + th.addThread(function() { + try { for (var i = 0; i < N; i++) $ch(name).set({ metric: "s", labels: { k: "1" }, ts: 1000 + i * 2 + w }, { v: i }); } catch(e) { errors.push("w" + w + ": " + e); } + }); + }); + th.addThread(function() { + try { + for (var i = 0; i < 100; i++) { + u.range(name, { metric: "s" }, { step: "1s", fn: "avg" }); + u.latest(name, {}); + $ch(name).getKeys(); + u.stats(name); + } + } catch(e) { errors.push("reader: " + e); } + }); + th.addThread(function() { + try { for (var i = 0; i < 300; i++) $ch(name).setAll([ "metric", "labels", "ts" ], [ { metric: "b", ts: i * 10, v: i }, { metric: "b", ts: i * 10 + 1, v: i } ]); } catch(e) { errors.push("batch: " + e); } + }); + th.start(); + th.stop(); + ow.test.assert(errors, [], "Concurrent timeseries access failed"); + ow.test.assert($ch(name).size(), N * 2 + 600, "Concurrent timeseries writes lost points"); + var ts = $ch(name).getKeys({ metric: "s" }).map(k => k.ts); + var sorted = true; + for (var i = 1; i < ts.length; i++) if (ts[i] <= ts[i - 1]) sorted = false; + ow.test.assert(sorted, true, "Concurrent timeseries writes broke the point order"); + var tot = u.range(name, { metric: "s" }, { step: "1d", fn: "count", tier: "1s" }).reduce((a, r) => a + r.v, 0); + ow.test.assert(tot, N * 2, "Concurrent timeseries writes broke the rollups"); + } finally { + try { $ch(name).destroy(); } catch(e) {} + } + }; + exports.testAll = function() { $ch("allA").destroy(); $ch("allB").destroy(); diff --git a/tests/autoTestAll.Channels.yaml b/tests/autoTestAll.Channels.yaml index 8d19c888..6ae10387 100644 --- a/tests/autoTestAll.Channels.yaml +++ b/tests/autoTestAll.Channels.yaml @@ -120,7 +120,37 @@ jobs: - name: Channels::Test MVS Utils from: Channels::Init to : oJob Test - exec: args.func = args.tests.testMVSUtils; + exec: args.func = args.tests.testMVSUtils; + + - name: Channels::MVS multifile basic + from: Channels::Init + to : oJob Test + exec: args.func = args.tests.testMVSMultifile; + + - name: Channels::MVS multifile manifest mismatch + from: Channels::Init + to : oJob Test + exec: args.func = args.tests.testMVSMultifileManifestMismatch; + + - name: Channels::MVS multifile pagination + from: Channels::Init + to : oJob Test + exec: args.func = args.tests.testMVSMultifilePagination; + + - name: Channels::MVS multifile sharing + from: Channels::Init + to : oJob Test + exec: args.func = args.tests.testMVSMultifileSharing; + + - name: Channels::MVS multifile reshard + from: Channels::Init + to : oJob Test + exec: args.func = args.tests.testMVSMultifileReshard; + + - name: Channels::MVS shift and pop + from: Channels::Init + to : oJob Test + exec: args.func = args.tests.testMVSShiftPop; - name: Channels::Graph channel from: Channels::Init @@ -132,6 +162,26 @@ jobs: to : oJob Test exec: args.func = args.tests.testGraphChannelStore; + - name: Channels::TimeSeries channel + from: Channels::Init + to : oJob Test + exec: args.func = args.tests.testTimeSeriesChannel; + + - name: Channels::TimeSeries channel store + from: Channels::Init + to : oJob Test + exec: args.func = args.tests.testTimeSeriesChannelStore; + + - name: Channels::TimeSeries integrations + from: Channels::Init + to : oJob Test + exec: args.func = args.tests.testTimeSeriesIntegrations; + + - name: Channels::TimeSeries concurrency + from: Channels::Init + to : oJob Test + exec: args.func = args.tests.testTimeSeriesConcurrency; + - name: Channels::Test channel (all) from: Channels::Init to : oJob Test @@ -198,6 +248,12 @@ todo: - Channels::Remote channel access auditing - Channels::Subscribers test - Channels::Test MVS Utils + - Channels::MVS multifile basic + - Channels::MVS multifile manifest mismatch + - Channels::MVS multifile pagination + - Channels::MVS multifile sharing + - Channels::MVS multifile reshard + - Channels::MVS shift and pop - Channels::Keep history util # Big @@ -221,6 +277,12 @@ todo: - Channels::Graph channel - Channels::Graph channel store + # TimeSeries + - Channels::TimeSeries channel + - Channels::TimeSeries channel store + - Channels::TimeSeries integrations + - Channels::TimeSeries concurrency + # Ignite #- name: Channels::Creating a channel # args: From 94e1db36dfe66fe9bd15d438e5254681d2ee5a16 Mon Sep 17 00:00:00 2001 From: nmaguiar Date: Fri, 16 Oct 2026 07:36:41 +0100 Subject: [PATCH 05/19] test(metrics): add autoTestAll.Metrics suite for owrap.metrics Add tests/autoTestAll.Metrics.{js,yaml} with focused coverage for the owrap.metrics module and register the new job in tests/autoTestAll.allJobs.yaml. Extend js/owrap.metrics.js with the minor changes needed to support the new test cases. --- js/owrap.metrics.js | 17 +++++++-- tests/autoTestAll.Metrics.js | 64 ++++++++++++++++++++++++++++++++++ tests/autoTestAll.Metrics.yaml | 26 ++++++++++++++ tests/autoTestAll.allJobs.yaml | 1 + 4 files changed, 106 insertions(+), 2 deletions(-) create mode 100644 tests/autoTestAll.Metrics.js create mode 100644 tests/autoTestAll.Metrics.yaml diff --git a/js/owrap.metrics.js b/js/owrap.metrics.js index a23174f0..774f1373 100644 --- a/js/owrap.metrics.js +++ b/js/owrap.metrics.js @@ -451,10 +451,23 @@ OpenWrap.metrics.prototype.fromOpenMetrics2Array = function(lines) { // Turn into a map of entries var dr = {}; var d = lines.filter(line => !line.startsWith("#") && line.length > 0).map(line => { - var res = line.match(/^([^\{]+?)({[^\}]+})? ([^ ]+?)( [^ ]+?)?$/); + // Braces and separators inside quoted values are label contents. + var res = line.match(/^([^\{\s]+)(?:\{((?:[^"{}]|"(?:[^"\\]|\\.)*")*)\})? ([^ ]+?)( [^ ]+?)?$/); + if (isNull(res)) throw new Error("Invalid OpenMetrics sample: " + line); + var labels = {}, rest = res[2] || ""; + while (rest.trim().length > 0) { + var label = rest.match(/^\s*([a-zA-Z_][a-zA-Z0-9_]*)\s*=\s*"((?:[^"\\]|\\[\\"n])*)"\s*(,|$)/); + if (isNull(label)) throw new Error("Invalid OpenMetrics labels: " + line); + // Decode once so a literal backslash followed by n stays literal. + var value = label[2].replace(/\\([\\"n])/g, function(match, escaped) { + return escaped == "n" ? "\n" : escaped; + }); + Object.defineProperty(labels, label[1], { value: value, enumerable: true, writable: true, configurable: true }); + rest = rest.substring(label[0].length); + } var data = { metric: res[1], - labels: (isNull(res[2]) ? {} : jsonParse(res[2].replace(/([^=,\{\}]+)=([^=,\{\}]+)/g, "\"$1\":$2"), true)), + labels: labels, value: Number(res[3]), timestamp: (isNumber(res[4]) ? Number(res[4]) : __) } diff --git a/tests/autoTestAll.Metrics.js b/tests/autoTestAll.Metrics.js new file mode 100644 index 00000000..bd290a6a --- /dev/null +++ b/tests/autoTestAll.Metrics.js @@ -0,0 +1,64 @@ +(function() { + exports.testOpenMetricsLabels = function() { + ow.loadMetrics(); + var expected = { + comma: "a,b=c", + quote: 'say "hello", x=y', + braces: "a}b{c", + empty: "", + path: "C:\\tmp\\file", + newline: "first\nsecond", + literal: "first\\nsecond", + ending: "last\\", + unicode: "café 中文" + }; + var labels = Object.keys(expected).map(function(key) { + return key + '="' + expected[key].replace(/\\/g, "\\\\").replace(/"/g, '\\"').replace(/\n/g, "\\n") + '"'; + }).join(","); + var line = "sample_total{" + labels + "} 12.5"; + [line, [line]].forEach(function(input) { + var result = ow.metrics.fromOpenMetrics2Array(input); + ow.test.assert(result.length, 1, "One sample expected"); + ow.test.assert(result[0].labels, expected, "Quoted label contents must survive parsing"); + ow.test.assert(result[0].value, 12.5, "Sample value preserved"); + }); + var special = ow.metrics.fromOpenMetrics2Array('sample{__proto__="value"} 1')[0].labels; + ow.test.assert(Object.prototype.hasOwnProperty.call(special, "__proto__"), true, "Label must be an own property"); + ow.test.assert(special.__proto__, "value", "Special label name preserved"); + }; + + exports.testOpenMetricsCompatibility = function() { + ow.loadMetrics(); + var result = ow.metrics.fromOpenMetrics2Array('# HELP sample A sample\n\nsample_total{code="200", path="/ok"} 3 123\nsample_count 4\nempty{} 0'); + ow.test.assert(result.length, 3, "Comments and blank lines ignored"); + result = ["sample_total", "sample_count", "empty"].map(function(name) { + return result.filter(function(sample) { return sample.metric == name; })[0]; + }); + ow.test.assert(result[0].metric, "sample_total", "Metric name preserved"); + ow.test.assert(result[0].labels, { code: "200", path: "/ok" }, "Ordinary labels preserved"); + ow.test.assert(result[0].value, 3, "Numeric conversion preserved"); + ow.test.assert(result[0].timestamp, 123, "Timestamp preserved"); + ow.test.assert(isUnDef(result[1].timestamp), true, "Absent timestamp preserved"); + ow.test.assert(result[0].prefix, "sample", "Prefix inference preserved"); + ow.test.assert(result[1].labels, {}, "Unlabeled sample"); + ow.test.assert(result[2].labels, {}, "Empty label set"); + }; + + exports.testOpenMetricsInvalidLabels = function() { + ow.loadMetrics(); + [ + 'sample{a="unterminated} 1', + 'sample{a="bad "quote""} 1', + 'sample{a="bad\\q"} 1', + 'sample{a=unquoted} 1', + 'sample{a="x" b="y"} 1', + 'sample{a="x",,b="y"} 1' + ].forEach(function(line) { + var rejected = false; + try { ow.metrics.fromOpenMetrics2Array(line); } catch(e) { + rejected = /Invalid OpenMetrics (sample|labels):/.test(String(e)); + } + ow.test.assert(rejected, true, "Malformed labels must produce a clear error: " + line); + }); + }; +})(); diff --git a/tests/autoTestAll.Metrics.yaml b/tests/autoTestAll.Metrics.yaml new file mode 100644 index 00000000..aa1236f0 --- /dev/null +++ b/tests/autoTestAll.Metrics.yaml @@ -0,0 +1,26 @@ +include: +- oJobTest.yaml + +jobs: +- name: Metrics::Init + exec: args.tests = require("autoTestAll.Metrics.js"); + +- name: Metrics::Quoted labels + from: Metrics::Init + to: oJob Test + exec: args.func = args.tests.testOpenMetricsLabels; + +- name: Metrics::Compatibility + from: Metrics::Init + to: oJob Test + exec: args.func = args.tests.testOpenMetricsCompatibility; + +- name: Metrics::Invalid labels + from: Metrics::Init + to: oJob Test + exec: args.func = args.tests.testOpenMetricsInvalidLabels; + +todo: +- Metrics::Quoted labels +- Metrics::Compatibility +- Metrics::Invalid labels diff --git a/tests/autoTestAll.allJobs.yaml b/tests/autoTestAll.allJobs.yaml index bbc9a90d..858b2f41 100644 --- a/tests/autoTestAll.allJobs.yaml +++ b/tests/autoTestAll.allJobs.yaml @@ -37,3 +37,4 @@ include: # - autoTestAll.SNMP.yaml // Test servers don't work - autoTestAll.Instrumentation.yaml +- autoTestAll.Metrics.yaml From e39f8318842b5050358e928eec0ba62cea79d486 Mon Sep 17 00:00:00 2001 From: nmaguiar Date: Fri, 16 Oct 2026 07:53:40 +0100 Subject: [PATCH 06/19] feat(channels): implement channel timeseries functionality - Update documentation in docs/channel-timeseries.md - Implement logic in js/owrap.ch.js - Add comprehensive tests in tests/autoTestAll.Channels.js and .yaml --- docs/channel-timeseries.md | 21 ++++++++-- js/owrap.ch.js | 51 +++++++++++++++++++++++ tests/autoTestAll.Channels.js | 71 +++++++++++++++++++++++++++++++++ tests/autoTestAll.Channels.yaml | 6 +++ 4 files changed, 146 insertions(+), 3 deletions(-) diff --git a/docs/channel-timeseries.md b/docs/channel-timeseries.md index 47593beb..51f43b51 100644 --- a/docs/channel-timeseries.md +++ b/docs/channel-timeseries.md @@ -199,19 +199,34 @@ $ch("events").set({ id: 1 }, { id: 1, room: "kitchen", t: 21.5, when: 1000 }); ```javascript var txt = ow.ch.utils.timeseries.toOpenMetrics("m", { metric: /^cpu/ }); // cpu{host="a"} 0.6 2 -// POST txt to a Prometheus push gateway: -$rest().post("http://pushgw:9091/metrics/job/demo", txt); $ch("copy").create(true, "timeseries", {}); ow.ch.utils.timeseries.fromOpenMetrics("copy", txt); ``` +### Push to a Prometheus push gateway + +`pushPrometheus(aName, aPushGwUrl, aJob, aSelector, aOptions)` sends the latest point of each matching series to a [Pushgateway](https://github.com/prometheus/pushgateway) under the job `aJob`. Series are grouped by the values of `groupBy` labels: each distinct combination is one request to `/metrics/job//