Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 23 additions & 5 deletions docs/data-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -1887,17 +1887,35 @@ An import reads the file as the pod is read, with the same mappers:
an IRI that may be short. A file this app exports states short
namespaces with a scheme;
- a JSON-LD term, `@base` or `@vocab` is refused (`deckFileUnreadable`)
when it's a compact IRI whose prefix, the part before its first `:`,
is a term of the node's contexts. The parser expands such a term
through the other with no bound on how deep, so two terms defined
through each other, or one through itself, never end. A file this
when it's another term of the node's contexts, or a compact IRI whose
prefix, the part before its first `:`, is one. The parser expands
such a term through the other with no bound on how deep, so two terms
defined through each other, or one through itself, never end. A term
defined as one that names a keyword names that keyword too, which the
checks below would read as a plain term. A file this
app exports states each namespace in full. With the base bounded and
each term defined in full, at most 512 characters, the limits below
bound the time;
- the parser reads the document as it streams, each object written
again with its context first. A parser that held every node to the
end would match each against every typed one, in time growing with
the square of the nodes;
the square of the nodes. It gets the text in parts of 4,096
characters, each in a task of its own, and no more after its first
error, so a parse that fails leaves the tab free after the part it
failed in;
- a JSON-LD keyword given a value the parser reports as an error is
refused (`deckFileUnreadable`) before it's parsed, under an alias
too. These are an `@id` that's no string, or a second one in an
object, and a `@type` that's no string or array of strings. So are an
`@index` that's no string and a `@reverse` that's no object or holds
a keyword. So are an `@included` member that's no node object, a
`@nest` member that's no object or holds `@value` or `@id`, and a
`@list` beside any key but `@index`. A `@set` object's members, and
an array's in it, count as members of the keyword it sits under. The
parser reports such an error and reads on, and each takes it far
longer than a triple, which the bound on the work below doesn't
count. As with the checks below, it covers the whole
document, a `@json` literal too. A file this app exports holds none;
- JSON nested deeper than 12 levels is refused (`deckFileUnreadable`),
before anything else reads it: the JSON-LD parser takes time far
beyond linear in the depth, and a file this app exports nests 5;
Expand Down
3 changes: 2 additions & 1 deletion docs/deck-library.md
Original file line number Diff line number Diff line change
Expand Up @@ -1050,7 +1050,8 @@ flowchart LR
and `@vocab` are refused (`deckFileUnreadable`) past it, counted with
the release's own address. Its terms and prefixes are refused past
their bound or when defined through another term, as a deck file's
are, and so is an RDF-star annotation or triple term
are. So are a keyword given a value the parser reports as an error
and an RDF-star annotation or triple term
([data-model.md](data-model.md#decks-as-files)). A Turtle release
with a triple term, an annotation or a reifier doesn't parse
(`releaseUnreadable`), however it's read: the parser reads none, as
Expand Down
34 changes: 34 additions & 0 deletions packages/solid/src/client/resources.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -196,3 +196,37 @@ describe("the headers that describe a resource, over the library", () => {
expect(handed.every((response) => response.headers.get("Link") === null)).toBe(true);
});
});

describe("parseQuads, of JSON-LD the parser reports errors in", () => {
/**
* The least CPU time, of three parses of `text`, until the main thread
* is free again: the parser may still work after its promise settles.
*/
async function cpuUntilFree(text: string): Promise<number> {
let least = Infinity;
for (let round = 0; round < 3; round++) {
const started = process.cpuUsage();
await resources.parseQuads(text, "jsonld", "https://pod.example/").catch(() => []);
await new Promise((resolve) => setTimeout(resolve, 0));
const { user, system } = process.cpuUsage(started);
least = Math.min(least, user + system);
}
return least;
}

it("rejects at the first error and reads no further than the part it is in (GHSA-c6fr-r6fm-4hjm)", async () => {
const count = 4_000;
const graph = (node: (at: number) => string) =>
`{"@context":{"ex":"https://example.org/ns#"},"@graph":[${Array.from({ length: count }, (_, at) => node(at)).join()}]}`;
const nodes = graph((at) => `{"@id":"ex:s${at}","ex:p":"v${at}"}`);
const errors = graph(() => '{"@id":5}');
await expect(
resources.parseQuads(nodes, "jsonld", "https://pod.example/"),
).resolves.toHaveLength(count);
await expect(resources.parseQuads(errors, "jsonld", "https://pod.example/")).rejects.toThrow(
"Found illegal @id '5'",
);
// The parser would take longer over the errors than over the nodes, did it read them all.
expect(await cpuUntilFree(errors)).toBeLessThan((await cpuUntilFree(nodes)) / 2);
});
});
56 changes: 49 additions & 7 deletions packages/solid/src/client/resources.ts
Original file line number Diff line number Diff line change
Expand Up @@ -498,9 +498,12 @@ export function parseQuads(
* its other keys (refuseRemoteContexts writes it so), or the text does
* not parse: otherwise the parser holds every node to the end, then
* matches each against every typed one, time growing with their square.
* The text is written in parts (textChunks), and the parser destroyed
* once `onQuad` says not to go on, so that it reads no further than the
* part it is in.
* The text is written in parts (textChunks), each in a task of its own
* (taskQueue), and the parser destroyed once `onQuad` says not to go on
* or at its first error, so that it reads no further than the part it
* is in. The parser reports some errors and reads on, each taking it far
* more time than a triple; and the parts written at once would hold the
* tab for all the work they queue, past the error that ends the parse.
*/
function parseJsonLd(
text: string,
Expand All @@ -522,13 +525,52 @@ function parseJsonLd(
load: (url) => Promise.reject(new Error(`A JSON-LD context elsewhere is not read: ${url}`)),
},
});
let stopped = false;
const stop = () => {
stopped = true;
parser.destroy();
};
parser.on("data", (quad: Quad) => {
if (!onQuad(quad)) parser.destroy();
if (!onQuad(quad)) stop();
});
parser.on("error", (error: Error) => {
stop();
onError(error);
});
parser.on("error", onError);
parser.on("end", done);
for (const chunk of textChunks(text)) parser.write(chunk);
parser.end();
const tasks = taskQueue();
(async () => {
try {
for (const chunk of textChunks(text)) {
parser.write(chunk);
await tasks.next();
if (stopped) break;
}
} finally {
tasks.close();
}
if (!stopped) parser.end();
})().catch(onError);
}

/**
* A wait for the next task of the event loop, after the microtasks
* queued so far, the parser's work on what it was written among them:
* a message to itself, which a browser delivers at once, where a
* timeout nested this deep waits some milliseconds.
*/
function taskQueue(): { next: () => Promise<void>; close: () => void } {
const channel = new MessageChannel();
let resume: () => void;
channel.port1.onmessage = () => resume();
return {
next: () =>
new Promise((resolve) => {
resume = resolve;
channel.port2.postMessage(null);
}),
close: () => channel.port1.close(),
};
}

/** The URL of the document a dataset was read from or saved to; null for one built here. */
Expand Down
113 changes: 106 additions & 7 deletions packages/solid/src/jsonLd.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import { growth } from "@solid-memo/domain/testing/growth";
import { jsonLdOf, refuseRemoteContexts as refuseIn } from "./jsonLd";
import { termOf } from "./ntriples";
import { MAX_TERM_CHARACTERS } from "./client/readLimits";
import type { ReadLimits } from "./client/interface";
import { parseQuads } from "./client/resources";

const EX = "https://example.org/ns#";
Expand Down Expand Up @@ -43,6 +44,27 @@ async function triplesOf(text: string): Promise<string[]> {
.sort();
}

/**
* The least CPU time, of three reads, that checking a deck file and
* parsing what passes takes, until the main thread is free again: the
* parser may still work after its promise settles.
*/
async function cpuUntilFree(text: string, limits: ReadLimits): Promise<number> {
let least = Infinity;
for (let round = 0; round < 3; round++) {
const started = process.cpuUsage();
try {
await parseQuads(refuseIn(text, EX, limits), "jsonld", EX, limits);
} catch {
// Refused, or not parsed: the time until then counts all the same.
}
await new Promise((resolve) => setTimeout(resolve, 0));
const { user, system } = process.cpuUsage(started);
least = Math.min(least, user + system);
}
return least;
}

describe("jsonLdOf", () => {
const quads = [
quad(iri(`${EX}a`), RDF_TYPE, iri(`${EX}Thing`)),
Expand Down Expand Up @@ -297,6 +319,9 @@ describe("refuseRemoteContexts", () => {
{ ex: EX, p: "ex:p" },
[{ ex: EX }, { "@vocab": "ex:" }],
{ ex: EX, "@base": "ex:a/" },
// A term defined as another, which may name a keyword.
{ "x:y": "@included", b: "x:y" },
{ "x:y": EX, b: "x:y" },
]) {
expect(() =>
refuseRemoteContexts(JSON.stringify({ "@context": context, "@id": "n" })),
Expand All @@ -317,13 +342,12 @@ describe("refuseRemoteContexts", () => {
// A file of nothing else, as large as a deck file may be, is refused too.
expect(() => refuseRemoteContexts(values(value, 100_000))).toThrow(refused);
}
// A string, number or boolean where a node belongs states nothing either.
for (const key of ["@graph", "@included"]) {
const nodes = (count: number) =>
`{"@context":{"ex":"${EX}"},"${key}":[${Array(count).fill('"a"').join(",")}]}`;
expect(() => refuseRemoteContexts(nodes(4_096))).not.toThrow();
expect(() => refuseRemoteContexts(nodes(4_097))).toThrow(refused);
}
// A string, number or boolean where a node belongs states nothing either (in `@included`, it is
// refused as invalid at once).
const nodes = (count: number) =>
`{"@context":{"ex":"${EX}"},"@graph":[${Array(count).fill('"a"').join(",")}]}`;
expect(() => refuseRemoteContexts(nodes(4_096))).not.toThrow();
expect(() => refuseRemoteContexts(nodes(4_097))).toThrow(refused);
expect(() => refuseRemoteContexts(`[${Array(4_097).fill("1").join(",")}]`)).toThrow(refused);
expect(() =>
refuseRemoteContexts(
Expand Down Expand Up @@ -439,6 +463,81 @@ describe("refuseRemoteContexts", () => {
}
});

it("refuses a keyword given a value the parser reports as an error and reads on past, however the keyword is named (GHSA-c6fr-r6fm-4hjm)", () => {
const refused = expect.objectContaining({
code: "deckFileUnreadable",
vars: { reason: "The file gives a JSON-LD keyword a value it may not have." },
});
const context = {
ex: EX,
i: "@id",
t: "@type",
inc: "@included",
n: "@nest",
v: "@value",
s: "@set",
};
const node = (members: object) => JSON.stringify({ "@context": context, ...members });
for (const members of [
{ "@graph": [{ "@id": 5 }] },
{ "@graph": [{ i: null }] },
{ "@id": ["ex:a"] },
{ "@id": "ex:a", i: "ex:b" },
{ "@id": "ex:a", t: 5 },
{ "@id": "ex:a", "@type": ["ex:T", ["ex:U"]] },
{ "@id": "ex:a", "@index": 5 },
{ "@id": "ex:a", "@reverse": "ex:b" },
{ "@id": "ex:a", "@reverse": { "@id": "ex:b" } },
{ "@id": "ex:a", inc: "s" },
{ "@id": "ex:a", "@included": [{ "@id": "ex:b" }, { v: "s" }] },
{ "@id": "ex:a", "@included": { "@list": [] } },
{ "@id": "ex:a", n: "x" },
{ "@id": "ex:a", "@nest": [{ "ex:p": "s" }, null] },
{ "@id": "ex:a", "@nest": { v: "s" } },
{ "@id": "ex:a", "@nest": { i: "ex:b" } },
// A `@set` object's members are the keyword's, in an array in it too, under an alias as well.
{ "@id": "ex:a", "@included": { "@set": [{ "@value": "v" }] } },
{ "@id": "ex:a", inc: { s: ["s"] } },
{ "@id": "ex:a", "@included": [{ "@set": [[{ "@id": "ex:b" }, "s"]] }] },
{ "@id": "ex:a", "@nest": { "@set": [{ "@value": "v" }] } },
{ "@id": "ex:a", n: { s: { s: ["s"] } } },
{ "@id": "ex:a", "@nest": { "@set": [{ "@id": "ex:b" }] } },
{ "@id": "ex:a", "ex:p": { "@list": ["s"], "@id": "ex:b" } },
{ "@id": "ex:a", "ex:p": { "@list": ["s"], "ex:q": "s" } },
]) {
expect(() => refuseRemoteContexts(node(members))).toThrow(refused);
}
// The same keywords given what the parser reads, and a context's own members, which name them.
const valid = node({
"@id": "ex:a",
t: ["ex:T", "ex:U"],
"@index": "i",
"@reverse": { "ex:p": { i: "ex:b" } },
inc: [{ i: "ex:c", "ex:p": "s" }, { s: [{ i: "ex:d" }] }],
n: [{ "ex:q": "s", "@type": "ex:V" }, { "@set": [{ "ex:q": "t" }] }],
"ex:r": { "@list": ["s"], "@index": "j" },
});
expect(() => refuseRemoteContexts(valid)).not.toThrow();
});

it("refuses up front a file of keywords given values the parser reports as errors, which would take it longer than an exported file at the same limits (GHSA-c6fr-r6fm-4hjm)", async () => {
const quads = 2_000;
const limits = { bytes: 64 * 1024 * 1024, quads, characters: 64 * 1024 * 1024 };
const exported = jsonLdOf(
Array.from({ length: quads }, (_, at) =>
quad(iri(`${EX}s${at % 100}`), `${EX}p`, literal(`v${at}`, at % 2 === 0 ? "en" : "")),
),
{ ex: EX },
);
// As many values as the bound on the work let through at these limits before.
const ids = `{"@graph":[${Array(3_400).fill('{"@id":5}').join()}]}`;
const included = `{"@context":{"x:y":"@included","b":"x:y"},"@id":"${EX}x","b":[${Array(3_600).fill('"s"').join()}]}`;
const base = await cpuUntilFree(exported, limits);
for (const text of [ids, included]) {
expect(await cpuUntilFree(text, limits)).toBeLessThan(base);
}
});

it("gives the document it checked, a key named twice as JSON.parse keeps it", () => {
expect(refuseRemoteContexts('{"@context": "https://evil.example/c", "@context": {}}')).toBe(
'{"@context":{}}',
Expand Down
Loading
Loading