Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 38 additions & 27 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,41 +5,50 @@ Parse, transform, and emit Markdown (or HTML) with a composable, AST-based API.

## Overview

markflow is built on [Flexmark-java](https://github.com/vsch/flexmark-java) and provides:
markflow is built on [commonmark-java](https://github.com/commonmark/commonmark-java) and provides:

- A clean Kotlin API for parsing Markdown into a mutable AST
- Composable transformation passes (heading manipulation, front matter promotion, etc.)
- Round-trip Markdown output via Flexmark's `Formatter`, with reliable front matter handling
- Round-trip Markdown output via commonmark-java's `MarkdownRenderer`, with reliable front matter handling
- A Gradle plugin exposing common transformations as cacheable, incremental tasks

It is **not** a static site generator. It is a document-processing library and build-tool
integration layer — the Markdown equivalent of what a CSS post-processor does for stylesheets.

## Design Decisions

### Parser: Flexmark-java
### Parser: commonmark-java

markflow uses [Flexmark-java](https://github.com/vsch/flexmark-java) as its parser and formatter.
Flexmark was chosen because:
markflow uses [commonmark-java](https://github.com/commonmark/commonmark-java) as its parser and
formatter. commonmark-java was chosen because:

- It has the most mature round-trip Markdown output (`Formatter`) of any JVM library
- Its extension API is clean and well-suited to custom AST node types
- It supports GFM tables, YAML front matter, and other common extensions out of the box
- It is Apache 2.0 licensed, compatible with all likely dependencies
- It is actively maintained (Atlassian origin, used by OpenJDK, Google, Gerrit)
- It is spec-compliant with [CommonMark](https://spec.commonmark.org/)
- It provides round-trip Markdown output via `MarkdownRenderer`
- It supports GFM tables, YAML front matter (including raw/opaque capture), strikethrough, footnotes,
and other common extensions out of the box
- It is BSD 2-Clause licensed, compatible with all likely dependencies

[JetBrains Markdown](https://github.com/JetBrains/markdown) was considered but ruled out: its AST
is largely immutable, it has no Markdown output support, and it has no front matter extension.
It remains the only JVM option for Kotlin Multiplatform targets, but KMP is not a current requirement.
**Alternatives considered:**

### Front Matter: Opaque Text Node
[Flexmark-java](https://github.com/vsch/flexmark-java) was the original choice and has a richer
extension ecosystem (TOC, definition lists, typographic quotes, etc.), but has been effectively
unmaintained since 2023 with 178+ open issues. The maintenance risk outweighs the extension breadth
for a library intended for long-term use.

Flexmark's built-in `YamlFrontMatterExtension` parses front matter into key/value pairs but does
not reliably round-trip it through the `Formatter`. markflow instead provides a custom extension
that captures the entire front matter block as a single opaque text node in the AST. This means:
[JetBrains Markdown](https://github.com/JetBrains/markdown) is actively developed but ruled out:
its AST is largely immutable, it has no Markdown output support, and it targets Kotlin Multiplatform
which is not a current requirement.

- The `Formatter` preserves front matter verbatim by default
### Front Matter: Opaque Capture via RawContentParser

commonmark-java's `YamlFrontMatterExtension` supports a `RawContentParser` mode that captures the
entire front matter block as a single raw string rather than parsing it into key/value pairs. This
means:

- The `MarkdownRenderer` preserves front matter verbatim by default
- Callers that need to inspect or modify front matter values do so via Jackson YAML, extracting
the raw text from the node, modifying it, and writing it back
the raw text, modifying it, and writing it back
- No front matter content is silently dropped or reordered

### Transformation Pipeline
Expand All @@ -59,18 +68,18 @@ markflow/

### `library`

The standalone Kotlin library JAR. Depends only on Flexmark-java and Jackson YAML. No build-tool
The standalone Kotlin library JAR. Depends only on commonmark-java and Jackson YAML. No build-tool
APIs. Usable from any JVM context: Gradle, Maven, a CLI, application code.

Key types:

| Type | Description |
|---|---|
| `MarkdownDocument` | A parsed document: mutable Flexmark `Document` + raw front matter string |
| `MarkdownDocument` | A parsed document: mutable commonmark-java `Document` + raw front matter string |
| `MarkdownParser` | Parses a `String` or `File` into a `MarkdownDocument` |
| `MarkdownTransformer` | Single-responsibility AST transformation pass |
| `MarkdownPipeline` | Ordered composition of `MarkdownTransformer` instances |
| `MarkdownFormatter` | Emits a `MarkdownDocument` as Markdown (via Flexmark `Formatter`) or HTML (via `HtmlRenderer`) |
| `MarkdownFormatter` | Emits a `MarkdownDocument` as Markdown (via `MarkdownRenderer`) or HTML (via `HtmlRenderer`) |
| `FrontMatter` | Typed view of the front matter block via Jackson YAML |

Built-in transformers:
Expand All @@ -82,7 +91,7 @@ Built-in transformers:
| `TableFormatter` | Pads GFM table columns to consistent widths |
| `SentencePerLineFormatter` | Splits paragraph text at sentence boundaries, one sentence per line |
| `TemplateSubstitutor` | Replaces `{{key}}` placeholders with provided values, skipping fenced code blocks |
| `ScreenshotPlaceholderExpander` | Expands `<!-- screenshot: name.png | alt -->` to `![alt](path/name.png)` |
| `HtmlCommentExpander` | Expands structured HTML comments to Markdown content via registered handlers |

### `gradle-plugin`

Expand Down Expand Up @@ -111,7 +120,7 @@ Sub-packages follow feature boundaries:
|---|---|
| `net.oxspring.markflow` | Public API: `MarkdownDocument`, `MarkdownParser`, `MarkdownPipeline`, `MarkdownFormatter` |
| `net.oxspring.markflow.transform` | Built-in `MarkdownTransformer` implementations |
| `net.oxspring.markflow.frontmatter` | `FrontMatter`, opaque front matter AST extension |
| `net.oxspring.markflow.frontmatter` | `FrontMatter`, front matter access utilities |
| `net.oxspring.markflow.gradle` | Gradle plugin and task implementations |

## MVP Scope (v1.0)
Expand All @@ -129,7 +138,7 @@ The v1.0 MVP targets feature parity with the custom Gradle tasks in the
markflow replaces this with a `ProcessMarkdownTask` configured with:

- `TemplateSubstitutor` (token substitution, fence-aware)
- `ScreenshotPlaceholderExpander`
- `HtmlCommentExpander` with a screenshot handler

### Replaces front matter title extraction in `BuildPdfTask`

Expand Down Expand Up @@ -158,16 +167,18 @@ markflow replaces this with a `ProcessMarkdownTask` configured with:
./gradlew build # assemble + check
```

Test framework: JUnit 5 (Jupiter) with AssertJ.
Test framework: JUnit 6 (Jupiter) with AssertJ.

## Key Dependencies

| Dependency | Version | License |
|---|---|---|
| Flexmark-java | 0.64.x | BSD 2-Clause |
| commonmark-java | 0.30.x | BSD 2-Clause |
| commonmark-ext-gfm-tables | 0.30.x | BSD 2-Clause |
| commonmark-ext-yaml-front-matter | 0.30.x | BSD 2-Clause |
| Jackson YAML | 2.x | Apache 2.0 |
| Kotlin | 2.x | Apache 2.0 |
| Gradle Plugin API | 8.x | Apache 2.0 |
| Gradle Plugin API | 9.x | Apache 2.0 |

## License

Expand Down
13 changes: 7 additions & 6 deletions gradle/libs.versions.toml
Original file line number Diff line number Diff line change
@@ -1,20 +1,21 @@
[versions]
assertj = "3.27.7"
flexmark = "0.64.8"
commonmark = "0.30.0"
git-versioning = "6.4.4"
jacksonYaml = "2.22.2"
jackson-yaml = "2.22.2"
junit5 = "6.1.3"
kotlin = "2.4.10"
kover = "0.9.9"
ktlint-gradle = "14.2.0"

[libraries]
assertj-core = { module = "org.assertj:assertj-core", version.ref = "assertj" }
flexmark-all = { module = "com.vladsch.flexmark:flexmark-all", version.ref = "flexmark" }
jackson-dataformat-yaml = { module = "com.fasterxml.jackson.dataformat:jackson-dataformat-yaml", version.ref = "jacksonYaml" }
jackson-module-kotlin = { module = "com.fasterxml.jackson.module:jackson-module-kotlin", version.ref = "jacksonYaml" }
commonmark = { module = "org.commonmark:commonmark", version.ref = "commonmark" }
commonmark-ext-gfm-tables = { module = "org.commonmark:commonmark-ext-gfm-tables", version.ref = "commonmark" }
commonmark-ext-yaml-front-matter = { module = "org.commonmark:commonmark-ext-yaml-front-matter", version.ref = "commonmark" }
jackson-dataformat-yaml = { module = "com.fasterxml.jackson.dataformat:jackson-dataformat-yaml", version.ref = "jackson-yaml" }
jackson-module-kotlin = { module = "com.fasterxml.jackson.module:jackson-module-kotlin", version.ref = "jackson-yaml" }
junit-jupiter = { module = "org.junit.jupiter:junit-jupiter", version.ref = "junit5" }
kotlin-stdlib = { module = "org.jetbrains.kotlin:kotlin-stdlib", version.ref = "kotlin" }

[plugins]
git-versioning = { id = "me.qoomon.git-versioning", version.ref = "git-versioning" }
Expand Down
4 changes: 3 additions & 1 deletion library/build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@ plugins {
}

dependencies {
implementation(libs.flexmark.all)
implementation(libs.commonmark)
implementation(libs.commonmark.ext.gfm.tables)
implementation(libs.commonmark.ext.yaml.front.matter)
implementation(libs.jackson.dataformat.yaml)
implementation(libs.jackson.module.kotlin)

Expand Down