From 0cf0112e56f7fd329b77261a25b53ec1d6e3f72a Mon Sep 17 00:00:00 2001 From: projectcontinuum Date: Tue, 27 Oct 2026 12:21:37 +0530 Subject: [PATCH] CONTNM-128: Step by step guide to develop custome features --- .../create-new-nodes/_category_.json | 8 + .../contributing/create-new-nodes/overview.md | 58 ++++++ .../create-new-nodes/step-1-root-project.md | 48 +++++ .../create-new-nodes/step-2-feature-module.md | 61 ++++++ .../create-new-nodes/step-3-write-the-node.md | 174 ++++++++++++++++++ .../step-4-node-documentation.md | 15 ++ .../create-new-nodes/step-5-unit-tests.md | 48 +++++ .../create-new-nodes/step-6-worker-module.md | 95 ++++++++++ .../create-new-nodes/step-7-build-and-test.md | 21 +++ .../create-new-nodes/step-8-run-locally.md | 22 +++ .../create-new-nodes/step-9-ship-it.md | 11 ++ .../create-new-nodes/troubleshooting.md | 18 ++ docs/contributing/feature-template.md | 4 + 13 files changed, 583 insertions(+) create mode 100644 docs/contributing/create-new-nodes/_category_.json create mode 100644 docs/contributing/create-new-nodes/overview.md create mode 100644 docs/contributing/create-new-nodes/step-1-root-project.md create mode 100644 docs/contributing/create-new-nodes/step-2-feature-module.md create mode 100644 docs/contributing/create-new-nodes/step-3-write-the-node.md create mode 100644 docs/contributing/create-new-nodes/step-4-node-documentation.md create mode 100644 docs/contributing/create-new-nodes/step-5-unit-tests.md create mode 100644 docs/contributing/create-new-nodes/step-6-worker-module.md create mode 100644 docs/contributing/create-new-nodes/step-7-build-and-test.md create mode 100644 docs/contributing/create-new-nodes/step-8-run-locally.md create mode 100644 docs/contributing/create-new-nodes/step-9-ship-it.md create mode 100644 docs/contributing/create-new-nodes/troubleshooting.md diff --git a/docs/contributing/create-new-nodes/_category_.json b/docs/contributing/create-new-nodes/_category_.json new file mode 100644 index 0000000..0deb03f --- /dev/null +++ b/docs/contributing/create-new-nodes/_category_.json @@ -0,0 +1,8 @@ +{ + "label": "Create New Nodes", + "position": 2, + "link": { + "type": "doc", + "id": "contributing/create-new-nodes/overview" + } +} diff --git a/docs/contributing/create-new-nodes/overview.md b/docs/contributing/create-new-nodes/overview.md new file mode 100644 index 0000000..fbc11d2 --- /dev/null +++ b/docs/contributing/create-new-nodes/overview.md @@ -0,0 +1,58 @@ +--- +sidebar_position: 1 +sidebar_label: Overview +title: Create New Nodes +slug: /contributing/create-new-nodes +--- + +# Create New Nodes + +This guide walks you through building a **feature worker** with your own workflow node, from an empty folder to a passing build and a worker that registers its node with the platform. The running example is a **Word Counter** node that counts the words in a text column and appends the count as a new column. + +You need to know some Kotlin and Gradle. You don't need to know Temporal, Kafka or Spring internals; the Continuum Gradle plugins and the worker starter handle those. + +:::info Versions used in this guide +Verified against **Continuum platform `0.0.20`** (Spring Boot 4.0.6, Kotlin 2.2, Jackson 3, Gradle 8.14, JDK 21). Check [Maven Central](https://central.sonatype.com/artifact/org.projectcontinuum.core/continuum-commons) for the latest platform version, and use the same version everywhere in your project. +::: + +## How the pieces fit + +A feature worker is a small Gradle multi-project: + +``` +continuum-feature-demo/ +├── settings.gradle.kts # plugin versions + module list +├── gradle.properties # continuumPlatformVersion, featureVersion +├── features/ +│ └── continuum-feature-demo/ # library module: your node classes +└── worker/ # thin Spring Boot app that hosts the nodes +``` + +- **Feature modules** (`org.projectcontinuum.feature` plugin) hold the node classes. They're plain libraries and can be published and reused. +- **The worker module** (`org.projectcontinuum.worker` plugin) is a Spring Boot app with no node logic of its own. It depends on your feature modules and on `continuum-worker-springboot-starter`. At startup the starter finds every `@ContinuumNode` bean, polls a Temporal task queue for node executions, and publishes one registration message per node to Kafka. + +See [Node Model & Feature Registration](../../architecture/node-model.md) for what happens after registration. + +## Prerequisites + +| Requirement | Notes | +|-------------|-------| +| JDK 21 | Eclipse Temurin recommended. The plugins set a Java 21 toolchain. | +| Gradle wrapper 8.14+ | Older wrappers (for example 8.4) can't load the Spring Boot 4 Gradle plugin. | +| Docker | Only needed to run the worker against the [local stack](../../getting-started/local-setup.md). | + +You **don't** need a GitHub token. All Continuum artifacts and Gradle plugins are published to **Maven Central**. + +## The steps + +1. [Create the root project](./step-1-root-project.md) +2. [Create the feature module](./step-2-feature-module.md) +3. [Write the node](./step-3-write-the-node.md) +4. [Add node documentation](./step-4-node-documentation.md) +5. [Write unit tests](./step-5-unit-tests.md) +6. [Create the worker module](./step-6-worker-module.md) +7. [Build and test](./step-7-build-and-test.md) +8. [Run the worker against the local stack](./step-8-run-locally.md) +9. [Ship it](./step-9-ship-it.md) + +If something goes wrong along the way, see [Troubleshooting](./troubleshooting.md). diff --git a/docs/contributing/create-new-nodes/step-1-root-project.md b/docs/contributing/create-new-nodes/step-1-root-project.md new file mode 100644 index 0000000..b1a7951 --- /dev/null +++ b/docs/contributing/create-new-nodes/step-1-root-project.md @@ -0,0 +1,48 @@ +--- +sidebar_position: 2 +sidebar_label: "1. Root project" +title: "Step 1: Create the root project" +--- + +# Step 1: Create the root project + +Create the folder and add a Gradle wrapper. If Gradle is installed, run `gradle wrapper --gradle-version 8.14`. Otherwise, copy `gradlew`, `gradlew.bat` and `gradle/wrapper/` from an existing feature repo such as `continuum-feature-base`. + +**`gradle.properties`** + +```properties +repoName=your-org/continuum-feature-demo +continuumPlatformVersion=0.0.20 +featureVersion=0.0.1 + +# Spring Boot 4 pulls in a large Kotlin compiler jar; give the Gradle daemon enough heap. +org.gradle.jvmargs=-Xmx4g -XX:MaxMetaspaceSize=1g +``` + +- `continuumPlatformVersion` selects the Gradle plugin version and the `continuum-commons` / `continuum-worker-springboot-starter` versions. +- `featureVersion` is your own version. +- `repoName` is used in published POM metadata and in the default container image name. + +**`settings.gradle.kts`** + +```kotlin +pluginManagement { + val continuumPlatformVersion: String by settings + repositories { + mavenLocal() + mavenCentral() + gradlePluginPortal() + } + plugins { + id("org.projectcontinuum.feature") version continuumPlatformVersion + id("org.projectcontinuum.worker") version continuumPlatformVersion + } +} + +rootProject.name = "continuum-feature-demo" + +include(":features:continuum-feature-demo") +include(":worker") +``` + +The plugins are resolved from Maven Central, so `mavenCentral()` must be in `pluginManagement.repositories`, not only in `gradlePluginPortal()`. diff --git a/docs/contributing/create-new-nodes/step-2-feature-module.md b/docs/contributing/create-new-nodes/step-2-feature-module.md new file mode 100644 index 0000000..44f03ed --- /dev/null +++ b/docs/contributing/create-new-nodes/step-2-feature-module.md @@ -0,0 +1,61 @@ +--- +sidebar_position: 3 +sidebar_label: "2. Feature module" +title: "Step 2: Create the feature module" +--- + +# Step 2: Create the feature module + +**`features/continuum-feature-demo/build.gradle.kts`** + +```kotlin +plugins { + id("org.projectcontinuum.feature") +} + +group = "org.projectcontinuum.feature.demo" +description = "Continuum Feature Demo — demo nodes" +version = property("featureVersion").toString() + +continuum { + continuumVersion.set(property("continuumPlatformVersion").toString()) +} +``` + +:::warning Always set `continuumVersion` +If you leave out the `continuum { }` block, the plugin falls back to a built-in default version that is older than the plugin itself. Read the value from `gradle.properties` so plugins and libraries always match. +::: + +The `org.projectcontinuum.feature` plugin applies Kotlin and Spring dependency management and adds these dependencies for you, so you don't declare them yourself: + +- Spring Boot starter, web, actuator and autoconfigure +- `continuum-commons` +- `kotlin-reflect` and `tools.jackson.module:jackson-module-kotlin` +- Test: `spring-boot-starter-test`, `kotlin-test-junit5`, `mockito-kotlin` + +It also registers the `validateContinuumNodeAnnotations` task, which runs before `classes`. Add a `dependencies { }` block only for extra libraries your nodes need. + +## Register the module with Spring + +The worker finds your nodes through Spring Boot auto-configuration, so each feature module needs two small files. + +**`src/main/kotlin/org/projectcontinuum/feature/demo/AutoConfigure.kt`** + +```kotlin +package org.projectcontinuum.feature.demo + +import org.springframework.context.annotation.ComponentScan +import org.springframework.context.annotation.Configuration + +@Configuration +@ComponentScan +class AutoConfigure +``` + +**`src/main/resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports`** + +```text +org.projectcontinuum.feature.demo.AutoConfigure +``` + +`@ComponentScan` with no arguments scans the package of `AutoConfigure` and its subpackages. If the imports file is missing or names the wrong class, the build still passes but the worker starts with none of your nodes. diff --git a/docs/contributing/create-new-nodes/step-3-write-the-node.md b/docs/contributing/create-new-nodes/step-3-write-the-node.md new file mode 100644 index 0000000..c978c03 --- /dev/null +++ b/docs/contributing/create-new-nodes/step-3-write-the-node.md @@ -0,0 +1,174 @@ +--- +sidebar_position: 4 +sidebar_label: "3. Write the node" +title: "Step 3: Write the node" +--- + +# Step 3: Write the node + +Put node classes in a `.node` subpackage. The platform derives your **feature ID** by removing the trailing `.node` segment from the class name. For example, `org.projectcontinuum.feature.demo.node.WordCounterNodeModel` gets the feature ID `org.projectcontinuum.feature.demo`. + +**`src/main/kotlin/org/projectcontinuum/feature/demo/node/WordCounterNodeModel.kt`** + +```kotlin +package org.projectcontinuum.feature.demo.node + +import org.projectcontinuum.core.commons.annotation.ContinuumNode +import org.projectcontinuum.core.commons.exception.NodeRuntimeException +import org.projectcontinuum.core.commons.model.ContinuumWorkflowModel +import org.projectcontinuum.core.commons.node.ProcessNodeModel +import org.projectcontinuum.core.commons.protocol.progress.NodeProgressCallback +import org.projectcontinuum.core.commons.utils.NodeInputReader +import org.projectcontinuum.core.commons.utils.NodeOutputWriter +import org.slf4j.LoggerFactory +import org.springframework.http.MediaType.TEXT_PLAIN_VALUE +import tools.jackson.core.type.TypeReference +import tools.jackson.databind.ObjectMapper + +@ContinuumNode +class WordCounterNodeModel( + private val objectMapper: ObjectMapper +) : ProcessNodeModel() { + companion object { + private val LOGGER = LoggerFactory.getLogger(WordCounterNodeModel::class.java) + private val WHITESPACE = Regex("\\s+") + } + + // 1. Ports: the map keys ("data") are the port IDs used in execute() + final override val inputPorts = mapOf( + "data" to ContinuumWorkflowModel.NodePort(name = "input table", contentType = TEXT_PLAIN_VALUE) + ) + final override val outputPorts = mapOf( + "data" to ContinuumWorkflowModel.NodePort(name = "table with word counts", contentType = TEXT_PLAIN_VALUE) + ) + + // 2. Where the node appears in the workbench node explorer + override val categories = listOf("Demo") + + // 3. JSON Schema for the node's properties + val propertiesSchema: Map = objectMapper.readValue( + """ + { + "type": "object", + "properties": { + "inputCol": { "type": "string", "title": "Input Column" }, + "outputCol": { "type": "string", "title": "Output Column" }, + "minWordLength": { "type": "integer", "title": "Minimum Word Length", "minimum": 1, "default": 1 } + }, + "required": ["inputCol", "outputCol"] + } + """.trimIndent(), + object : TypeReference>() {} + ) + + // 4. JSON Forms UI schema: layout of the configuration dialog + val propertiesUiSchema: Map = objectMapper.readValue( + """ + { + "type": "VerticalLayout", + "elements": [ + { "type": "Control", "scope": "#/properties/inputCol" }, + { "type": "Control", "scope": "#/properties/outputCol" }, + { "type": "Control", "scope": "#/properties/minWordLength" } + ] + } + """.trimIndent(), + object : TypeReference>() {} + ) + + // 5. Metadata: what the workbench and registry see + override val metadata = ContinuumWorkflowModel.NodeData( + id = this.javaClass.name, + nodeModel = this.javaClass.name, + title = "Word Counter", + subTitle = "Count words per row", + description = "Counts the words in a text column and appends the count as a new column", + icon = """""", + inputs = inputPorts, + outputs = outputPorts, + properties = mapOf("inputCol" to "text", "outputCol" to "word_count", "minWordLength" to 1), + propertiesSchema = propertiesSchema, + propertiesUISchema = propertiesUiSchema + ) + + // 6. The node's logic + override fun execute( + properties: Map?, + inputs: Map, + nodeOutputWriter: NodeOutputWriter, + nodeProgressCallback: NodeProgressCallback + ) { + val inputCol = properties?.get("inputCol") as String? + ?: throw NodeRuntimeException(workflowId = "", nodeId = "", message = "inputCol is not provided") + val outputCol = properties["outputCol"] as String? + ?: throw NodeRuntimeException(workflowId = "", nodeId = "", message = "outputCol is not provided") + val minWordLength = (properties["minWordLength"] as Number?)?.toInt() ?: 1 + + val reader = inputs["data"] ?: throw NodeRuntimeException( + isRetriable = false, workflowId = "", nodeId = "", message = "Input port 'data' is not connected" + ) + val totalRows = reader.getRowCount() + + nodeOutputWriter.createOutputPortWriter("data").use { writer -> + reader.use { + var row = reader.read() + var rowNumber = 0L + while (row != null) { + val text = row[inputCol]?.toString() ?: "" + val count = text.trim().split(WHITESPACE).count { it.length >= minWordLength } + + writer.write(rowNumber, row.toMutableMap().apply { this[outputCol] = count }) + rowNumber++ + if (totalRows > 0) nodeProgressCallback.report((rowNumber * 100 / totalRows).toInt()) + row = reader.read() + } + LOGGER.info("Counted words in $rowNumber rows") + } + } + } +} +``` + +## The node contract + +| Rule | Why | +|------|-----| +| Annotate with `@ContinuumNode`, not `@Component`. | `@ContinuumNode` makes the bean prototype-scoped, so each execution gets a fresh instance. The build fails if a node class is missing it (see [Step 7](./step-7-build-and-test.md)). | +| Extend `ProcessNodeModel` for nodes with inputs, or `TriggerNodeModel` for source nodes. | `TriggerNodeModel` has only `outputPorts` and `execute(properties, nodeOutputWriter)`. | +| Set `metadata.id` and `metadata.nodeModel` to `this.javaClass.name`. | The fully qualified class name is the node's identity in the registry and in saved workflows. Renaming or moving the class breaks existing workflows that use it. | +| Use Jackson 3 imports (`tools.jackson.*`). | Platform `0.0.20` is on Spring Boot 4 / Jackson 3. `com.fasterxml.jackson.databind` isn't on the classpath and won't compile. | +| Take `ObjectMapper` as a constructor parameter. | Spring injects it. In unit tests, pass `jacksonObjectMapper()`. | + +## Properties and validation + +- `metadata.properties` holds the default values used when a node is dropped onto the canvas. +- Before `execute()` is called, `ProcessNodeModel.run()` **validates the configured properties against `propertiesSchema`**. Values that break `required`, `type`, `minimum` and similar rules fail the node before your code runs. You can still check values in `execute()` for clearer error messages. +- `propertiesUISchema` is a [JSON Forms](https://jsonforms.io/) UI schema, used only by the workbench to draw the configuration dialog. For credential pickers (`options.format = "credential"`), see [Credentials](../../architecture/node-model.md#credentials). You can try schemas in the [JSON Forms playground](https://github.com/projectcontinuum-org/continuum-jsonform-playground) before wiring them into a node. + +## Reading and writing rows + +- `inputs[portId]` is a `NodeInputReader`. `read()` returns the next row as `Map`, or `null` at the end of the table. `getRowCount()` returns the total number of rows. +- `nodeOutputWriter.createOutputPortWriter(portId)` returns a writer. `write(rowNumber, row)` takes a `Long` row index starting at `0`. +- Wrap both reader and writer in `.use { }`. The output Parquet file is only complete once the writer is closed. +- If an input port isn't connected, its key is missing from `inputs`. + +Cell values must be one of these types. Any other type makes `write()` throw `IllegalArgumentException: Unsupported type`. + +| Kotlin type | Stored as | +|-------------|-----------| +| `String` | `application/vnd.continuum.x-string` | +| `Int`, `Long` | `application/vnd.continuum.x-int`, `x-long` | +| `Float`, `Double` | `application/vnd.continuum.x-float`, `x-double` | +| `Boolean` | `application/vnd.continuum.x-boolean` | +| `List<*>`, `Map<*, *>` | `application/json` | +| SVG `org.w3c.dom.Document` | `image/svg+xml` | + +Convert anything else first, for example call `toString()` on a `LocalDate` or `BigDecimal`. A `null` value can't be stored either: leave the key out instead. + +## Errors and retries + +Every node runs as a Temporal activity, so an exception thrown from `execute()` is **retried** by default. Throw `NodeRuntimeException(isRetriable = false, ...)` for errors that a retry won't fix, such as missing configuration or a disconnected port. You can pass empty strings for `workflowId` and `nodeId`. The existing nodes do the same. + +## Progress + +Call `nodeProgressCallback.report(percent)` with a value from 0 to 100 as you go. The workbench uses it to show live progress on the node. diff --git a/docs/contributing/create-new-nodes/step-4-node-documentation.md b/docs/contributing/create-new-nodes/step-4-node-documentation.md new file mode 100644 index 0000000..6262afa --- /dev/null +++ b/docs/contributing/create-new-nodes/step-4-node-documentation.md @@ -0,0 +1,15 @@ +--- +sidebar_position: 5 +sidebar_label: "4. Node documentation" +title: "Step 4: Add node documentation" +--- + +# Step 4: Add node documentation + +Each node has a Markdown help page that the workbench shows next to it. It's loaded from the classpath on startup, at a path that matches the node's package and class name exactly: + +```text +src/main/resources/org/projectcontinuum/feature/demo/node/WordCounterNodeModel.doc.md +``` + +If the file is missing, the node still works, but its help page reads *"Documentation not found for WordCounterNodeModel"*. A good help page covers what the node does, its ports, a properties table, and a small input/output example. See the `.doc.md` files in [`continuum-feature-base`](https://github.com/projectcontinuum-org/continuum-feature-base) for the house style. diff --git a/docs/contributing/create-new-nodes/step-5-unit-tests.md b/docs/contributing/create-new-nodes/step-5-unit-tests.md new file mode 100644 index 0000000..4a8c358 --- /dev/null +++ b/docs/contributing/create-new-nodes/step-5-unit-tests.md @@ -0,0 +1,48 @@ +--- +sidebar_position: 6 +sidebar_label: "5. Unit tests" +title: "Step 5: Write unit tests" +--- + +# Step 5: Write unit tests + +`execute()` only depends on its arguments, so you can test it with Mockito mocks and no Spring context: + +**`src/test/kotlin/org/projectcontinuum/feature/demo/node/WordCounterNodeModelTest.kt`** + +```kotlin +class WordCounterNodeModelTest { + private val outputWriter = mock() + private val portWriter = mock() + private val node = WordCounterNodeModel(jacksonObjectMapper()) + + @BeforeEach + fun setUp() { + whenever(outputWriter.createOutputPortWriter("data")).thenReturn(portWriter) + } + + @Test + fun `counts words and preserves original columns`() { + val reader = mockReader(listOf(mapOf("id" to 1, "text" to "the quick brown fox"))) + + node.execute(mapOf("inputCol" to "text", "outputCol" to "word_count"), mapOf("data" to reader), outputWriter, mock()) + + val rows = argumentCaptor>() + verify(portWriter).write(any(), rows.capture()) + assertEquals(4, rows.firstValue["word_count"]) + assertEquals(1, rows.firstValue["id"]) + } + + private fun mockReader(rows: List>): NodeInputReader { + val iterator = rows.iterator() + return mock().also { r -> + whenever(r.read()).thenAnswer { if (iterator.hasNext()) iterator.next() else null } + whenever(r.getRowCount()).thenReturn(rows.size.toLong()) + } + } +} +``` + +The imports come from `org.junit.jupiter.api`, `org.mockito.kotlin`, `kotlin.test` and `tools.jackson.module.kotlin.jacksonObjectMapper`. The feature plugin already adds all of them as test dependencies. + +These tests call `execute()` directly, so they skip the schema validation step. To test validation too, build a `ContinuumWorkflowModel.Node` and call `node.run(...)` instead. diff --git a/docs/contributing/create-new-nodes/step-6-worker-module.md b/docs/contributing/create-new-nodes/step-6-worker-module.md new file mode 100644 index 0000000..3748d25 --- /dev/null +++ b/docs/contributing/create-new-nodes/step-6-worker-module.md @@ -0,0 +1,95 @@ +--- +sidebar_position: 7 +sidebar_label: "6. Worker module" +title: "Step 6: Create the worker module" +--- + +# Step 6: Create the worker module + +**`worker/build.gradle.kts`** + +```kotlin +plugins { + id("org.projectcontinuum.worker") +} + +group = "org.projectcontinuum.app.worker.demo" +description = "Continuum Feature Demo Worker" +version = property("featureVersion").toString() + +continuum { + continuumVersion.set(property("continuumPlatformVersion").toString()) + // Optional: set this to build a container image with `./gradlew :worker:jib` + // jibToImage.set("docker.io/your-org/continuum-feature-demo:${version}") +} + +dependencies { + implementation(project(":features:continuum-feature-demo")) +} +``` + +**`worker/src/main/kotlin/org/projectcontinuum/app/worker/demo/App.kt`** + +```kotlin +package org.projectcontinuum.app.worker.demo + +import org.springframework.boot.autoconfigure.SpringBootApplication +import org.springframework.boot.runApplication + +@SpringBootApplication +class App + +fun main(args: Array) { + runApplication(*args) +} +``` + +**`worker/src/main/resources/application.yaml`** + +```yaml +spring.application.name: continuum-feature-demo-worker + +# The worker runs a web server. Its default port 8080 clashes with +# continuum-api-server in the local stack. +server.port: ${SERVER_PORT:8090} + +spring: + temporal: + namespace: ${TEMPORAL_NAMESPACE:default} + connection: + target: ${TEMPORAL_SERVER_ADDRESS:localhost:7233} + workersAutoDiscovery: + packages: + - org.projectcontinuum.core.worker.starter + +continuum.core: + worker: + # Every worker needs its OWN task queue (see the warning below). + node-task-queue: ${CONTINUUM_NODE_TASK_QUEUE:CONTINUUM_FEATURE_DEMO_TASK_QUEUE} + +logging: + level: + root: error + org.projectcontinuum.core.worker: info + org.projectcontinuum.feature.demo: debug +``` + +:::danger Give every worker a unique task queue +If `node-task-queue` isn't set, a worker listens on the shared default queue `ACTIVITY_TASK_QUEUE`. When two workers share a queue, Temporal can send a node to a worker that doesn't have it, and the node fails. Every shipped worker sets its own queue (for example `CONTINUUM_FEATURE_BASE_TASK_QUEUE`), and yours must too. The worker sends its queue name in its registration message, so the orchestrator knows where to route your nodes. +::: + +## Environment variables + +You don't need to configure Kafka, MinIO, the API server or the credentials server here. `continuum-worker-springboot-starter` ships defaults that point at the [local stack](../../getting-started/local-setup.md) ports, and each one can be overridden with an environment variable: + +| Variable | Default | Purpose | +|----------|---------|---------| +| `TEMPORAL_SERVER_ADDRESS` | `localhost:7233` | Temporal frontend | +| `CONTINUUM_KAFKA_MAIN_BROKER_LIST` | `localhost:39092,localhost:39093,localhost:39094` | Kafka brokers for node registration | +| `CONTINUUM_KAFKA_MAIN_SCHEMA_REGISTRY` | `http://localhost:38080` | Avro schema registry | +| `CONTINUUM_MINIO_ENDPOINT` | `http://localhost:39000` | Object store for Parquet inputs and outputs | +| `CONTINUUM_MINIO_USERNAME` / `_PASSWORD` | `minioadmin` / `minioadmin` | MinIO credentials | +| `CONTINUUM_API_SERVER_BASE_URL` | `http://localhost:8080` | API server | +| `CONTINUUM_CREDENTIALS_SERVER_BASE_URL` | `http://localhost:8083` | Only used by nodes that call `getCredential()` | +| `CONTINUUM_NODE_TASK_QUEUE` | none (falls back to `ACTIVITY_TASK_QUEUE`) | This worker's Temporal task queue | +| `CONTINUUM_CACHE_STORAGE_PATH` | `./.continuum-cache/workflow-data` | Local staging folder for Parquet files | diff --git a/docs/contributing/create-new-nodes/step-7-build-and-test.md b/docs/contributing/create-new-nodes/step-7-build-and-test.md new file mode 100644 index 0000000..dc63809 --- /dev/null +++ b/docs/contributing/create-new-nodes/step-7-build-and-test.md @@ -0,0 +1,21 @@ +--- +sidebar_position: 8 +sidebar_label: "7. Build and test" +title: "Step 7: Build and test" +--- + +# Step 7: Build and test + +```bash +./gradlew build +``` + +This compiles every module, runs `validateContinuumNodeAnnotations`, runs the unit tests and builds `worker/build/libs/worker-.jar`. + +To check that the annotation guard works, swap `@ContinuumNode` for `@Component` and run `./gradlew classes`. The build fails with: + +```text +> Task :features:continuum-feature-demo:validateContinuumNodeAnnotations FAILED + Continuum Node Annotation Validation Failed + The following classes implement ContinuumNodeModel but are missing the @ContinuumNode annotation: +``` diff --git a/docs/contributing/create-new-nodes/step-8-run-locally.md b/docs/contributing/create-new-nodes/step-8-run-locally.md new file mode 100644 index 0000000..ac18d89 --- /dev/null +++ b/docs/contributing/create-new-nodes/step-8-run-locally.md @@ -0,0 +1,22 @@ +--- +sidebar_position: 9 +sidebar_label: "8. Run locally" +title: "Step 8: Run the worker against the local stack" +--- + +# Step 8: Run the worker against the local stack + +1. Start the platform with the [Local Setup](../../getting-started/local-setup.md) compose file and wait until every service is healthy. +2. Start your worker: + + ```bash + ./gradlew :worker:bootRun + ``` + +3. Check that the worker loaded your node. This endpoint returns the node tree that the worker serves: + + ```bash + curl http://localhost:8090/api/v1/node-repo + ``` + +4. Open the [workbench](http://127.0.0.1:3002). **Word Counter** appears under the **Demo** category. Drop it onto the canvas after a node that outputs a `text` column, such as **Create Table**, and run the workflow. diff --git a/docs/contributing/create-new-nodes/step-9-ship-it.md b/docs/contributing/create-new-nodes/step-9-ship-it.md new file mode 100644 index 0000000..8af4000 --- /dev/null +++ b/docs/contributing/create-new-nodes/step-9-ship-it.md @@ -0,0 +1,11 @@ +--- +sidebar_position: 10 +sidebar_label: "9. Ship it" +title: "Step 9: Ship it" +--- + +# Step 9: Ship it + +- **Container image:** set `jibToImage` in `worker/build.gradle.kts` and run `./gradlew :worker:jib`. Jib reads registry credentials from `DOCKER_REPO_USERNAME` / `DOCKER_REPO_PASSWORD`. The base image defaults to `eclipse-temurin:21-jre` and can be changed with `jibFromImage`. +- **Kubernetes:** deploy the worker as its own release in the platform's namespace and set the [environment variables](./step-6-worker-module.md#environment-variables) from Step 6, including a unique `CONTINUUM_NODE_TASK_QUEUE`. The charts in the existing feature repos (for example `continuum-feature-base/charts/`) are a working reference. See [Deployment](../../deployment/helm-charts.md). +- **Publishing feature modules:** the plugins configure Maven Central publishing through JReleaser by default. If you don't publish libraries, set `publishToMavenCentral.set(false)` in the `continuum { }` block. diff --git a/docs/contributing/create-new-nodes/troubleshooting.md b/docs/contributing/create-new-nodes/troubleshooting.md new file mode 100644 index 0000000..c714d8e --- /dev/null +++ b/docs/contributing/create-new-nodes/troubleshooting.md @@ -0,0 +1,18 @@ +--- +sidebar_position: 11 +sidebar_label: Troubleshooting +title: Troubleshooting +--- + +# Troubleshooting + +| Symptom | Cause and fix | +|---------|---------------| +| `Plugin [id: 'org.projectcontinuum.worker', version: '…'] was not found` | That version isn't on Maven Central (for example an old `-SNAPSHOT`), or `mavenCentral()` is missing from `pluginManagement.repositories`. Use a released `continuumPlatformVersion`. See [Step 1](./step-1-root-project.md). | +| `Unresolved reference 'databind'` / `'ObjectMapper'` | You're using Jackson 2 imports (`com.fasterxml.jackson.*`). Switch to `tools.jackson.core.type.TypeReference` and `tools.jackson.databind.ObjectMapper`. | +| `validateContinuumNodeAnnotations FAILED` | A class extends `ProcessNodeModel` or `TriggerNodeModel` without `@ContinuumNode`. | +| Worker starts but your node isn't listed | The `AutoConfiguration.imports` file is missing or points at the wrong class, or the node is outside the `AutoConfigure` package. See [Step 2](./step-2-feature-module.md#register-the-module-with-spring). | +| `Port 8080 was already in use` on `bootRun` | The local API server already uses 8080. Set `server.port` (see [Step 6](./step-6-worker-module.md)). | +| Node fails at random, or runs on the wrong worker | Several workers share a task queue. Set a unique `CONTINUUM_NODE_TASK_QUEUE`. | +| `IllegalArgumentException: Unsupported type` while writing | A cell value isn't one of the [supported types](./step-3-write-the-node.md#reading-and-writing-rows). | +| Help page reads "Documentation not found" | The `.doc.md` path doesn't match the node's package and class name. See [Step 4](./step-4-node-documentation.md). | diff --git a/docs/contributing/feature-template.md b/docs/contributing/feature-template.md index 9b1bff1..6e63fb4 100644 --- a/docs/contributing/feature-template.md +++ b/docs/contributing/feature-template.md @@ -9,6 +9,10 @@ Continuum is designed to be extended by **adding new workers**, not modifying ex To build your own set of nodes, start from [`continuum-feature-template`](https://github.com/projectcontinuum-org/continuum-feature-template) — a scaffold for a standalone Gradle project with the Spring Boot worker setup, Temporal registration, and `@ContinuumNode` conventions already wired up. +:::tip +The template builds against platform `0.0.20` out of the box and includes a Kotlin and a Java example node, CI/CD workflows, a Helm chart and a local Docker stack. Its README covers renaming it for your own feature. For a walkthrough of writing a node from scratch, see [Create New Nodes](./create-new-nodes/overview.md). +::: + Once your worker is running and registering nodes, it can be deployed as an independent release into the same Kubernetes namespace as the core platform (see [Deployment](../deployment/helm-charts.md)) — no changes to `continuum-platform-core` or any other feature repo required. For questions or to report issues, use the [GitHub issue tracker](https://github.com/projectcontinuum-org/continuum-platform-core/issues) on the core repo.