[SYSTEMDS-3681] Add stepGLM builtin function. by Bruno-Ignaz · Pull Request #2523 · apache/systemds

Bruno-Ignaz · 2026-06-29T13:15:39Z

Hi,
This is a homework pull-requests for the course AMLs at TUB.
This is also my first pull-request :)

I refactored ./scripts/algorithm/StepGLM.dml into a importable function.

Over 900 lines of code removed
Replacement of global variables with function parameters
Replacement of custom GLM solver glm_fit with adapter that uses the builtin solver m_gml from ./scripts/builtin/gml.dml
Rewrite of the AIC calculation.

The probit inverse link function is now approximated (AIC calculation):

"it does not matter much whether one uses a probit model or a logit model, except in cases where data are heavily concentrated in the tails" (page 1487 in [1])

Additionally:

./scripts/algorithms/TestBuiltinStepGLM.dml performs two checks for function output.

Sources

[1]: Takeshi, Qualitative Response Models: A Survey (1981)

…nested blocks

codecov · 2026-06-30T09:02:14Z

Codecov Report

❌ Patch coverage is 78.61285% with 959 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.39%. Comparing base (b150d8e) to head (056c7f9).
⚠️ Report is 108 commits behind head on main.

Files with missing lines	Patch %	Lines
...ntime/ooc/cache/io/OOCBufferedDataInputStream.java	0.00%	143 Missing ⚠️
...g/apache/sysds/runtime/ooc/cache/OOCCacheImpl.java	70.53%	81 Missing and 61 partials ⚠️
.../org/apache/sysds/runtime/ooc/cache/OOCFuture.java	62.28%	54 Missing and 12 partials ⚠️
...instructions/ooc/AggregateUnaryOOCInstruction.java	42.64%	37 Missing and 2 partials ⚠️
...ime/ooc/cache/collections/MaskedOnceArrayList.java	65.17%	25 Missing and 14 partials ⚠️
...runtime/ooc/cache/collections/MaskedOnceArray.java	53.94%	26 Missing and 9 partials ⚠️
...oc/cache/collections/SegmentedStreamTableList.java	65.34%	27 Missing and 8 partials ⚠️
.../org/apache/sysds/runtime/io/FrameReaderDelta.java	82.51%	22 Missing and 10 partials ⚠️
...untime/instructions/ooc/TernaryOOCInstruction.java	32.60%	26 Missing and 5 partials ⚠️
...rg/apache/sysds/runtime/io/FrameReaderParquet.java	78.10%	16 Missing and 14 partials ⚠️
... and 75 more

Additional details and impacted files

@@             Coverage Diff             @@
##               main    #2523     +/-   ##
===========================================
  Coverage     71.38%   71.39%             
- Complexity    48753    50546   +1793     
===========================================
  Files          1571     1632     +61     
  Lines        188912   196238   +7326     
  Branches      37067    38201   +1134     
===========================================
+ Hits         134857   140095   +5238     
- Misses        43593    45129   +1536     
- Partials      10462    11014    +552

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

christinadionysio

Thank you for you PR, @Bruno-Ignaz!
I have a few minor things that should be improved before we can push your code into the upstream. It would be appreciated if you could have a look.

Bruno-Ignaz · 2026-08-16T21:32:08Z

Thanks @christinadionysio for the detailed feedback!
Currently, I'm busy preparing for the exam.
The week after the 25th, I'm probably less distracted :)

christinadionysio · 2026-09-01T12:07:48Z

@Bruno-Ignaz could you please look at the failing github action? After this is fixed, I will merge it in.

Wire startLocalFedMonitoring through FederatedWorkerUtils.waitForWorker so the monitoring backend's port-bind is polled instead of slept on (fixes flaky FederatedCoordinatorIntegrationCRUDTest), migrate FederatedLogicalTest to the bulk startLocalFedWorkers(int[]) API, and drop the now-unused FED_WORKER_WAIT_S and FED_MONITOR_WAIT constants.

…ts (apache#2472) FedWorkerReadMatrixCompress.verifyRead failed roughly once per ten component-test CI runs because it called FederatedTestUtils.wait(1000) to give the worker time to finish its async compression (kicked off by CompressedMatrixBlockFactory.compressAsync), then asserted that the returned block was a CompressedMatrixBlock. On a contended runner the 1 s sleep was not enough, the subsequent read returned the still- uncompressed block, and the assertion failed. Surefire's rerunFailingTestsCount=2 hid this as a "Flake" rather than a job failure. Add FedWorkerBase.awaitCompressed(long id), which polls getMatrixBlock at 25 ms intervals for up to COMPRESS_TIMEOUT_MS (10 s) and returns as soon as the worker reports the compressed form, or returns the last- observed block on timeout so the caller's assertion still produces a meaningful failure. Convert the three call sites that used the fixed-sleep anti-pattern: - FedWorkerReadMatrixCompress.verifyRead (the actual CI flake) - FedWorkerMatrixCompress.verifySameOrAlsoCompressedAsLocalCompress (polls only when local compresses, so the "do not compress" parametrization stays fast) - FedWorkerMatrixMultiplyWorkload.verifySameOrAlsoCompressedAsLocalCompress Remove the now-unused FederatedTestUtils.wait helper so the anti-pattern is harder to reintroduce.

Fix intermittent ~26 minute hangs in the **.component.c**.** GitHub Actions job. The forks were finishing their test classes but failing to exit cleanly (leaked non-daemon threads from test executors), and Surefire 3.0.0 did not reliably enforce its fork shutdown timeout.

Bumps [actions/setup-node](https://github.com/actions/setup-node) from 6 to 7. - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](actions/setup-node@v6...v7) --- updated-dependencies: - dependency-name: actions/setup-node dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Rewrites the Parquet frame reader to read columns via parquet's column API. Co-authored-by: Jakob-al28 <jakob28@mail.uni-paderborn.com>

Add three scenarios only an external engine can set up: a log that has rolled into a parquet checkpoint with a delete after it, DATE/TIMESTAMP/DECIMAL columns that have no SystemDS value type and must be rejected on the Delta type and a table SystemDS creates that Spark then appends to. Also stop countParquet from counting checkpoint files as data files.

This patch adds a min-max variance measure to Scuro's hyperparameter tuning functionality in order to avoid overfitting to the validation score. Additionally, it adds the missing parameter setters to multiple representations.

Closes apache#2579.

Assisted-by: AI

In this patch the node executor for Representation DAGs is improved and made mode efficient. Additionally, the code was cleaned up. Assisted-by: AI

Assisted-by: AI

Closes 2499. Signed-off-by: Grigorii Turchenko <turchenko.gregory@gmail.com>

Assisted-by: AI

…d test environments Closes apache#2587.

On newer windows version, the check for physical memory via wmic runs into a "cannot run program" error on hard crashes. This patch adds a more robust version that checks the modern way, old way, and in all cases prevents any crash because this physical memory size is just used for warnings that SystemDS runs in a JVM with too little memory.

Bumps [docker/login-action](https://github.com/docker/login-action) from 4.5.2 to 4.6.0. - [Release notes](https://github.com/docker/login-action/releases) - [Commits](docker/login-action@v4.5.2...v4.6.0) --- updated-dependencies: - dependency-name: docker/login-action dependency-version: 4.6.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

This patch improves the representation DAGs, reduces duplicates and finds good initial parameters for window operators. Additionally, it collects more statistics of the optimization runs. Assisted-by: AI

This patch adds new representations for physiological data modalities, including new aggregators and context operations. Additionally, it refines existing operators with additional batch functionality and memory estimations. Assisted-by: AI

This patch adds improvements to multiple text, visual and audio representations. It includes a LazyLoading approach for visual modalities where the raw data is read when needed in the CustomDataloader. It adds two new representations: OpenFace and ImageBind. Assisted-by: AI

In this patch a new multimodal GA optimizer is added that implements various mutation operations. It reuses the worker pool and shared memory modules of the node executor. The GA optimizer uses the deap library. Assisted-by: AI

This patch adds functionality to apply representations on the test set only and perform inference on it. It measures the performance metrics as well as runtime for each step. Assisted-by: AI

This reverts commit 55e6bf6.

Bruno-Ignaz · 2026-09-01T13:35:09Z

These aren't my changes in the conflict. Can I just accept all the incoming ones?

This reverts commit 3962ff1.

Bruno-Ignaz added 9 commits May 24, 2026 10:35

refactor StepGLM.dml from a script into a function and resolve a few …

274f0c8

…nested blocks

rename builtin StepGLM.dml to stepglm.dml

78018f9

rename builtin stepglm.dml to stepGLM.dml

739c5d2

fix syntax errors in stepGLM

9dc956b

fix glm_fit call syntax

e60c2dd

remove fit_glm

9de039d

replace glm_fit with m_glm

482349c

add missing glm_fit replacment

8e24e9c

edit test script

ca5429b

github-project-automation Bot added this to SystemDS PR Queue Jun 29, 2026

github-project-automation Bot moved this to In Progress in SystemDS PR Queue Jun 29, 2026

add license to TestBuiltinStepGLM.dml

3321316

Bruno-Ignaz added 5 commits July 1, 2026 15:31

rename stepGLM to m_stepGLM; register stepGLM in Builtins.java

e13e83a

add BuiltinSTEPGlmTest.java

c341063

indent BuiltinSTEPGlmTest.java

8887e5a

replace spaces with tabs in BuiltinSTEPGlmTest.java

159fa87

format BuiltinSTEPGlmTest.java via jdtls

b7e64b8

christinadionysio requested changes Aug 11, 2026

View reviewed changes

github-project-automation Bot moved this from In Progress to In Review in SystemDS PR Queue Aug 11, 2026

Bruno-Ignaz added 4 commits August 30, 2026 23:12

remove print statments

59b8d88

remove comment

bc4dfe2

rm old stepGLM.dml script

69ccca9

replace misleading comments

8aa902b

Baunsgaard and others added 4 commits September 1, 2026 14:25

[MINOR] Fix incorrect formatting of various tests

9876057

dependabot Bot and others added 25 commits September 1, 2026 14:25

[SYSTEMDS-3929] Speed up Parquet frame reader/writer (apache#2528)

08249af

Rewrites the Parquet frame reader to read columns via parquet's column API. Co-authored-by: Jakob-al28 <jakob28@mail.uni-paderborn.com>

[SYSTEMDS-3958] Add XML options for sparsity rewrites.

9e9902a

Closes apache#2579.

[SYSTEMDS-3891] Wire OOC Instructions with New Primitives

b333755

[SYSTEMDS-3891] OOC Non-blocking Join and Primitive Simplification

6077976

[SYSTEMDS-3956] Add AI Policy

e672d4f

Assisted-by: AI

[SYSTEMDS-3940] Improve Scuro Node Executor

2b6212f

In this patch the node executor for Representation DAGs is improved and made mode efficient. Additionally, the code was cleaned up. Assisted-by: AI

[SYSTEMDS-3956] Extend AI Policy for IP Provenance

6401a62

[SYSTEMDS-3891] Wire OOC Aggregate Unary and Bugfix Race Condition

e2e1f5d

Assisted-by: AI

[SYSTEMDS-3863] Add PowerTransformer built-in functions

e1376a0

Closes 2499. Signed-off-by: Grigorii Turchenko <turchenko.gregory@gmail.com>

[SYSTEMDS-3891] Add Generic OOC Reduce

b810e54

Assisted-by: AI

[SYSTEMDS-3891] Add NaryJoinOOCPrimitive and Generic Join

88309f0

Assisted-by: AI

[SYSTEMDS-3891] Add GeneralMMultOOCPrimitive

9b0584c

Assisted-by: AI

[SYSTEMDS-3960] Improve robustness against network delays in federate…

3f6326a

…d test environments Closes apache#2587.

[SYSTEMDS-3887] Improve Scuro DAGs

044f96f

This patch improves the representation DAGs, reduces duplicates and finds good initial parameters for window operators. Additionally, it collects more statistics of the optimization runs. Assisted-by: AI

[SYSTEMDS-3967] Add a multimodal GA optimizer approach

3254d94

In this patch a new multimodal GA optimizer is added that implements various mutation operations. It reuses the worker pool and shared memory modules of the node executor. The GA optimizer uses the deap library. Assisted-by: AI

[SYSTEMDS-3968] Add test set evaluator - apache#2602

b76bbd6

This patch adds functionality to apply representations on the test set only and perform inference on it. It measures the performance metrics as well as runtime for each step. Assisted-by: AI

dev/format-changed.sh

55e6bf6

Revert "dev/format-changed.sh"

056c7f9

This reverts commit 55e6bf6.

Bruno-Ignaz added 2 commits September 1, 2026 16:28

dev/format-changed.sh

3962ff1

Revert "dev/format-changed.sh"

325d24f

This reverts commit 3962ff1.

Bruno-Ignaz force-pushed the main branch from 8f1a738 to 325d24f Compare September 1, 2026 14:53

Merge branch 'main' into main

35cb6a7

Navigation Menu

Uh oh!

[SYSTEMDS-3681] Add stepGLM builtin function. - #2523

[SYSTEMDS-3681] Add stepGLM builtin function.#2523
Bruno-Ignaz wants to merge 132 commits into
apache:mainfrom
Bruno-Ignaz:main

Bruno-Ignaz commented Jun 29, 2026

Uh oh!

codecov Bot commented Jun 30, 2026 •

edited

Loading

Uh oh!

christinadionysio left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Bruno-Ignaz commented Aug 16, 2026

Uh oh!

christinadionysio commented Sep 1, 2026

Uh oh!

Bruno-Ignaz commented Sep 1, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

15 participants

Uh oh!

Conversation

Bruno-Ignaz commented Jun 29, 2026

Sources

Uh oh!

codecov Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

christinadionysio left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Bruno-Ignaz commented Aug 16, 2026

Uh oh!

christinadionysio commented Sep 1, 2026

Uh oh!

Bruno-Ignaz commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

15 participants

codecov Bot commented Jun 30, 2026 •

edited

Loading

Bruno-Ignaz commented Sep 1, 2026 •

edited

Loading