Repository navigation
blog: add post about Camel Core CI improvements - #1831
Conversation
|
🚀 Preview is available at https://pr-1831--camel.netlify.app |
davsclaus
left a comment
There was a problem hiding this comment.
TL;DR
Comment — a well-documented post on how the Camel Core CI went from a few green builds a year to several a day. The front matter, images, tables and reference-style Develocity links all render correctly in the Netlify preview, and the PR counts I checked match GitHub (3,326 merged on main in 2025, 415 backports to 4.18.x in 2026). Ready to merge once the numbers question below is settled.
Blockers
- None
Other findings: 1 question (build-time figures), 2 markup suggestions, 2 optional content ideas. See the inline comments.
Thanks for writing this up, Aurélien. The before/after tables and the Map.<String, String> entry witness trick are really useful to share.
This review covers project conventions and content checks only. It is not a substitute for specialized review tools or static analysis.
This review was generated by an AI agent and may contain inaccuracies. Please verify all suggestions before applying.
| | Metric | A year ago | Today | | ||
| |--------|-----------|-------| | ||
| | Green Jenkins builds on main | 1 to 4 per year | several per days | | ||
| | Duration of the Jenkins build for main | 5h45| 1h24 | |
There was a problem hiding this comment.
Question (non-blocking): the build times in the post don't seem to match each other:
- here: main build 5h45 → 1h24
- the
-T1Ctable further down: ppc64le 5h45 → 1h41, x86_64 with Sonar 5h20 → 1h47, x86_64 without Sonar 4h09 → 56min - "What's next": "our 1h25 build with Sonar"
The 5h45 matches ppc64le, but 1h24 is none of the "after" values. If 1h24 is the wall-clock time of the whole matrix and the table shows single cells, or the numbers were measured at different times, one sentence saying so would help. Otherwise readers may think one of the figures is a mistake.
There was a problem hiding this comment.
this top-level includes several improvements, not only the -T1C, which explains the difference with the other specific -T1C table
| * Lack of synchronization between expected state of the system and next steps in the test | ||
|
|
||
| Some generic solutions: | ||
| * Use of @Isolated, this is often the easy solution, at the cost of performance. |
There was a problem hiding this comment.
Suggestion (markup): the rest of the post puts annotations, flags and paths in code spans (`@Isolated` on line 241, `-T1C` in the headings), but a few are plain text: @Isolated here, -T1C on line 143, .m2 on line 84 and etc/scripts/regen.sh on line 132.
| * Use of @Isolated, this is often the easy solution, at the cost of performance. | |
| * Use of `@Isolated`, this is often the easy solution, at the cost of performance. |
There was a problem hiding this comment.
Done: @Isolated, -T1C, .m2 and etc/scripts/regen.sh are now in code spans.
|
|
||
| On top of that, we started hitting the **GitHub Actions limits**, which are **shared among the whole Apache GitHub organization**. The queue time can go up to dozens of hours on worst days, regularly dozens of minutes. On a 30 minutes job that's quite consequent and slowing down the development process. | ||
|
|
||
| It was time to give a bit more love to our CIs. The focus was first on Jenkins, but several actions benefitted to both of them. This is an on-going work but let's share some of the improvements results today as we reached an important milestone last week on Jenkins CI. |
There was a problem hiding this comment.
Optional idea: most readers of the blog use Camel and don't contribute to it, and for them the payoff of this work is indirect. A stable, faster CI is what makes the jump to 820 backports this year possible, and it lets LTS patch releases ship sooner with more fixes. A sentence here saying what this means for people who use Camel would give them a reason to keep reading. How to say it is your call.
There was a problem hiding this comment.
Good point. I added an introduction stating that a stable and fast CI is what allows a higher throughput of new features and CVE fixes, and that faster, more reliable feedback means more changes backported to the maintained branches and security fixes reaching users sooner.
| Some generic solutions: | ||
| * Use of @Isolated, this is often the easy solution, at the cost of performance. | ||
| * Increase timeout. Beware, it is often the first solution that AI agents provide, it is not that often working. Need to check first if the test is close to the timeout when passing, otherwise it is usually another problem and it will only increase the time before the test is failing. | ||
| * Use more conditional checks with `Awaitility` to ensure that systems are ready, or what we want to test is in the expected state |
There was a problem hiding this comment.
Optional idea: the Camel Integration Testing Pitfalls post from last week covers the route-testing side (timer-driven routes, eager initialization, weak assertions), so the two posts go well together. For example:
| * Use more conditional checks with `Awaitility` to ensure that systems are ready, or what we want to test is in the expected state | |
| * Use more conditional checks with `Awaitility` to ensure that systems are ready, or what we want to test is in the expected state | |
| For pitfalls in how the routes themselves are tested, see also [Camel Integration Testing Pitfalls](/blog/2026/10/camel-testing-pitfalls/). |
There was a problem hiding this comment.
Thanks for the pointer. I did not add the cross-link in this post, to keep it focused on the CI side.
| - Investigate/improve Sonar which is not leveraging parallelization or multi-core CPUs at all. Currently adding 30 minutes on our 1h25 build with Sonar. | ||
| - Investigate build and test time of modules on critical paths | ||
| - Investigate other Camel projects | ||
| - Apply new recommendation policies for GitHub Workflows which are emerging in Apache community, for instance from discussions in Slack channel [#project-workflow-optimizations](https://the-asf.slack.com/archives/C0B835NQG6P) |
There was a problem hiding this comment.
Note (non-blocking): only people with an ASF Slack account can open the-asf.slack.com, so most readers will hit a login page. You could say that in the text, or link to a public thread on a mailing list if there is one.
There was a problem hiding this comment.
Done: the text now says it is the ASF Slack and requires an Apache account or an invitation.
53115a4 to
e208820
Compare
| - (Re)activate test parallelization for some modules where it was disabled. Can be both from Maven JUnit config or with `@Isolated` or even `@Order` JUnit annotations. | ||
| - Tweak test parallelization with the best number of threads and forks. | ||
| - Investigate/improve Sonar which is not leveraging parallelization or multi-core CPUs at all. Currently adding 30 minutes to our 1h25 build with Sonar. | ||
| - Investigate build and test time of modules on critical paths. A [critical path analysis](https://issues.apache.org/jira/browse/CAMEL-24290?focusedId=18143592&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-18143592) based on Develocity build scans lists the most interesting candidates. | ||
| - Investigate other Camel projects | ||
| - Apply new recommendation policies for GitHub Workflows which are emerging in Apache community, for instance from discussions in Slack channel [#project-workflow-optimizations](https://the-asf.slack.com/archives/C0B835NQG6P) (ASF Slack, requires an Apache account or an invitation) | ||
| - Leverage the new [CI usage dashboard](https://ci-test.apache.org/#/overview?) to monitor the usage of our jobs and to spot the ones consuming the most shared resources |
There was a problem hiding this comment.
I have more ideas but I prefer to stay simple here
|
🚀 Preview is available at https://pr-1831--camel.netlify.app |
e208820 to
2711f21
Compare
|
🚀 Preview is available at https://pr-1831--camel.netlify.app |
Co-authored-by: IBM Bob 2.2.1 Signed-off-by: Aurélien Pupier <apupier@ibm.com>
2711f21 to
a020f66
Compare
|
🚀 Preview is available at https://pr-1831--camel.netlify.app |
No description provided.