interpreter: implement RS regexp and blank-line record splitting by Franklin-Qi · Pull Request #133 · uutils/awk

Franklin-Qi · 2026-10-09T08:50:58Z

Wire RT as a builtin and finish the missing RS modes (multi-char regexp and RS="") so record boundaries and RT match gawk.

Closes: #124

github-actions · 2026-10-09T08:53:38Z

GNU awk testsuite comparison:

Test results comparison:
  Current:   TOTAL: 666 / PASSED: 97 / FAILED: 471 / SKIPPED: 98
  Reference: TOTAL: 666 / PASSED: 90 / FAILED: 478 / SKIPPED: 98

Changes from main branch:
  TOTAL: +0
  PASSED: +7
  FAILED: -7

New test failures (1):
  - rscompat

Test improvements (8):
  + fpat8
  + fsnul1
  + leadnl
  + nlinstr
  + nulrsend
  + onlynl
  + rs
  + rsnul1nl

github-actions · 2026-10-09T09:14:55Z

GNU awk testsuite comparison:

Test results comparison:
  Current:   TOTAL: 666 / PASSED: 98 / FAILED: 470 / SKIPPED: 98
  Reference: TOTAL: 666 / PASSED: 90 / FAILED: 478 / SKIPPED: 98

Changes from main branch:
  TOTAL: +0
  PASSED: +8
  FAILED: -8

Test improvements (8):
  + fpat8
  + fsnul1
  + leadnl
  + nlinstr
  + nulrsend
  + onlynl
  + rs
  + rsnul1nl

github-actions · 2026-10-10T10:29:18Z

GNU awk testsuite comparison:

Test results comparison:
  Current:   TOTAL: 666 / PASSED: 97 / FAILED: 471 / SKIPPED: 98
  Reference: TOTAL: 666 / PASSED: 90 / FAILED: 478 / SKIPPED: 98

Changes from main branch:
  TOTAL: +0
  PASSED: +7
  FAILED: -7

New test failures (1):
  - rscompat

Test improvements (8):
  + fpat8
  + fsnul1
  + leadnl
  + nlinstr
  + nulrsend
  + onlynl
  + rs
  + rsnul1nl

github-actions · 2026-10-16T03:06:48Z

GNU awk testsuite comparison:

Test results comparison:
  Current:   TOTAL: 666 / PASSED: 98 / FAILED: 470 / SKIPPED: 98
  Reference: TOTAL: 666 / PASSED: 90 / FAILED: 478 / SKIPPED: 98

Changes from main branch:
  TOTAL: +0
  PASSED: +8
  FAILED: -8

Test improvements (8):
  + fpat8
  + fsnul1
  + leadnl
  + nlinstr
  + nulrsend
  + onlynl
  + rs
  + rsnul1nl

Alonely0 · 2026-10-16T08:28:34Z

Sorry about the delay, I'm in EuroRust this week. Let me get the PRs reviewed today or this weekend.

Franklin-Qi · 2026-10-16T09:28:01Z

Sorry about the delay, I'm in EuroRust this week. Let me get the PRs reviewed today or this weekend.

@Alonely0 Wow, That’s great! Congratulations on attending the conference—please feel free to take care of this once you’re less busy. Btw, Europe is a great place, I was there just last year. Thanks again.

sylvestre · 2026-10-18T08:50:20Z

a bunch of jobs are failing

Alonely0

That's a lot of code hahah, github is going insane sobs. There's a bit more byte-slice indexing than I'd like, I'm slightly concerned we might split up UTF-8 codepoints.

Btw, Can you check the behavior is consistent with everything that's detailed here, especially about RS="" vs RS="\n\n+", and add tests for it? Thanks! https://www.gnu.org/software/gawk/manual/html_node/Multiple-Line.html#index-dark-corner-multiline-records

Also, I maintain the minrx & minrx-sys crates, so if there's a feature we could use there or something you'd like to see changed, lmk.

Alonely0 · 2026-10-19T07:54:59Z

+    /// Unconsumed input left after a regexp `RS` match; cleared on each new file.
+    rs_leftover: StdVec<u8>,
+    /// Cached regexp `RS` automaton (`pattern`, `regex`); invalidated on change.
+    rs_regex_cache: Option<(StdVec<u8>, minrx::Regex)>,


Let's try to drop rs_regex_cache-adjacent duplication. I'm afraid the TODO was partially stale, since I had been working to cache it at the type level (see vm/types.rs's AwkRegex struct). We should be able to do one of two things:

Reuse the actual RS in the symbols table by converting it to a regex, iff gawk does observably type-taint it to be a regex when matching (confirm by triggering at least one regex match w/ RS and doing a print typeof(RS)).

Store a new regex Value in rs_regex_cache: Option<Value>.

Either of the two allow us to add/reuse the methods of Value, instead of re-implementing its layout and functionality (just expose the extra methods you may need, like minrx::find_iter() if it's not there).

Alonely0 · 2026-10-19T07:57:22Z


-    pub fn read_record_blank_lines(&mut self, mut _reader: impl BufRead) -> Result<bool> {
-        todo!()
+fn find_delim(haystack: &[u8], delim: &[u8]) -> Option<usize> {


find_delim could use a one-line doc comment.

Alonely0 · 2026-10-19T07:57:53Z

+fn find_delim(haystack: &[u8], delim: &[u8]) -> Option<usize> {
+    match delim {
+        [] => None,
+        [b] => haystack.iter().position(|x| x == b),


This can use memchr::memchr to optimize it further. In fact, we can add match arms up to memchr3.

Alonely0 · 2026-10-19T08:13:27Z

+///
+/// Returns `(content_len, rt_len)` where `buf[..content_len]` is the record and
+/// `buf[content_len..content_len + rt_len]` is the run of separator newlines.
+fn blank_line_split(buf: &[u8]) -> Option<(usize, usize)> {


Would this be cleaner with iterators? I'm also kinda concerned about respecting UTF-8 character boundaries; even though we need to support invalid UTF-8, we still must respect those.

Alonely0 · 2026-10-19T08:41:24Z

a bunch of jobs are failing

Sorry for putting it off. Let me get it addressed this week; should only require setting up the compiler on windows and patching minrx on wasm.

Alonely0 · 2026-10-24T16:47:21Z

sorry, needs to be rebased!

Wire RT as a builtin and finish the missing RS modes (multi-char regexp and RS=""). Cache the RS automaton, keep reading when a match touches the buffer end, and consume rs_leftover when RS switches mid-file. Closes: uutils#124

github-actions · 2026-10-24T17:45:23Z

GNU awk testsuite comparison:

Test results comparison:
  Current:   TOTAL: 666 / PASSED: 110 / FAILED: 458 / SKIPPED: 98
  Reference: TOTAL: 666 / PASSED: 100 / FAILED: 468 / SKIPPED: 98

Changes from main branch:
  TOTAL: +0
  PASSED: +10
  FAILED: -10

Test improvements (10):
  + anchor
  + fpat8
  + fsnul1
  + leadnl
  + nlinstr
  + nulrsend
  + onlynl
  + rs
  + rsnul1nl
  + rsnulw