diff --git a/CHANGELOG.md b/CHANGELOG.md index 1753fda6..af84e19f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,12 @@ because it turns other people's test suites red. ### Breaking +- **A TIFF of an odd size is not byte for byte what 0.5.0 wrote.** Its + directory, which says where the pixels are, came last and so started on an + odd byte, where the TIFF specification has it start on an even one. Every + reader tried took it, but a strict one may not. It now stands right after + the eight byte header, and the pixels are the same. A TIFF of an even size is + unchanged. - **Some AVIF pictures and WebM and MP4 films are not byte for byte what 0.5.0 wrote.** The AV1 encoder underneath, gav1d, is now 0.2.6. Where the right or bottom edge of a picture cuts through a block, the encoder now @@ -87,8 +93,8 @@ because it turns other people's test suites red. outcome is `unspecified`, offers both answers in the new `policy_options` field and says why. `tfg damage ` lists those formats. - **A file cut short says where the cut fell and how much of its content is - left.** In some formats a large file is mostly padding beside its content, - so a cut can take all of the content or none of it. The `truncate` entry in + left.** A file of a few kilobytes cut by 1% or 2% can lose less of its + content than that, or none of it. The `truncate` entry in the manifest's `damage` now carries `cut`: `in` (`content`, `padding`, `structure` or `end`) and `content_kept_bytes`, and `content_bytes` where the length of the content is known. A cut in the padding also gets the @@ -138,6 +144,13 @@ because it turns other people's test suites red. cut. Chrome and Firefox stop a cut MP4 with an error and end a cut WebM without one. The manifest of a cut WebM now says how long its pictures are, where it said nothing. Films nothing cuts are unchanged. +- **`truncate` cuts the picture of a BMP, a TIFF or a WebP by the percentage + you set, also when you set its width and height.** With a small picture in a + large file, a cut used to take only padding and leave a WebP whole, or take + the whole picture of a BMP or a TIFF. Now `--damage truncate:missing=50` + removes half of the pixels. Pillow shows the rows left of a cut BMP or TIFF, + and Preview on a Mac opens a cut TIFF without a warning. Files nothing cuts + are unchanged. - **tfg is on npm, for tests in Node.** `npm install --save-dev testing-files-generator` gives Playwright, Vitest diff --git a/README.md b/README.md index 69f91854..27d7c4b5 100644 --- a/README.md +++ b/README.md @@ -522,11 +522,12 @@ how many more there are, and the manifest lists every one. The label is still written before the damage, so the damage can overwrite it, and the manifest does not promise it is there. -In most formats `truncate` removes the same percentage of the content as of the -file: `missing=50` leaves about the top half of a PNG's picture, half of a WAV's sound -or half of a text's lines, the way an interrupted upload does. In some formats a -large file is mostly padding beside its content, so a cut there can take all of the -content or none of it. The manifest says where each cut fell: the `truncate` entry +`truncate` removes the same percentage of the content as of the file: +`missing=50` leaves about the top half of a PNG's picture, half of a WAV's sound +or half of a text's lines, the way an interrupted upload does. A file of a few +kilobytes cut by 1% or 2% can lose less of its content than that, or none of it +when the cut falls in what the format puts after the content, such as a ZIP's +list of its files. The manifest says where each cut fell: the `truncate` entry carries `cut`, with `in` (`content`, `padding`, `structure` or `end`) and `content_kept_bytes`, and with `content_bytes`, how long the content of the whole file is, where that is known. A cut in the padding also gets the note diff --git a/internal/damage/truncate.go b/internal/damage/truncate.go index b387b8af..dd24cec1 100644 --- a/internal/damage/truncate.go +++ b/internal/damage/truncate.go @@ -49,17 +49,23 @@ func init() { Min: fewestMissing, Max: mostMissing, Default: strconv.Itoa(defaultMissing), // The last two sentences are what a person cannot see from the - // outside. In most formats the cut removes the same percentage - // of the picture, sound or text, png among them since it lays - // its padding out for the cut. In the rest the padding stands - // only on one side of the content, so a large file cut at any - // share keeps all of it or none (docs/TRUNCATE-2026-11-11.md - // section 7.1, and notYetCutAtTheShare in internal/guard for - // which). The manifest says per file where the cut fell. + // outside. Every format lays its padding out for the cut, so the + // cut removes the same percentage of the content + // (docs/TRUNCATE-2026-11-11.md sections 7 to 15, and + // TestACutTakesTheShareOfTheContentItIsAskedFor). In a file of + // a few kilobytes it can take less: a WebP with less than 26 B + // of padding cannot put any in front of its picture and keeps up + // to that much more of it, and where what a format puts after + // its content - the central directory of a ZIP or an Office + // document, that WebP's padding - is longer than a cut of 1% or + // 2%, the cut falls in it and leaves the content whole + // (docs/TRUNCATE-2026-11-11.md section 15.4, measured in + // docs/REVIEW-244-2026-11-14.md). The manifest says per file + // where the cut fell. Detail: "How much of the whole file is gone, as a percentage. " + "At least this much is cut - a few bytes more where the format cannot end exactly there. " + - "In most formats the cut removes the same percentage of the content. " + - "In some, a large file is mostly padding beside its content, so the cut can take all of the content or none - " + + "The cut removes the same percentage of the content - of the picture, the sound, the pages or the text. " + + "A file of a few kilobytes cut by 1% or 2% can lose less of its content than that, or none of it when the cut falls in what the format puts after the content, such as a ZIP's list of its files - " + "the manifest says what each cut left.", }, }, diff --git a/internal/format/bmp/bmp.go b/internal/format/bmp/bmp.go index 6ab69d10..ae70fbeb 100644 --- a/internal/format/bmp/bmp.go +++ b/internal/format/bmp/bmp.go @@ -105,6 +105,9 @@ type memo struct { label string // gap is how many bytes sit between the header and the pixel data. gap int64 + // back is how many bytes of padding follow the pixels, only in a file + // laid out for a cut (splitForACut). + back int64 } // stride is the length of one row on disk. Every row is rounded up to a @@ -150,6 +153,9 @@ func (generator) Plan(r format.Request) (format.Plan, error) { } m := memo{width: w, height: h, seed: r.Seed, label: label, gap: r.Bytes - bare} + if r.CutAt > 0 { + splitForACut(&m, r.Bytes, r.CutAt) + } p := format.Plan{ Bytes: r.Bytes, @@ -290,10 +296,13 @@ func (generator) Write(ctx context.Context, w io.Writer, p format.Plan) error { } // The map, all of it known from the plan: the two headers, the gap that - // makes up the size, and the pixels. The gap stands BEFORE the pixels. + // makes up the size, and the pixels. The gap stands BEFORE the pixels, + // and in a file laid out for a cut the rest of the padding behind them. + pixels := pixelBytes(m.width, m.height) format.Mark(w, format.RoleStructure, 0, headers) format.Mark(w, format.RolePadding, headers, m.gap) - format.Mark(w, format.RoleContent, headers+m.gap, pixelBytes(m.width, m.height)) + format.Mark(w, format.RoleContent, headers+m.gap, pixels) + format.Mark(w, format.RolePadding, headers+m.gap+pixels, m.back) if err := writeHeaders(w, m, p.Bytes); err != nil { return err @@ -301,7 +310,10 @@ func (generator) Write(ctx context.Context, w io.Writer, p format.Plan) error { if err := writeGap(ctx, w, m.seed, m.gap); err != nil { return err } - return writePixels(ctx, w, m) + if err := writePixels(ctx, w, m); err != nil { + return err + } + return writeGap(ctx, w, m.seed+1, m.back) } func writeHeaders(w io.Writer, m memo, total int64) error { diff --git a/internal/format/bmp/cut.go b/internal/format/bmp/cut.go new file mode 100644 index 00000000..4162f19e --- /dev/null +++ b/internal/format/bmp/cut.go @@ -0,0 +1,20 @@ +package bmp + +import "github.com/donislawdev/TestingFilesGenerator/internal/format" + +// splitForACut splits the padding of a BMP around its pixels for a cut after +// its first cutAt bytes (format.Request.CutAt), so that the cut keeps the +// share of the pixels it keeps of the whole file: in front of them, in the gap +// bfOffBits points past, as much as the share wants, and the rest behind them, +// where no field points. It is raw bytes on either side, so every length +// lands. Ten decoders give the pixels of a BMP nothing cuts from the gap split +// and from all of it behind, Go's x/image/bmp only with no gap in front - it +// refuses every BMP with one (docs/TRUNCATE-2026-11-11.md section 15.3). +// +// A BMP nothing cuts keeps all of its padding in front, byte for byte what it +// was (D11). +func splitForACut(m *memo, total, cutAt int64) { + pad := m.gap + m.gap = format.SplitFront(pad, headers, pixelBytes(m.width, m.height), cutAt, total, 1) + m.back = pad - m.gap +} diff --git a/internal/format/tiff/cut.go b/internal/format/tiff/cut.go new file mode 100644 index 00000000..3769a662 --- /dev/null +++ b/internal/format/tiff/cut.go @@ -0,0 +1,73 @@ +package tiff + +import ( + "context" + "io" + + "github.com/donislawdev/TestingFilesGenerator/internal/format" +) + +// layOut says where a TIFF's directory and padding go. A TIFF a damage will cut +// is laid out for the cut (splitForACut). One of an odd size puts its directory +// right behind the header, with all of its padding still in front of the +// pixels: TIFF 6.0 starts a directory on a word boundary, and last, where the +// 172 B of the directory and its values end the file, it would start on an odd +// byte (O342). Nine decoders give the same pixels from that layout +// (docs/TRUNCATE-2026-11-11.md section 15.3). The rest keep the directory last. +func layOut(m *memo, total, cutAt int64) { + switch { + case cutAt > 0: + splitForACut(m, total, cutAt) + case total%2 == 1: + m.directoryFirst = true + } +} + +// splitForACut lays a TIFF out for a cut after its first cutAt bytes +// (format.Request.CutAt), so that the cut keeps the share of the pixels it +// keeps of the whole file. +// +// The directory and the values it points at move up to right behind the +// header. TIFF 6.0 lets the directory stand anywhere after the header on a +// word boundary, and byte 8 is one. Left last, it is the end of the file: 172 +// B that a cut of 1% of a file of a few kilobytes falls in. The padding then +// goes around the pixels - in front of them as much as the share wants, the +// rest behind them, where no field points. It is raw bytes on either side, so +// every length lands. Nine decoders give the pixels of a TIFF nothing cuts +// from the directory first with the padding in front, split and behind, and +// exiftool reads it (docs/TRUNCATE-2026-11-11.md section 15.3). +// +// A TIFF nothing cuts keeps its directory last and its padding in front, byte +// for byte what it was (D11). +func splitForACut(m *memo, total, cutAt int64) { + pad := m.gap + m.directoryFirst = true + m.gap = format.SplitFront(pad, header+directory+heap, pixelBytes(m.width, m.height), cutAt, total, 1) + m.back = pad - m.gap +} + +// writeDirectoryFirst writes a TIFF whose directory follows its header - one +// laid out for a cut, or one of an odd size: the header, the directory and its +// values, the padding in front, the pixels and the padding behind. +func writeDirectoryFirst(ctx context.Context, w io.Writer, m memo) error { + head := int64(header + directory + heap) + pixels := pixelBytes(m.width, m.height) + format.Mark(w, format.RoleStructure, 0, head) + format.Mark(w, format.RolePadding, head, m.gap) + format.Mark(w, format.RoleContent, head+m.gap, pixels) + format.Mark(w, format.RolePadding, head+m.gap+pixels, m.back) + + if err := writeHeader(w, m); err != nil { + return err + } + if err := writeDirectory(w, m); err != nil { + return err + } + if err := writeGap(ctx, w, m.seed, m.gap); err != nil { + return err + } + if err := writePixels(ctx, w, m); err != nil { + return err + } + return writeGap(ctx, w, m.seed+1, m.back) +} diff --git a/internal/format/tiff/pixels.go b/internal/format/tiff/pixels.go new file mode 100644 index 00000000..776a4f7f --- /dev/null +++ b/internal/format/tiff/pixels.go @@ -0,0 +1,93 @@ +package tiff + +import ( + "context" + "image" + "image/color" + "io" + + "github.com/donislawdev/TestingFilesGenerator/internal/format/imagelabel" +) + +// writePixels streams the picture a row at a time. +// +// Only the label band is ever built as an image, and it is at most 24 rows +// tall whatever the picture is. Everything else comes from the formula +// straight into one reused row buffer, so a 64 MiB picture costs a row rather +// than 64 MiB. +// +// A TIFF stores its rows top down, so the label band is written first - the +// one place this is simpler than BMP, which stores them the other way up. +func writePixels(ctx context.Context, w io.Writer, m memo) error { + row := make([]byte, m.width*samplesPerPixel) + off := int(m.seed % 256) + + band := labelBand(m, off) + bandH := 0 + if band != nil { + bandH = band.Bounds().Dy() + } + + for y := 0; y < m.height; y++ { + if y%64 == 0 { + select { + case <-ctx.Done(): + return ctx.Err() + default: + } + } + if y < bandH { + copyBandRow(row, band, y, m.width) + } else { + fillRow(row, y, m.width, off) + } + if _, err := w.Write(row); err != nil { + return err + } + } + return nil +} + +// fillRow writes one row of the gradient in the order a TIFF wants it: red, +// green, blue. +func fillRow(row []byte, y, width, off int) { + for x := 0; x < width; x++ { + row[x*3] = uint8((x + off) % 256) + row[x*3+1] = uint8((y + off) % 256) + row[x*3+2] = uint8((x + y + off) % 256) + } +} + +func copyBandRow(row []byte, band *image.RGBA, y, width int) { + for x := 0; x < width; x++ { + i := band.PixOffset(x, y) + row[x*3] = band.Pix[i] + row[x*3+1] = band.Pix[i+1] + row[x*3+2] = band.Pix[i+2] + } +} + +// labelBand rasterises the top strip of the picture, label included, or +// returns nil when there is no label to draw. +func labelBand(m memo, off int) *image.RGBA { + if m.label == "" || !imagelabel.Fits(m.width, len(m.label)) { + return nil + } + h := maxBandHeight + if m.height < h { + h = m.height + } + img := image.NewRGBA(image.Rect(0, 0, m.width, h)) + for y := 0; y < h; y++ { + for x := 0; x < m.width; x++ { + img.SetRGBA(x, y, color.RGBA{ + R: uint8((x + off) % 256), + G: uint8((y + off) % 256), + B: uint8((x + y + off) % 256), + A: 255, + }) + } + } + imagelabel.Draw(img, m.label) + return img +} diff --git a/internal/format/tiff/tiff.go b/internal/format/tiff/tiff.go index e0e25fcc..58e3e5eb 100644 --- a/internal/format/tiff/tiff.go +++ b/internal/format/tiff/tiff.go @@ -20,15 +20,16 @@ // itself talks about: everything before that offset is space the file // describes rather than space it keeps quiet about - the same shape as // bfOffBits in BMP. Measured on five independent readers at every size from -// 1 B to 10 MiB (docs/MVP-FORMATS.md section 2.11). +// 1 B to 10 MiB (docs/MVP-FORMATS.md section 2.11). A TIFF a damage will cut +// puts its directory first and splits the padding around the pixels instead +// (cut.go), and one of an odd size puts its directory first, so it starts on +// a word boundary. package tiff import ( "context" "encoding/binary" "fmt" - "image" - "image/color" "io" "github.com/donislawdev/TestingFilesGenerator/internal/core" @@ -148,8 +149,16 @@ type memo struct { width, height int seed uint64 label string - // gap is how many bytes sit between the header and the pixel data. + // gap is how many bytes sit between the header and the pixel data, or + // between the directory's values and the pixel data when the directory + // comes first. gap int64 + // back is how many bytes of padding follow the pixels, only in a file + // laid out for a cut (splitForACut). directoryFirst says the directory + // follows the header, in a file laid out for a cut and in one of an odd + // size. + back int64 + directoryFirst bool } // pixelBytes is the length of the image data. A TIFF row is not rounded up to @@ -190,6 +199,7 @@ func (generator) Plan(r format.Request) (format.Plan, error) { } m := memo{width: w, height: h, seed: r.Seed, label: label, gap: r.Bytes - bare} + layOut(&m, r.Bytes, r.CutAt) p := format.Plan{ Bytes: r.Bytes, @@ -327,6 +337,9 @@ func (generator) Write(ctx context.Context, w io.Writer, p format.Plan) error { return ctx.Err() default: } + if m.directoryFirst { + return writeDirectoryFirst(ctx, w, m) + } // The map, all of it known from the plan: the header, the gap that makes // up the size, the pixels, and the directory last, which says where they @@ -350,11 +363,17 @@ func (generator) Write(ctx context.Context, w io.Writer, p format.Plan) error { // pixelsAt is where the image data begins, which is what StripOffsets holds. func pixelsAt(m memo) int64 { + if m.directoryFirst { + return header + directory + heap + m.gap + } return header + m.gap } // directoryAt is where the IFD begins, which is what the header points at. func directoryAt(m memo) int64 { + if m.directoryFirst { + return header + } return pixelsAt(m) + pixelBytes(m.width, m.height) } @@ -465,86 +484,3 @@ func writeGap(ctx context.Context, w io.Writer, seed uint64, n int64) error { } return nil } - -// writePixels streams the picture a row at a time. -// -// Only the label band is ever built as an image, and it is at most 24 rows -// tall whatever the picture is. Everything else comes from the formula -// straight into one reused row buffer, so a 64 MiB picture costs a row rather -// than 64 MiB. -// -// A TIFF stores its rows top down, so the label band is written first - the -// one place this is simpler than BMP, which stores them the other way up. -func writePixels(ctx context.Context, w io.Writer, m memo) error { - row := make([]byte, m.width*samplesPerPixel) - off := int(m.seed % 256) - - band := labelBand(m, off) - bandH := 0 - if band != nil { - bandH = band.Bounds().Dy() - } - - for y := 0; y < m.height; y++ { - if y%64 == 0 { - select { - case <-ctx.Done(): - return ctx.Err() - default: - } - } - if y < bandH { - copyBandRow(row, band, y, m.width) - } else { - fillRow(row, y, m.width, off) - } - if _, err := w.Write(row); err != nil { - return err - } - } - return nil -} - -// fillRow writes one row of the gradient in the order a TIFF wants it: red, -// green, blue. -func fillRow(row []byte, y, width, off int) { - for x := 0; x < width; x++ { - row[x*3] = uint8((x + off) % 256) - row[x*3+1] = uint8((y + off) % 256) - row[x*3+2] = uint8((x + y + off) % 256) - } -} - -func copyBandRow(row []byte, band *image.RGBA, y, width int) { - for x := 0; x < width; x++ { - i := band.PixOffset(x, y) - row[x*3] = band.Pix[i] - row[x*3+1] = band.Pix[i+1] - row[x*3+2] = band.Pix[i+2] - } -} - -// labelBand rasterises the top strip of the picture, label included, or -// returns nil when there is no label to draw. -func labelBand(m memo, off int) *image.RGBA { - if m.label == "" || !imagelabel.Fits(m.width, len(m.label)) { - return nil - } - h := maxBandHeight - if m.height < h { - h = m.height - } - img := image.NewRGBA(image.Rect(0, 0, m.width, h)) - for y := 0; y < h; y++ { - for x := 0; x < m.width; x++ { - img.SetRGBA(x, y, color.RGBA{ - R: uint8((x + off) % 256), - G: uint8((y + off) % 256), - B: uint8((x + y + off) % 256), - A: 255, - }) - } - } - imagelabel.Draw(img, m.label) - return img -} diff --git a/internal/format/webp/cut.go b/internal/format/webp/cut.go new file mode 100644 index 00000000..82d7e7a9 --- /dev/null +++ b/internal/format/webp/cut.go @@ -0,0 +1,88 @@ +package webp + +import ( + "context" + "encoding/binary" + "io" + + "github.com/donislawdev/TestingFilesGenerator/internal/format" +) + +const ( + // vp8xBlock is the extended format's VP8X chunk whole: its header, and ten + // bytes of flags, reserved bits and the canvas size. + vp8xBlock = chunkHeader + 4 + 10 + // leastChunk is the smallest chunk there is: a header and nothing in it. + leastChunk = chunkHeader + 4 +) + +// splitForACut lays a WebP out for a cut after its first cutAt bytes +// (format.Request.CutAt), so that the cut keeps the share of the picture it +// keeps of the whole file, given the picture's chunk (picture bytes, its +// header included) and the padding settlePadding left behind it. +// +// Padding in front of the picture needs the extended format. RFC 9649 lets +// the simple one hold the picture's chunk and nothing else, and lets unknown +// chunks of the extended one appear out of order (section 2.7) - so a VP8X +// chunk first and the private chunk in front of the picture. Ten decoders +// give the pixels of a WebP nothing cuts from that, and eight of them refuse +// the private chunk in front without the VP8X (docs/TRUNCATE-2026-11-11.md +// section 15.3). +// +// The front is a chunk, so it is even and at least eight bytes, and the VP8X +// before it costs eighteen. It rounds up, so a cut takes at least the share, +// as format.SplitFront has it. Where the share wants nothing in front, or the +// padding is too short for the VP8X and a chunk, the file keeps the simple +// format with all of its padding behind the picture - then the cut keeps up +// to those 26 B more of the picture, and in a file of a few kilobytes cut by +// 1% it may fall in that padding. +// +// A WebP nothing cuts is byte for byte what it was (D11). +func splitForACut(m *memo, picture, total, cutAt int64) { + pad := total - riffHeader - picture + if cutAt <= 0 || cutAt >= total || pad < vp8xBlock+leastChunk || + format.PaddingInFront(riffHeader, picture, cutAt, total) <= 0 { + return + } + room := pad - vp8xBlock + front := max(leastChunk, format.SplitFront(room, riffHeader+vp8xBlock, picture, cutAt, total, leastChunk)) + front += front % 2 + if front > room { + front -= 2 + } + m.front = front + settlePadding(m, room-front) +} + +// inFront is what a WebP laid out for a cut puts between its RIFF header and +// its picture: the VP8X chunk, and the private chunk in front. A WebP nothing +// cuts puts neither. +func inFront(m memo) (structure, padding int64) { + if m.front == 0 { + return 0, 0 + } + return vp8xBlock, m.front +} + +// writeFront writes the VP8X chunk and the private chunk in front of the +// picture, when the plan has them. The flags stay clear - no colour profile, +// no transparency, no metadata, no animation - and the canvas is the +// picture's size. +func writeFront(ctx context.Context, w io.Writer, m memo) error { + if m.front == 0 { + return nil + } + var x [vp8xBlock]byte + copy(x[:4], "VP8X") + binary.LittleEndian.PutUint32(x[4:8], vp8xBlock-leastChunk) + putUint24(x[12:15], m.width-1) + putUint24(x[15:18], m.height-1) + if err := writeAll(w, x[:]); err != nil { + return err + } + return writeFiller(ctx, w, paddingName, m.seed+2, m.front-leastChunk) +} + +func putUint24(b []byte, v int) { + b[0], b[1], b[2] = byte(v), byte(v>>8), byte(v>>16) +} diff --git a/internal/format/webp/webp.go b/internal/format/webp/webp.go index 0c6ee09f..1faa320b 100644 --- a/internal/format/webp/webp.go +++ b/internal/format/webp/webp.go @@ -31,6 +31,8 @@ // private chunk, and one to seven bytes after the end of the RIFF payload // carry whatever is left. Measured on six readers (docs/MVP-FORMATS.md section // 2.13): there is no dead zone at all, which is better than PNG or GIF manage. +// A WebP a damage will cut moves part of the padding in front of the picture, +// in the extended format (cut.go). package webp import ( @@ -126,6 +128,10 @@ type memo struct { // which is not the same as one carrying nothing. chunk int64 tail int64 + // front is the private chunk in front of the picture, its header + // included, in a file laid out for a cut (splitForACut), and zero in one + // nothing cuts. + front int64 } // bareBytes is the file with no padding of any kind. @@ -167,6 +173,9 @@ func (generator) Plan(r format.Request) (format.Plan, error) { m := memo{width: w, height: h, seed: r.Seed, label: label} settlePadding(&m, r.Bytes-bare) + if r.CutAt > 0 { + splitForACut(&m, bare-riffHeader, r.Bytes, r.CutAt) + } p := format.Plan{ Bytes: r.Bytes, @@ -325,20 +334,24 @@ func (generator) Write(ctx context.Context, w io.Writer, p format.Plan) error { // The length of the bitstream is arithmetic, so the headers can go out // before a single pixel is coded and nothing has to be held back. stream := streamBytes(int64(m.width) * int64(m.height)) - body := chunkHeader + 4 + stream + stream%2 + picture := chunkHeader + 4 + stream + stream%2 + vp8x, front := inFront(m) + body := vp8x + front + picture if m.chunk >= 0 { body += chunkHeader + 4 + m.chunk } - // The map, all of it known from the plan: the container's header, the - // picture's chunk, and what makes up the size after it - the private - // chunk and the bytes past the container's end. - picture := chunkHeader + 4 + stream + stream%2 - format.Mark(w, format.RoleStructure, 0, riffHeader) - format.Mark(w, format.RoleContent, riffHeader, picture) - format.Mark(w, format.RolePadding, riffHeader+picture, p.Bytes-riffHeader-picture) + // The map, all of it known from the plan: the container's header (and + // the VP8X chunk of a file laid out for a cut), the private chunk in + // front, the picture's chunk, and what makes up the size after it - the + // private chunk and the bytes past the container's end. + head := riffHeader + vp8x + format.Mark(w, format.RoleStructure, 0, head) + format.Mark(w, format.RolePadding, head, front) + format.Mark(w, format.RoleContent, head+front, picture) + format.Mark(w, format.RolePadding, head+front+picture, p.Bytes-head-front-picture) - if err := writeContainer(w, body, stream); err != nil { + if err := writeContainer(ctx, w, m, body, stream); err != nil { return err } @@ -370,26 +383,32 @@ func (generator) Write(ctx context.Context, w io.Writer, p format.Plan) error { } // writeContainer emits everything before the first coded pixel: the RIFF -// header, the form name, and the header of the chunk the bitstream goes in. +// header, the form name, the VP8X chunk and the private chunk in front of a +// file laid out for a cut, and the header of the chunk the bitstream goes in. // // It is a function of its own rather than five more error checks inside Write, // because this project counts how many functions carry a lot of decisions and // Write was the sixth to reach the band. -func writeContainer(w io.Writer, body, stream int64) error { +func writeContainer(ctx context.Context, w io.Writer, m memo, body, stream int64) error { for _, part := range []struct { - text string - num uint32 + text string + num uint32 + front bool }{ {text: "RIFF"}, {num: uint32(4 + body)}, {text: "WEBP"}, + {front: true}, {text: "VP8L"}, {num: uint32(stream)}, } { var err error - if part.text != "" { + switch { + case part.front: + err = writeFront(ctx, w, m) + case part.text != "": err = writeAll(w, []byte(part.text)) - } else { + default: err = writeUint32(w, part.num) } if err != nil { diff --git a/internal/guard/cutmanifest_test.go b/internal/guard/cutmanifest_test.go index 1232ee63..424f8ae7 100644 --- a/internal/guard/cutmanifest_test.go +++ b/internal/guard/cutmanifest_test.go @@ -37,9 +37,12 @@ type cutCase struct { // the content is left, and gets a note when the cut missed the content. // // The shapes a cut takes (docs/TRUNCATE-2026-11-11.md sections 2, 7 and 8). -// A WebP grows its picture to fill the file and keeps what is left over -// behind it, which at 4 KiB is more than the 41 B a cut of 1% takes, so the -// cut leaves all of the picture. A PNG splits its padding around the picture +// A WebP of 305 B is 309 B whole when a cut takes 1%, and the 17 B of padding +// it keeps behind its picture cannot pay for what it would take in front - a +// VP8X chunk and a chunk, 26 B - so they stay behind it and the 4 B the cut +// takes are padding: the one shape left of a cut behind the content once +// every format splits its padding for a cut (docs/TRUNCATE-2026-11-11.md +// section 15.4). A PNG splits its padding around the picture // for a cut, so the cut takes the share asked for, and so does a tar.gz. A // text is its words, so the cut falls in them. Two cuts in a row end the file where the second one does, and // the cut is said of that one. A damage that cuts nothing has no cut to report. @@ -48,9 +51,9 @@ type cutCase struct { // one that claims content it does not have. func TestACutFileSaysWhereTheCutFellAndWhatIsLeft(t *testing.T) { cases := []cutCase{ - // A GIF, a PDF and an MP4 were this case until each split its - // padding for the cut. - {name: "behind", format: "webp", size: 4 << 10, missing: "1", in: "padding", kept: "all", note: "after the content"}, + // A GIF, a PDF, an MP4 and a WebP of 4 KiB were this case until each + // split its padding for the cut. + {name: "behind", format: "webp", size: 305, missing: "1", in: "padding", kept: "all", note: "after the content"}, {name: "picture", format: "png", size: 1 << 20, missing: "90", in: "content", kept: "share"}, // 4 KiB on disk is 40 KiB whole. It was the case of a cut in the padding // in front of the content, with an icon and a JPEG before it, until it @@ -181,7 +184,7 @@ func TestTheRecordOfACutRunIsNeverLongerThanItsForecast(t *testing.T) { } opt := engine.Options{OutDir: t.TempDir(), Seed: 7, Command: "test"} planned, err := engine.Plan([]engine.Target{ - cut("picture", "png", 1<<20, "90"), cut("behind", "webp", 4<<10, "1"), cut("text", "txt", 1000, "50"), + cut("picture", "png", 1<<20, "90"), cut("behind", "webp", 305, "1"), cut("text", "txt", 1000, "50"), }, opt) if err != nil { t.Fatalf("planning: %v", err) diff --git a/internal/guard/cutshare_test.go b/internal/guard/cutshare_test.go index 132628c7..c8f8167f 100644 --- a/internal/guard/cutshare_test.go +++ b/internal/guard/cutshare_test.go @@ -31,12 +31,10 @@ type notYet struct { // leaves the list, so the list cannot go stale in either direction. A format // that leaves is held to every size and share, and so is a new format unless // it is added here with its reason. -var notYetCutAtTheShare = map[string]notYet{ - // Both take the share at 64 KiB and above. At 4 KiB what stands behind the - // picture is more than a small share, so a cut of 1% keeps all of it. - "webp": {smallBehind, 4 << 10, 1}, - "tiff": {smallBehind, 4 << 10, 1}, -} +// +// Empty since webp, tiff and bmp split their padding for a cut +// (docs/TRUNCATE-2026-11-11.md section 15): every format takes the share. +var notYetCutAtTheShare = map[string]notYet{} // listedBeforeTheRule is every format notYetCutAtTheShare may name: the ones // that missed the share on 2026-11-12, when the owner decided that every format @@ -45,8 +43,8 @@ var notYetCutAtTheShare = map[string]notYet{ // grows, so a new format cannot wait on the ratchet - tools/doctor.py runs this // guard for it, and it stays red until the format splits its padding for // format.Request.CutAt. A format that leaves the ratchet leaves this list too, -// so it cannot go back. -var listedBeforeTheRule = []string{"webp", "tiff"} +// so it cannot go back - and every one has left it. +var listedBeforeTheRule []string // listedAfterTheRule is a problem for every format notYetAt names that // listedBeforeTheRule does not. @@ -61,10 +59,7 @@ func listedAfterTheRule(notYetAt map[string]notYet) []string { return out } -const ( - paddingBehind = "its padding stands only behind its content, so a large file cut at any share keeps all of it" - smallBehind = "in a small file the padding or directory behind its content is more than a small share" -) +const paddingBehind = "its padding stands only behind its content, so a large file cut at any share keeps all of it" // shareSlack is how far, in points of a percent, the share of the content a // cut takes may be from the share missing asks for. Rounding to whole chunks, @@ -114,6 +109,11 @@ func TestACutTakesTheShareOfTheContentItIsAskedFor(t *testing.T) { targets = append(targets, engine.Target{ID: id, Format: f.ID, Sizes: engine.Uniform(1, c[0]), Label: true, Damage: chain}) } } + sized := sizedTargets(missingOf) + if len(sized) == 0 { + t.Errorf("no format takes a width and a height, so no picture smaller than its file was cut") + } + targets = append(targets, sized...) opt := engine.Options{OutDir: t.TempDir(), Seed: 7, ManifestName: engine.DefaultManifestName} planned, err := engine.Plan(targets, opt) if err != nil { @@ -155,6 +155,11 @@ func TestACutTakesTheShareOfTheContentItIsAskedFor(t *testing.T) { for _, problem := range shareVerdicts(problems, notYetCutAtTheShare) { t.Error(problem) } + for _, target := range sized { + if problem := problems[target.ID]; problem != "" { + t.Errorf("%s: %s", target.ID, problem) + } + } for _, problem := range absentFromTheManifest(missingOf, inManifest) { t.Error(problem) } @@ -200,6 +205,45 @@ func TestACutTakesTheShareOfTheContentItIsAskedFor(t *testing.T) { } } +// sizedPicture is the width and height sizedTargets names: 14 KiB of pixels in +// a BMP, a TIFF or a WebP of 64 KiB. Measured 2026-11-14: at 40x30 an AVIF's +// picture is 49 B, so the one byte a cut rounds by is two points of a share +// and more than shareSlack. At 80x60 the smallest is a PNG's 106 B. +var sizedPicture = map[string]string{"width": "80", "height": "60"} + +// sizedTargets is the files the sweep cuts with a width and a height named, of +// every format that takes both and takes the share, each recorded in +// missingOf. +// +// A picture whose width and height are left out grows to fill its file in +// bmp, tiff and webp, so there the share of the file is nearly the share of +// the picture and a format that does not split its padding passes. Named, the +// picture is small and the rest of the file is padding - the file all three +// missed the share in while the sweep cut only the defaults +// (docs/TRUNCATE-2026-11-11.md section 15.1). +func sizedTargets(missingOf map[string]int64) []engine.Target { + var out []engine.Target + for _, f := range format.All() { + if _, listed := notYetCutAtTheShare[f.ID]; listed || !takesSetting(f, "width") || !takesSetting(f, "height") { + continue + } + for _, missing := range shareMissing { + chain := damage.Chain{{ID: damage.Truncate, Values: damage.Values{damage.SettingMissing: strconv.FormatInt(missing, 10)}}} + size := max(64<<10, engine.SmallestDamaged(f, chain, format.Request{Label: true, Properties: sizedPicture})) + id := fmt.Sprintf("%s-sized-%d", f.ID, missing) + missingOf[id] = missing + out = append(out, engine.Target{ID: id, Format: f.ID, Sizes: engine.Uniform(1, size), Label: true, + Properties: sizedPicture, Damage: chain}) + } + } + return out +} + +// takesSetting is whether a format takes the setting named. +func takesSetting(f format.Descriptor, name string) bool { + return slices.ContainsFunc(f.Properties, func(p format.Property) bool { return p.Name == name }) +} + // absentFromTheManifest is every file planned for the sweep that the manifest // has no entry for. shareVerdicts asks only the cases a file was written for, // so without this a format that loses one of its files to the run still diff --git a/internal/guard/filemap_test.go b/internal/guard/filemap_test.go index 7d491e94..9d6a400e 100644 --- a/internal/guard/filemap_test.go +++ b/internal/guard/filemap_test.go @@ -302,7 +302,9 @@ func askTheLastEndCanaries(t *testing.T, dir string) { // zeros for a byte decoded again and again, and one whose tar header gives a // size below zero, which held the walk through the tar in place // (docs/REVIEW-240-2026-11-13.md). A loop comes back as the guard's time -// running out, a refusal of another kind as the words it gives. +// running out, a refusal of another kind as the words it gives. And a TIFF +// with its directory first and its last pixel byte gone, which was mapped +// with its pixels running past the end of the file (review of #244). func askTheReaderOfMapsToStop(t *testing.T, dir string) { t.Helper() // Letters drawn unevenly and without a period, so the stream codes mostly @@ -317,21 +319,24 @@ func askTheReaderOfMapsToStop(t *testing.T, dir string) { header := make([]byte, 512) copy(header, "a.txt") copy(header[124:], "-0000001000") + // A TIFF of an odd size has its directory right behind the header. + tiff := writeEncoded(t, "tiff", 4097, nil) cases := []struct { - name, says string - data []byte - setting map[string]string + format, name, says string + data []byte + setting map[string]string }{ - {"cut", "runs off the end of the file at bit", squeezed[:len(squeezed)/2], map[string]string{"compression": "best"}}, - {"below-zero", "gives a size below zero", gzipped(t, append(header, make([]byte, 1024)...), gzip.NoCompression), nil}, + {"targz", "cut", "runs off the end of the file at bit", squeezed[:len(squeezed)/2], map[string]string{"compression": "best"}}, + {"targz", "below-zero", "gives a size below zero", gzipped(t, append(header, make([]byte, 1024)...), gzip.NoCompression), nil}, + {"tiff", "short", "the strips run to 4097 and the file is 4096 B", tiff[:len(tiff)-1], nil}, } reqs := make([]oracle.LayoutRequest, len(cases)) for i, c := range cases { - path := filepath.Join(dir, "canary-"+c.name+".targz") + path := filepath.Join(dir, "canary-"+c.name+"."+c.format) if err := os.WriteFile(path, c.data, 0o600); err != nil { t.Fatal(err) } - reqs[i] = oracle.LayoutRequest{Format: "targz", Path: path, Settings: c.setting} + reqs[i] = oracle.LayoutRequest{Format: c.format, Path: path, Settings: c.setting} } got, _, err := oracle.LayoutBatch(t.Context(), reqs) if err != nil { @@ -339,7 +344,7 @@ func askTheReaderOfMapsToStop(t *testing.T, dir string) { } for i, c := range cases { if !strings.Contains(got[i].Error, c.says) { - t.Errorf("canary: the reader of maps answered a tar.gz %s with %q, where %q was due", c.name, got[i].Error, c.says) + t.Errorf("canary: the reader of maps answered a %s %s with %q, where %q was due", c.format, c.name, got[i].Error, c.says) } } } diff --git a/internal/guard/generatorbytes_test.go b/internal/guard/generatorbytes_test.go index ea92528e..e8dee78d 100644 --- a/internal/guard/generatorbytes_test.go +++ b/internal/guard/generatorbytes_test.go @@ -102,6 +102,12 @@ func goldenCases() map[string]engine.Target { // rather than in the code, and this is what closes it. "tiff_64kib_width_only": {ID: "g", Format: "tiff", Sizes: engine.Uniform(1, 65536), Label: true, Properties: map[string]string{"width": "128"}}, + // An odd size. TIFF 6.0 starts a directory on a word boundary, and + // last in a file of an odd size it would start on an odd byte, so + // such a file puts it right behind the header (O342) - a layout the + // three cases above, all of an even size, never reach. + "tiff_odd_size": {ID: "g", Format: "tiff", Sizes: engine.Uniform(1, 65537), Label: true, + Properties: map[string]string{"width": "64", "height": "64"}}, // WebP, pinned from three sides. Its size is arithmetic rather than // whatever a compressor decides, so a drift here can be a wrong SIZE and // not only different bytes - the same shape as BMP and TIFF. diff --git a/internal/guard/mutationcoverage_test.go b/internal/guard/mutationcoverage_test.go index 43cb9cd3..026e4f1b 100644 --- a/internal/guard/mutationcoverage_test.go +++ b/internal/guard/mutationcoverage_test.go @@ -173,6 +173,12 @@ var provenByProbe = map[string]string{ "front of the clusters (a cue giving 226 where no cluster starts), with a side of a WebM's padding allowed below a " + "Void (16000 B came to 16008 B), with an MP4 that never widens to co64 (5 GiB: no co64 in the head) and with the " + "widening's growth left out of the head the share is counted from (40 B more in front than the share).", + "TestABMPATIFFOrAWebPLaidOutForACutComesToEverySizeAndKeepsItsPixels": "its own canaries - a TIFF's StripOffsets and " + + "a BMP's bfOffBits a byte off must be refused - its reach check, every layout of the split met, and red on " + + "2026-11-14 with a WebP that never splits (nothing in front where the share wants 1 B), a TIFF that never splits " + + "(the directory at 12, not behind the header), a WebP front not rounded to an even chunk (16000 B came to 16001 B), " + + "a TIFF StripOffsets that leaves out the directory's values (the pixels given at 158, written at 180) and a BMP " + + "whose padding behind the pixels is not written (59 B came to 58 B).", "TestAPDFLaidOutForACutComesToEverySizeAndItsOffsetsLand": "its own canaries - a table that sends one object a byte off " + "must be refused, and the layout of a document nothing cuts must fail a cut at its half - its reach check, every " + "layout of the split met, red on 2026-11-13 with nothing in front of a document under 400 B of padding (398 sizes, " + diff --git a/internal/guard/picturecut_test.go b/internal/guard/picturecut_test.go new file mode 100644 index 00000000..679e61b3 --- /dev/null +++ b/internal/guard/picturecut_test.go @@ -0,0 +1,260 @@ +package guard + +import ( + "bytes" + "encoding/binary" + "errors" + "fmt" + "testing" + + "github.com/donislawdev/TestingFilesGenerator/internal/format" + _ "github.com/donislawdev/TestingFilesGenerator/internal/format/all" +) + +// A BMP, a TIFF or a WebP laid out for a cut comes to every size asked for, +// its pixels are where its own headers say and are the pixels of the file +// nothing cuts, and the cut keeps the share of them. +// +// Each grows its picture to fill a file whose width and height are left out, +// and with them named the picture is small and the rest of the file padding. +// For a cut each splits that padding around the pixels (format.SplitFront): +// a BMP between bfOffBits and the pixels and behind them, a TIFF the same +// with its directory moved up behind the header, a WebP in a private chunk in +// front of the picture - which needs a VP8X chunk first, so its front is that +// and an even chunk of at least eight bytes, or nothing - and behind it. The +// share guard cuts two sizes of each. This writes consecutive sizes cut at +// the half, and cuts of one size a byte at a time where the share starts to +// want padding in front and where the padding behind runs out, with the +// picture's size named and left out, holds each to its size, its pointers and +// the share, and asserts it met each layout (docs/TRUNCATE-2026-11-11.md +// section 15.4). The pixels are compared as bytes rather than decoded: the +// same bytes where the headers point give the same picture in every reader, +// which is what section 15.3 measured for the layouts. +// +// Canary: a TIFF whose StripOffsets is a byte off, and a BMP whose bfOffBits +// is, are refused. +func TestABMPATIFFOrAWebPLaidOutForACutComesToEverySizeAndKeepsItsPixels(t *testing.T) { + named := map[string]string{"width": "16", "height": "12"} + for _, id := range []string{"bmp", "tiff", "webp"} { + desc, err := format.Get(id) + if err != nil { + t.Fatal(err) + } + met := map[string]bool{} + for _, props := range []map[string]string{nil, named} { + wholes := map[int64][]byte{} + check := func(size, cutAt int64) { + data, regions := writePicture(t, desc, size, cutAt, props) + layout, err := pictureLayoutOf(id, data, regions, size, cutAt) + if err != nil { + t.Errorf("%s %v %d B cut at %d: %v", id, props, size, cutAt, err) + return + } + met[layout] = true + if _, ok := wholes[size]; !ok { + whole, wholeRegions := writePicture(t, desc, size, 0, props) + wholes[size] = whole[pictureRegion(wholeRegions).At:][:pictureRegion(wholeRegions).Length] + } + pics := pictureRegion(regions) + if !bytes.Equal(data[pics.At:pics.At+pics.Length], wholes[size]) { + t.Errorf("%s %v %d B cut at %d, %s: the pixels are not those of the file nothing cuts", id, props, size, cutAt, layout) + } + if err := picturePointers(id, data, pics); err != nil { + t.Errorf("%s %v %d B cut at %d, %s: %v", id, props, size, cutAt, layout, err) + } + } + smallest := desc.SmallestAccepted(format.Request{Label: true, Properties: props}) + for size := smallest; size < smallest+6; size++ { + check(size, size/2) + } + if props == nil { + // A picture that fills its file leaves less than a row of + // padding, so a byte of padding is hundreds of bytes of cut: + // the sweeps below are for the small picture. + continue + } + // Cuts a byte apart where the share starts to want padding in + // front - past head times the whole file over the part that is + // not the picture - up to a VP8X and a chunk later, and near the + // end, where what the share leaves behind runs out. + const size = 16_000 + _, regions := writePicture(t, desc, size, 0, props) + pics := pictureRegion(regions) + head := pictureHead(id, pics) + rest := size - pics.Length + start := max(1, (head-3)*size/rest) + for cutAt := start; cutAt < start+(vp8xAndAChunk+5)*size/rest; cutAt++ { + check(size, cutAt) + } + for j := int64(1); j <= (vp8xAndAChunk+3)*size/rest; j++ { + check(size, size-j) + } + } + for _, layout := range pictureLayouts[id] { + if !met[layout] { + t.Errorf("%s: no size met the layout %q - met %v", id, layout, met) + } + } + } + + // Canaries: a pointer a byte off. + for _, id := range []string{"tiff", "bmp"} { + desc, _ := format.Get(id) + data, regions := writePicture(t, desc, 64<<10, 32<<10, named) + pics := pictureRegion(regions) + broken := bytes.Clone(data) + at := map[string]int{"tiff": 8 + 2 + 12*5 + 8, "bmp": 10}[id] + binary.LittleEndian.PutUint32(broken[at:], binary.LittleEndian.Uint32(broken[at:])+1) + if picturePointers(id, broken, pics) == nil { + t.Errorf("canary: a %s whose pixel offset is a byte off passed", id) + } + } +} + +// vp8xAndAChunk is the least a WebP puts in front of its picture for a cut: +// the VP8X chunk and the smallest chunk there is. +const vp8xAndAChunk = 18 + 8 + +// pictureLayouts is every layout of the split each format has to meet. +var pictureLayouts = map[string][]string{ + "bmp": {"nothing in front, the share wants none", "the share"}, + "tiff": {"nothing in front, the share wants none", "the share"}, + "webp": {"nothing in front, the share wants none", "a chunk in front, the share wants less", "the share", + "all in front, what would stay behind is shorter than a chunk"}, +} + +// writePicture writes a file of desc through its own plan, with the settings +// props names, and gives its bytes and its map. +func writePicture(t *testing.T, desc format.Descriptor, size, cutAt int64, props map[string]string) ([]byte, []format.Region) { + t.Helper() + p, err := desc.Generator.Plan(format.Request{Bytes: size, Label: true, Seed: 7, CutAt: cutAt, Properties: props}) + if err != nil { + t.Fatalf("%s %d B: planning: %v", desc.ID, size, err) + } + var out bytes.Buffer + rec := &format.MapRecorder{W: &out} + if err := desc.Generator.Write(t.Context(), rec, p); err != nil { + t.Fatalf("%s %d B: writing: %v", desc.ID, size, err) + } + return out.Bytes(), rec.Map(size, true) +} + +// pictureHead is what stands in front of all of the padding of a file laid +// out for a cut: the headers of a BMP, a TIFF's header, directory and values, +// a WebP's RIFF header - its VP8X chunk counts as the price of a front. +func pictureHead(id string, pics format.Region) int64 { + return map[string]int64{"bmp": 54, "tiff": 8 + 150 + 22, "webp": 12}[id] +} + +// pictureLayoutOf says which layout of the split a file laid out for a cut has, +// or what is wrong with it: its length, and padding in front of the picture +// that is not the share, as format.SplitFront has it. +func pictureLayoutOf(id string, data []byte, regions []format.Region, size, cutAt int64) (string, error) { + if int64(len(data)) != size { + return "", fmt.Errorf("came to %d B", len(data)) + } + if id != "webp" { + return filmLayoutOfMap(regions, size, cutAt, 1) + } + pics := pictureRegion(regions) + if pics.Length <= 0 { + return "", errors.New("no picture marked") + } + front, back := int64(0), size-pics.At-pics.Length + if pics.At > 12 { + front = pics.At - 12 - (vp8xAndAChunk - 8) + } + pad := size - 12 - pics.Length + simple := format.PaddingInFront(12, pics.Length, cutAt, size) + switch { + case front == 0 && simple <= 0: + return "nothing in front, the share wants none", nil + case front == 0 && pad < vp8xAndAChunk: + return "nothing in front, too little padding for a VP8X and a chunk", nil + case front == 0: + return "", fmt.Errorf("nothing in front and %d B behind, where the share wants %d B in front", back, simple) + case front < 8 || front%2 != 0: + return "", fmt.Errorf("%d B in front, where a chunk is even and at least 8 B", front) + } + want := format.PaddingInFront(12+vp8xAndAChunk-8, pics.Length, cutAt, size) + switch { + case front == 8 && want < 8: + return "a chunk in front, the share wants less", nil + case front >= want && front-want < 2: + // The front rounds up to an even chunk, a byte at most. + return "the share", nil + case front > want && back < 8+2: + return "all in front, what would stay behind is shorter than a chunk", nil + } + return "", fmt.Errorf("%d B in front and %d B behind, where the share wants %d B in front", front, back, want) +} + +// picturePointers requires a BMP's, a TIFF's or a WebP's own headers to put +// its pixels where its map says, and a file to be as long as it says. +func picturePointers(id string, data []byte, pics format.Region) error { + le := binary.LittleEndian + switch id { + case "bmp": + if at := int64(le.Uint32(data[10:14])); at != pics.At { + return fmt.Errorf("bfOffBits gives %d and the pixels are at %d", at, pics.At) + } + if n := int64(le.Uint32(data[2:6])); n != int64(len(data)) { + return fmt.Errorf("bfSize gives %d B for a %d B file", n, len(data)) + } + case "tiff": + ifd := int64(le.Uint32(data[4:8])) + if ifd != 8 { + return fmt.Errorf("the directory is at %d, not right behind the header", ifd) + } + strips := map[uint16]int64{} + for i := range int64(le.Uint16(data[ifd:])) { + e := data[ifd+2+12*i:] + strips[le.Uint16(e)] = int64(le.Uint32(e[8:12])) + } + if strips[273] != pics.At || strips[279] != pics.Length { + return fmt.Errorf("the directory gives the pixels at %d for %d B, and they are at %d for %d B", strips[273], strips[279], pics.At, pics.Length) + } + case "webp": + return webpPointers(data, pics) + } + return nil +} + +// webpPointers walks a WebP's chunks: they have to fill the length the RIFF +// header gives, with at most seven bytes past it, the picture's chunk has to +// be where the map says, and a VP8X chunk has to stand first and give the +// picture's size. +func webpPointers(data []byte, pics format.Region) error { + le := binary.LittleEndian + end := 8 + int64(le.Uint32(data[4:8])) + if tail := int64(len(data)) - end; tail < 0 || tail > 7 { + return fmt.Errorf("%d B past the RIFF payload", tail) + } + var names []string + at, picture := int64(12), int64(-1) + for at+8 <= end { + name, n := string(data[at:at+4]), int64(le.Uint32(data[at+4:at+8])) + names = append(names, name) + if name == "VP8L" { + picture = at + } + at += 8 + n + n%2 + } + if at != end || picture != pics.At { + return fmt.Errorf("the chunks %v end at %d of %d, the picture's at %d where the map has it at %d", names, at, end, picture, pics.At) + } + if names[0] != "VP8X" { + if picture != 12 { + return fmt.Errorf("the chunks are %v, something in front of the picture without a VP8X", names) + } + return nil + } + bits := le.Uint32(data[picture+9:]) + w, h := bits&0x3FFF, bits>>14&0x3FFF + cw := uint32(data[24]) | uint32(data[25])<<8 | uint32(data[26])<<16 + ch := uint32(data[27]) | uint32(data[28])<<8 | uint32(data[29])<<16 + if cw != w || ch != h || !bytes.Equal(data[20:24], make([]byte, 4)) { + return fmt.Errorf("the VP8X gives a %dx%d canvas and flags % x for a %dx%d picture", cw+1, ch+1, data[20:24], w+1, h+1) + } + return nil +} diff --git a/internal/guard/strictverdict_test.go b/internal/guard/strictverdict_test.go index 05097c63..17431c1d 100644 --- a/internal/guard/strictverdict_test.go +++ b/internal/guard/strictverdict_test.go @@ -78,6 +78,11 @@ func TestTheStructuralCheckerRefusesBrokenBytesInsteadOfStopping(t *testing.T) { // one row table with its head zeroed came back OK, because the only // thing asked of it was whether its rows agreed with each other. {"csv with a NUL inside a field", "csv", nulInField(t, csv), oracle.StrictFail, "control character"}, + // The header of a lossless picture is five bytes. A shorter VP8L + // chunk stopped the check with an exception, the last one with no + // bytes after it (review of #244). + {"webp with an empty picture chunk", "webp", shortPicture(0), oracle.StrictFail, "its header alone is 5 B"}, + {"webp with four bytes of picture chunk", "webp", shortPicture(4), oracle.StrictFail, "its header alone is 5 B"}, } dir := t.TempDir() @@ -108,6 +113,18 @@ func TestTheStructuralCheckerRefusesBrokenBytesInsteadOfStopping(t *testing.T) { } } +// shortPicture is a WebP whose only chunk is a VP8L chunk of n bytes, each the +// signature byte - shorter than its own header when n is under five. +func shortPicture(n int) []byte { + chunk := append([]byte("VP8L"), binary.LittleEndian.AppendUint32(nil, uint32(n))...) + chunk = append(chunk, bytes.Repeat([]byte{0x2F}, n)...) + if n%2 == 1 { + chunk = append(chunk, 0) + } + out := append([]byte("RIFF"), binary.LittleEndian.AppendUint32(nil, uint32(4+len(chunk)))...) + return append(append(out, "WEBP"...), chunk...) +} + // strictPlant is run by the guard below. It loads the checker, puts checks that // misbehave in place of real ones and asks through the same path --batch // takes, printing one answer per line. diff --git a/internal/guard/testdata/generator-golden.json b/internal/guard/testdata/generator-golden.json index e5ed1054..e941b526 100644 --- a/internal/guard/testdata/generator-golden.json +++ b/internal/guard/testdata/generator-golden.json @@ -251,6 +251,11 @@ "sha256": "c5c38af41809c7e98333fc2a6bdcca36a955f70ab4413dbfde9b9ec4dd3d0b8f", "measured_on": "2026-08-29" }, + "tiff_odd_size": { + "bytes": 65537, + "sha256": "c286f5108925dd9fe9b7b22a38e7cac5e74a8858586634b6662bac473a11ded4", + "measured_on": "2026-11-14" + }, "txt_4kib": { "bytes": 4096, "sha256": "26a697bd184868acaa1da3f474b32a063e1b76e55ced42e38e79fdc63cf66161" diff --git a/internal/gui/text/locale/registry/en.json b/internal/gui/text/locale/registry/en.json index 8a8fc968..4d00e737 100644 --- a/internal/gui/text/locale/registry/en.json +++ b/internal/gui/text/locale/registry/en.json @@ -1296,8 +1296,8 @@ }, "Detail.damage/truncate.missing": { "description": "The sentence under the missing setting of the truncate damage, after what it takes.", - "hash": "sha256-912187dc5311", - "other": "How much of the whole file is gone, as a percentage. At least this much is cut - a few bytes more where the format cannot end exactly there. In most formats the cut removes the same percentage of the content. In some, a large file is mostly padding beside its content, so the cut can take all of the content or none - the manifest says what each cut left." + "hash": "sha256-e856fe0e9778", + "other": "How much of the whole file is gone, as a percentage. At least this much is cut - a few bytes more where the format cannot end exactly there. The cut removes the same percentage of the content - of the picture, the sound, the pages or the text. A file of a few kilobytes cut by 1% or 2% can lose less of its content than that, or none of it when the cut falls in what the format puts after the content, such as a ZIP's list of its files - the manifest says what each cut left." }, "Detail.damage/zero-head.bytes": { "description": "The sentence under the bytes setting of the zero-head damage, after what it takes.", diff --git a/internal/gui/text/locale/registry/pl.json b/internal/gui/text/locale/registry/pl.json index ef1ef471..a127dfb2 100644 --- a/internal/gui/text/locale/registry/pl.json +++ b/internal/gui/text/locale/registry/pl.json @@ -258,7 +258,7 @@ "Choice.winding.reversed": { "hash": "sha256-bc64f64fb479", "other": "odwrócony" }, "Choice.winding.rfc7946": { "hash": "sha256-38755035559d", "other": "rfc7946" }, "Detail.damage/append-junk.bytes": { "hash": "sha256-4403339e8eb1", "other": "Ile losowych bajtów trafia za koniec pliku. Poniżej 64 niektóre formaty wychodzą z uszkodzeniem, na które żaden czytnik nie narzeka." }, - "Detail.damage/truncate.missing": { "hash": "sha256-912187dc5311", "other": "Jaka część całego pliku znika, w procentach. Ucięte jest co najmniej tyle - o kilka bajtów więcej tam, gdzie format nie może się skończyć dokładnie w tym miejscu. W większości formatów cięcie usuwa ten sam procent treści. W niektórych duży plik to głównie dopełnienie obok treści, więc cięcie może zabrać całą treść albo nic z niej - manifest mówi, co zostawiło każde cięcie." }, + "Detail.damage/truncate.missing": { "hash": "sha256-e856fe0e9778", "other": "Jaka część całego pliku znika, w procentach. Ucięte jest co najmniej tyle - o kilka bajtów więcej tam, gdzie format nie może się skończyć dokładnie w tym miejscu. Cięcie usuwa ten sam procent treści - obrazu, dźwięku, stron albo tekstu. Plik o rozmiarze kilku kilobajtów ucinany o 1% lub 2% może stracić mniej treści, albo nic, gdy cięcie trafi w to, co format stawia za treścią, na przykład w spis plików archiwum ZIP - manifest mówi, co zostawiło każde cięcie." }, "Detail.damage/zero-head.bytes": { "hash": "sha256-8700126f6712", "other": "Ile bajtów na początku zostaje wyzerowanych. Poniżej czterech niektóre formaty wychodzą z uszkodzeniem, na które żaden czytnik nie narzeka." }, "Detail.format/avif.height": { "hash": "sha256-4b336c279584", "other": "Jak wysoki jest obraz. Gdy pole zostaje puste, wybierany jest rozmiar, który mieści się w zamówionych bajtach." }, "Detail.format/avif.quality": { "hash": "sha256-f6e1c3e1af83", "other": "Ile szczegółów zachowuje obraz. Wyższa wartość wygląda lepiej i zajmuje więcej pliku, zostawiając mniej miejsca na dopełnienie." }, diff --git a/internal/oracle/layout.py b/internal/oracle/layout.py index c20c6199..0b5290d1 100644 --- a/internal/oracle/layout.py +++ b/internal/oracle/layout.py @@ -99,16 +99,17 @@ def layout_wav(data): def layout_webp(data): - """The container's header, the picture's chunk, any other chunk as padding, - and whatever lies past the length the container declares as padding.""" + """The container's header and a VP8X chunk, the picture's chunk, any other + chunk as padding, and whatever lies past the length the container declares + as padding.""" if data[:4] != b"RIFF" or data[8:12] != b"WEBP": raise NoMap("no RIFF WEBP header") declared = 8 + struct.unpack(" len(data): raise NoMap(f"the container declares {declared} B and the file is {len(data)} B") - roles = {b"VP8L": CONTENT, b"VP8 ": CONTENT} + roles = {b"VP8L": CONTENT, b"VP8 ": CONTENT, b"VP8X": STRUCTURE} out = [[STRUCTURE, 0, 12]] - out += [[r if r != STRUCTURE else PADDING, at, n] for r, at, n in riff_chunks(data[:declared], 12, roles)] + out += [[roles.get(data[at:at + 4], PADDING), at, n] for _, at, n in riff_chunks(data[:declared], 12, roles)] out.append([PADDING, declared, len(data) - declared]) return out @@ -545,15 +546,17 @@ def layout_targz(data, settings): def layout_bmp(data): """The two headers, then whatever lies before the offset the file header - gives for the pixels, then the pixels the information header counts.""" + gives for the pixels, then the pixels the information header counts, then + whatever lies behind them.""" if data[:2] != b"BM": raise NoMap("no BM signature") pixels_at = struct.unpack(" len(data): raise NoMap(f"the pixels run from {pixels_at} for {pixels} B and the file is {len(data)} B") - return [[STRUCTURE, 0, head], [PADDING, head, pixels_at - head], [CONTENT, pixels_at, pixels]] + return [[STRUCTURE, 0, head], [PADDING, head, pixels_at - head], [CONTENT, pixels_at, pixels], + [PADDING, pixels_at + pixels, len(data) - pixels_at - pixels]] def sub_blocks_end(data, pos): @@ -665,6 +668,11 @@ def layout_ico(data): return out +# TIFF_SIZES is how many bytes a value of each TIFF data type the generator +# writes takes. +TIFF_SIZES = {1: 1, 2: 1, 3: 2, 4: 4, 5: 8, 7: 1} + + def tiff_values(data, kind, count, value_at): """The values of one directory entry, inline or where it points.""" width = {3: 2, 4: 4}[kind] @@ -676,21 +684,33 @@ def tiff_values(data, kind, count, value_at): def layout_tiff(data): """The header, the strips the directory points at, and the directory - the end, as the generator writes it last. What lies between the header and the - first strip is not anything the directory names.""" + first strip is not anything the directory names. In a file laid out for a + cut the directory and its values follow the header, and what lies between + them and the first strip, and behind the last, is padding.""" if data[:4] != b"II*\x00": raise NoMap("no little-endian TIFF header") ifd_at = struct.unpack(" 4: + head = max(head, struct.unpack(" len(data): + raise NoMap(f"the strips run to {last} and the file is {len(data)} B") + return [[STRUCTURE, 0, head], [PADDING, head, first - head], [CONTENT, first, last - first], + [PADDING, last, len(data) - last]] if last != ifd_at: raise NoMap(f"the strips end at {last} and the directory starts at {ifd_at}") return [[STRUCTURE, 0, 8], [PADDING, 8, first - 8], [CONTENT, first, last - first], diff --git a/internal/oracle/strict.py b/internal/oracle/strict.py index f8d20f32..7855dffc 100644 --- a/internal/oracle/strict.py +++ b/internal/oracle/strict.py @@ -1392,8 +1392,12 @@ def check_bmp(data): if offbits + stride * rows > len(data): fail(f"the pixels run to {offbits + stride * rows} and the file ends at {len(data)}") + # The padding stands in front of the pixels, in the gap bfOffBits points + # past, and in a file laid out for a cut partly behind them, where no field + # points (internal/format/bmp/cut.go). gap = offbits - 54 - ok(f"{width}x{rows}, {depth} bit, stride {stride} B, gap {gap} B, size field agrees") + behind = len(data) - offbits - stride * rows + ok(f"{width}x{rows}, {depth} bit, stride {stride} B, gap {gap} B, {behind} B behind the pixels, size field agrees") def gif_blocks(data, pos): @@ -1850,6 +1854,11 @@ def check_tiff(data): fail(f"the magic number is {magic} and a TIFF says 42") ifd_at = struct.unpack(" len(data): fail(f"the directory is said to start at {ifd_at} and the file ends at {len(data)}") @@ -1861,7 +1870,7 @@ def check_tiff(data): fail(f"a directory of {count} entries runs to {end} and the file ends at {len(data)}") sizes = {1: 1, 2: 1, 3: 2, 4: 4, 5: 8, 7: 1} - tags, previous = {}, -1 + tags, previous, values_end = {}, -1, end for i in range(count): at = ifd_at + 2 + 12 * i tag, kind, n = struct.unpack(" 4: off = struct.unpack(" len(data): fail(f"the value of tag {tag} runs to {off + total} and the file ends at {len(data)}") value = data[off:off + total] + values_end = max(values_end, off + total) else: value = raw[:total] tags[tag] = (kind, n, value) @@ -1932,13 +1944,23 @@ def one(tag, name): fail(f"the pixels are said to start at {offset}, inside the header") if offset + counts > len(data): fail(f"the pixels run to {offset + counts} and the file ends at {len(data)}") - # The pixels have to end where the directory begins, because this - # generator puts its padding in front of them and nothing between. - if offset + counts != ifd_at: + # The generator writes one of two layouts. A file nothing cuts has its + # padding in front of the pixels and the directory right behind them, so + # the pixels end where the directory begins. A file a damage will cut has + # the directory and its values right behind the header, and the padding + # split around the pixels (internal/format/tiff/cut.go). Anything else is a + # file this generator does not write. + if ifd_at == 8: + if offset < values_end: + fail(f"the pixels start at {offset}, inside the directory and its values, which run to {values_end}") + layout = (f"the directory first, {offset - values_end} B in front of the pixels and " + f"{len(data) - offset - counts} B behind them") + elif offset + counts == ifd_at: + layout = f"gap {offset - 8} B" + else: fail(f"the pixels end at {offset + counts} and the directory begins at {ifd_at}") - gap = offset - 8 - ok(f"{width}x{height}, {samples * 8} bit, uncompressed, gap {gap} B, " + ok(f"{width}x{height}, {samples * 8} bit, uncompressed, {layout}, " f"{count} entries in one directory, byte counts agree with the geometry") @@ -1977,7 +1999,7 @@ def check_webp(data): fail(f"{trailing} B sit after the RIFF payload, and only the odd byte belongs there - " "the bulk of the padding belongs in a chunk") - pos, image, private, seen = 12, 0, 0, [] + pos, chunks = 12, [] while pos + 8 <= payload_end: tag = data[pos:pos + 4] size = struct.unpack(" payload_end: fail(f"the chunk {tag.decode('latin1')!r} says {size} B and only " f"{payload_end - pos - 8} B are left") - seen.append(tag.decode("latin1")) - if tag in (b"VP8L", b"VP8 ", b"VP8X"): - image += 1 - if tag == b"VP8L" and data[pos + 8] != 0x2F: - fail("the lossless stream does not start with its signature byte") - else: - private += 1 + chunks.append((tag, pos + 8, size)) pos = end + (size & 1) if pos != payload_end: fail(f"the chunks end at {pos} and the payload ends at {payload_end}") - if image != 1: - fail(f"the file carries {image} image chunks and a still WebP has one") + tags = [t for t, _, _ in chunks] + seen = [t.decode("latin1") for t in tags] + if tags.count(b"VP8L") != 1 or b"VP8 " in tags: + fail(f"the file carries the chunks {seen}, and a still lossless WebP has one VP8L chunk") + picture = tags.index(b"VP8L") + _, at, size = chunks[picture] + # The signature byte and the four that carry the picture's size come + # first. A chunk shorter than that has no header to read, and reading it + # anyway would take the next chunk's bytes, or none, for it. + if size < 5: + fail(f"the VP8L chunk carries {size} B and its header alone is 5 B") + if data[at] != 0x2F: + fail("the lossless stream does not start with its signature byte") + bits = struct.unpack("> 14) & 0x3FFF) + 1 + + # RFC 9649 has the simple format carry the VP8L chunk and nothing in front + # of it (section 2.6), and lets unknown chunks of the extended one, which + # opens with a VP8X chunk, appear out of order (section 2.7). The generator + # writes the extended format only for a file a damage will cut, with + # padding in front of the picture (internal/format/webp/cut.go). Eight + # decoders of ten refuse a chunk in front of the picture without the VP8X + # (docs/TRUNCATE-2026-11-11.md section 15.3). + if b"VP8X" in tags: + if tags.index(b"VP8X") != 0 or tags.count(b"VP8X") != 1: + fail(f"the chunks are {seen}, and the VP8X chunk stands first and once") + _, at, size = chunks[0] + if size != 10: + fail(f"the VP8X chunk is {size} B and it carries 10") + if data[at:at + 4] != bytes(4): + fail("the VP8X chunk claims a colour profile, alpha, metadata or animation the file does not carry") + canvas = (int.from_bytes(data[at + 4:at + 7], "little") + 1, int.from_bytes(data[at + 7:at + 10], "little") + 1) + if canvas != (width, height): + fail(f"the VP8X canvas is {canvas[0]}x{canvas[1]} and the picture is {width}x{height}") + form, private = "extended", len(tags) - 2 + else: + if picture != 0: + fail(f"the chunks are {seen}: something stands in front of the picture without a VP8X chunk first") + form, private = "simple", len(tags) - 1 - ok(f"riff {declared} B, chunks {seen}, {private} private, {trailing} B after the payload") + ok(f"riff {declared} B, {form} format, chunks {seen}, {private} private, {trailing} B after the payload") def iso_boxes(data, start, end, depth=0):