Support archive format version 9 chunk alignment padding

[SkowronskiAndrew]

Summary

Support for a tentative upcoming Unity Archive format change, done proactively to help test that the format change works as expected and does not cause a challenge to ongoing UnityDataTool support.

The proposed Unity change bumps the Unity Archive format to version 9 and adds a new header flag,
kArchiveBlockPaddingBetweenChunks (bit 10). When it is set, each chunk-based
(non-streamed) storage block is followed by zero padding up to the next 16-byte
boundary of the data section. The point is binary patching: previously a size change
in one compressed chunk shifted the position of every chunk after it, so a small
content change dirtied the whole tail of the archive.

The padding belongs to no block — it is excluded from the block sizes, from the
uncompressed data size, and from the CRC. That matters because the block list stores
only sizes, never offsets, so any reader has to accumulate block positions from the
block sizes. Without this change UnityDataTools computes a wrong file offset for every
block after the first padded one. The format version bump to 9 exists so that older
runtimes reject the new layout instead of silently reading it at the wrong offsets.

Changes

Parsing

• UnityBinaryFormat/ArchiveDetector.cs — align the accumulated block offset to 16
  bytes after each non-streamed block when the flag is set. The alignment is applied to
  the offset relative to the start of the data section, which is what
  ArchiveStorageReader::Initialize does (m_BlocksOffsets[0] = 0) and what the writer's
  AlignChunkWritePosition does (it aligns on m_StoredCompressedDataSize, not on the
  absolute file position). In practice the data section always starts on a 16-byte
  boundary for any archive that can carry this flag, so the blocks come out aligned within
  the file too — see the note below.
• Replaced the ad-hoc const uint flagX locals scattered across three methods with a
  shared ArchiveFlags class, now that the flags are needed in more than one place.

Reporting

• archive header lists BlockPaddingBetweenChunks as a recognized flag rather than
  falling back to the raw 0x400.
• archive info gained a Block Padding Size / blockPaddingSize field. Without it
  Data Size silently stops accounting for the whole data section on the new format.

Test data

• UnityProjects/LeadingEdge/Assets/Editor/BuildAssetBundles.cs gained a BuildLz4
  entry point (ChunkBasedCompression) that writes the same bundle layout to a separate
  folder. The existing LZMA reference bundles are deliberately left untouched — we want
  both the v8 single-block and the v9 padded-chunk layouts as reference data.
• TestCommon/Data/LeadingEdgeBuilds/AssetBundlesLz4/ — the resulting v9 bundles (490 KB).
  The scenes bundle is the interesting one: 11 blocks, 92 bytes of padding, with
  BuildPlayer-Scene1.sharedAssets spanning blocks 0-7.

Testing

dotnet test — 784 of 785 pass; 1 expected failure, see below.

New coverage in UnityDataTool.Tests/ArchiveTests.cs. The main test,
ArchiveBlocks_BlockPaddingBetweenChunks_OffsetsSkipThePadding, deliberately hardcodes no
offsets: for each block it asserts the computed offset is 16-aligned relative to the data
section, that the gap is smaller than one alignment unit, and that every byte in the gap
is actually zero in the file. That fails loudly if either the alignment base or the
streamed-block exclusion is wrong.

Manual verification that the new format is read correctly:

• Padding bytes confirmed zero by direct hex inspection of the archive.
• archive extract (which goes through the native library, independently of the C# parser)
  produces correct sizes for a file spanning 8 padded chunks.
• dump of the LZ4 scenes bundle is byte-identical to the LZMA one.
• analyze over both bundle sets: all 38 objects match on type, name, size and CRC32.
  Only AssetBundleManifest differs, as expected since the manifest bundle name changed.
  This confirms the padding really is excluded from the CRC.
• The object-boundary chunking from the companion commits shows up too:
  Scene1.sharedAssets ends at 760788, exactly where block 7 ends.

Is the data section itself always 16-byte aligned?

Worth recording, because if it were not, "aligned relative to the data section" and
"aligned within the file" would diverge and the patching benefit would be lost.
The correct behavior was confirmed inside Unity implementation and also checked against all 42 UnityFS archives in the test data:

Version	Count	Flags	Data offset % 16
v6	3	BlocksAndDirectoryInfoCombined	unaligned
v7	4	BlocksAndDirectoryInfoCombined	unaligned
v8	25	+ BlockInfoNeedPaddingAtStart	0
v8	2	+ BlocksInfoAtTheEnd	0
v9	8	+ BlockPaddingBetweenChunks	0

The only unaligned data sections are Unity 2019.4-2021.3 era v6/v7 files, which have
neither flag and can never set the v9-only padding flag.

Draft: one test fails until the native library is updated

This PR intentionally does not include an updated UnityFileSystemApi native library.
Locally this work was validated against a debug build of
buildpipeline/archive_alignment; shipping it would mean committing a 9.7 MB debug DLL and
would only cover Windows.

With the currently committed (v8-era) library, exactly one test fails:

Failed ArchiveExtract_BlockPaddingBetweenChunks_FilesExtractedSuccessfully

The old library rejects a v9 archive cleanly — Invalid file format — rather than
misreading it, which is precisely what the version bump is for. All the pure-C# commands
(header, blocks, list, info) already work on v9 with the committed library, since
they do not go through the native code.

To take this out of draft, official UnityFileSystemApi builds for all three platforms
need to be dropped in once the format change lands. Until then the tool cannot exact content from v9 archives on
any platform.

[Copilot]

Pull request overview

This PR adds proactive support for Unity Archive format v9’s BlockPaddingBetweenChunks flag by updating the C# archive parser to correctly skip 16-byte alignment padding between non-streamed blocks, and extends CLI reporting + test data/tests to validate the new layout.

Changes:

• Update archive block offset accumulation to align after each non-streamed block when BlockPaddingBetweenChunks is set, and consolidate header flag constants into a shared ArchiveFlags definition.
• Improve reporting: recognize the new flag in archive header output and report total block padding size in archive info.
• Add LeadingEdge LZ4 (chunk-based) AssetBundle build output and new tests that validate padding bytes and alignment behavior.

Reviewed changes

Copilot reviewed 14 out of 22 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
UnityBinaryFormat/ArchiveDetector.cs	Adds v9 padding-aware block offset accumulation and introduces shared ArchiveFlags.
Archive/ArchiveTool.cs	Reports BlockPaddingBetweenChunks as a known flag and adds blockPaddingSize reporting in archive info.
UnityDataTool.Tests/ArchiveTests.cs	Adds tests for v9 flag reporting, block alignment/padding verification, and native extraction on padded archives.
UnityProjects/LeadingEdge/Assets/Editor/BuildAssetBundles.cs	Adds a second build entry point to generate LZ4 chunk-based bundles into a separate folder.
UnityProjects/LeadingEdge/AGENTS.md	Documents the new LZ4 build entry point and its purpose.
TestCommon/Data/LeadingEdgeBuilds/AGENTS.md	Documents the new AssetBundlesLz4/ test-data folder and why it exists.
TestCommon/Data/LeadingEdgeBuilds/AssetBundlesLz4/AssetBundlesLz4.manifest	Adds new LeadingEdge LZ4 bundle manifest test data.
TestCommon/Data/LeadingEdgeBuilds/AssetBundlesLz4/a.manifest	Adds new LeadingEdge LZ4 bundle manifest test data.
TestCommon/Data/LeadingEdgeBuilds/AssetBundlesLz4/6.manifest	Adds new LeadingEdge LZ4 bundle manifest test data.
TestCommon/Data/LeadingEdgeBuilds/AssetBundlesLz4/assetbundleroot.manifest	Adds new LeadingEdge LZ4 bundle manifest test data.
TestCommon/Data/LeadingEdgeBuilds/AssetBundlesLz4/directaudioclipreference.manifest	Adds new LeadingEdge LZ4 bundle manifest test data.
TestCommon/Data/LeadingEdgeBuilds/AssetBundlesLz4/serializationdemo.manifest	Adds new LeadingEdge LZ4 bundle manifest test data.
TestCommon/Data/LeadingEdgeBuilds/AssetBundlesLz4/scenes.manifest	Adds new LeadingEdge LZ4 bundle manifest test data.
TestCommon/Data/LeadingEdgeBuilds/AssetBundlesLz4/singleaudioclipdirectreference.manifest	Adds new LeadingEdge LZ4 bundle manifest test data.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.