From 80ba5781dac0331fa8675e18523c6304f43b0a28 Mon Sep 17 00:00:00 2001 From: gqian-coder Date: Fri, 23 Oct 2026 10:59:52 -0400 Subject: [PATCH 1/2] Build the MGARD 1.6.0 data-compatibility decoder by default MGARD_ENABLE_DATA_COMPATIBILITY now defaults to ON. With it OFF, a 1.7 build cannot decompress anything written by MGARD 1.6.x, the previous release: decompress returns DataCompatibilityNotEnabledFailure. Callers that do not check the status cannot tell. ADIOS2's mgard operator, for one, ignores it, so reading 1.6.x data through an OFF build returns zeros with no error. Builds that never read 1.6.x data can still turn the option off. Compressed output is the same with the option ON or OFF. Update the option text, the MGARD-X guide and the 1.7.0 release notes. The "roughly doubles the build time" note is replaced by a neutral one: on one Frontier node (HIP gfx90a + SERIAL + MDR, tests on, -j32) the full build took 7m24s with the decoder instead of 6m42s, and libmgard.so grew from 258 MB to 344 MB. Co-Authored-By: Claude Opus 5.5 --- CMakeLists.txt | 2 +- doc/MGARD-X.md | 6 +++--- doc/release notes/1.7.0.md | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index f435d0071..7c0e42f61 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -47,7 +47,7 @@ option(MGARD_ENABLE_UNSTRUCTURED "Enable experimental unstructured mesh support. option(MGARD_ENABLE_TIMING "Enable timing." OFF) -option(MGARD_ENABLE_DATA_COMPATIBILITY "Also build the separate MGARD 1.6.0-compatible decompression pipeline (src/mgard-x-160) so data compressed by MGARD 1.6.x can be decompressed. Roughly doubles MGARD-X build time." OFF) +option(MGARD_ENABLE_DATA_COMPATIBILITY "Also build the separate MGARD 1.6.0-compatible decompression pipeline (src/mgard-x-160) so data compressed by MGARD 1.6.x can be decompressed. Adds build time and library size; turn OFF if no 1.6.x data needs to be read." ON) include(CTest) diff --git a/doc/MGARD-X.md b/doc/MGARD-X.md index d062a76c6..c26df1ff0 100644 --- a/doc/MGARD-X.md +++ b/doc/MGARD-X.md @@ -85,13 +85,13 @@ MGARD-X is a portable implementation of the MGARD lossy compressor supporting va |MGARD\_ENABLE\_SYCL|ON/OFF|OFF|Enable portable GPU compression/decompression with SYCL on Intel GPUs| |MGARD\_ENABLE\_AUTO\_TUNING|ON/OFF|OFF|Enable auto tuning feature| |MGARD\_ENABLE\_MDR|ON/OFF|OFF|Enable building MDR and MDR-X. MPI is optional: the parallel ```pmdr-x-qoi``` and ```pmdr-x-qoi-io``` executables are built only when MPI is found| -|MGARD\_ENABLE\_DATA\_COMPATIBILITY|ON/OFF|OFF|Also build the MGARD 1.6.0-compatible decompression pipeline so that data compressed by MGARD 1.6.x and earlier can be decompressed (see [Decompressing data from earlier versions](#decompressing-data-from-earlier-versions)). Roughly doubles the MGARD-X build time| +|MGARD\_ENABLE\_DATA\_COMPATIBILITY|ON/OFF|ON|Also build the MGARD 1.6.0-compatible decompression pipeline so that data compressed by MGARD 1.6.x and earlier can be decompressed (see [Decompressing data from earlier versions](#decompressing-data-from-earlier-versions)). Adds build time and library size; turn OFF if no 1.6.x data needs to be read| ## Decompressing data from earlier versions Data compressed by MGARD 1.7.0 uses a different format from MGARD 1.6.x (hierarchy geometry, Huffman layout, default number of levels, and domain decomposition header). Decompression reads the software version recorded in the header of the compressed data and handles each version as follows: * **Compressed by 1.7.0:** decompressed by the 1.7.0 pipeline. -* **Compressed by 1.6.x or earlier:** decompressed by a separate pipeline that reproduces the MGARD 1.6.0 decompressor, if MGARD-X is built with ```-DMGARD_ENABLE_DATA_COMPATIBILITY=ON```. The result is the same as decompressing with MGARD 1.6.0 (bit-identical on CPUs). Without this option, decompression returns ```mgard_x::compress_status_type::DataCompatibilityNotEnabledFailure```. +* **Compressed by 1.6.x or earlier:** decompressed by a separate pipeline that reproduces the MGARD 1.6.0 decompressor, which is built by default (```MGARD_ENABLE_DATA_COMPATIBILITY=ON```). The result is the same as decompressing with MGARD 1.6.0 (bit-identical on CPUs). If MGARD-X is built with ```-DMGARD_ENABLE_DATA_COMPATIBILITY=OFF```, decompression returns ```mgard_x::compress_status_type::DataCompatibilityNotEnabledFailure```. This is transparent to users: the same CLI and high-level APIs decompress both formats, and decompression needs no extra configuration. Compression always produces the 1.7.0 format. Data compressed by MGARD 1.6.x with the variable-size domain decomposition (```domain_decomposition_type::Variable```) is not supported, because its block sizes are not recorded in the header. @@ -230,7 +230,7 @@ An executable ```mgard-x``` will be built after building the MGARD-X library. To + mgard_x::compress_status_type::NotSupportHigherNumberOfDimensionsFailure + mgard_x::compress_status_type::NotSupportDataTypeFailure + mgard_x::compress_status_type::BackendNotAvailableFailure - + mgard_x::compress_status_type::DataCompatibilityNotEnabledFailure: the data was compressed by MGARD 1.6.x or earlier and MGARD-X was built without ```MGARD_ENABLE_DATA_COMPATIBILITY``` (see [Decompressing data from earlier versions](#decompressing-data-from-earlier-versions)) + + mgard_x::compress_status_type::DataCompatibilityNotEnabledFailure: the data was compressed by MGARD 1.6.x or earlier and MGARD-X was built with ```MGARD_ENABLE_DATA_COMPATIBILITY=OFF``` (see [Decompressing data from earlier versions](#decompressing-data-from-earlier-versions)) ## Using low-level APIs * **Step 1: Build Hierarchy** diff --git a/doc/release notes/1.7.0.md b/doc/release notes/1.7.0.md index 6cc1cd7f2..eecd9f954 100644 --- a/doc/release notes/1.7.0.md +++ b/doc/release notes/1.7.0.md @@ -9,7 +9,7 @@ * A BlockDelta (FLE-based) lossless backend for mgard-x, as a Huffman alternative. * A portable, warp-cooperative LZ4 implementation for mgard-x, removing the nvcomp dependency. * Blackwell (sm_120) GPU build support. -* Decompression of data compressed by MGARD 1.6.x and earlier, through a separate MGARD 1.6.0-compatible decompression pipeline enabled with the new CMake option `MGARD_ENABLE_DATA_COMPATIBILITY` (default OFF). Decompression detects the version from the header and routes 1.6.x data automatically; output matches MGARD 1.6.0 (bit-identical on CPUs). Builds without the option report such data with the new status `DataCompatibilityNotEnabledFailure`. +* Decompression of data compressed by MGARD 1.6.x and earlier, through a separate MGARD 1.6.0-compatible decompression pipeline enabled with the new CMake option `MGARD_ENABLE_DATA_COMPATIBILITY` (default ON). Decompression detects the version from the header and routes 1.6.x data automatically; output matches MGARD 1.6.0 (bit-identical on CPUs). Builds with the option OFF report such data with the new status `DataCompatibilityNotEnabledFailure`. * Generic block domain decomposition with two modes (`Config::domain_decomposition_mode`): `Auto` (default) decomposes only when the data does not fit the memory limit and picks the block size, and `Manual` uses a user-given block shape (`Config::domain_decomposition_block_size`), which may differ per dimension. Available in the `mgard-x` and `mdr-x` CLIs as `-dd auto|manual` and `-dd-size`. * `MemoryManager::Copy3D` in RuntimeX (native 3D copies on CUDA and HIP), used for copying blocks between host and device in fewer transfers. * Opt-in ZSTD compression of MDR-X bitplane groups (`lossless = Huffman_Zstd`; `mdr-x -l huffman-zstd`). From e0fa7a33f0b8629e3ed145639553d807ba007675 Mon Sep 17 00:00:00 2001 From: gqian-coder Date: Fri, 23 Oct 2026 11:30:41 -0400 Subject: [PATCH 2/2] ci: refresh the apt package lists before installing packages The build workflow failed in build-MGARD's first step, `sudo apt-get install ...`, with exit code 100, before the code was checked out; the same step passed on master on 2026-10-14. The usual cause is stale package lists: hosted runners ship with the lists from when the image was built, and the mirror drops package versions that have since been superseded. Run `apt-get update` first in every job that installs packages, as GitHub recommends for its hosted runners. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/build.yml | 4 ++-- .github/workflows/format.yml | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index d3b5adb9b..fa02ea6d5 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -17,7 +17,7 @@ jobs: build-MOAB: runs-on: ubuntu-24.04 steps: - - run: sudo apt-get install mpich libmpich-dev liblapack-dev + - run: sudo apt-get update && sudo apt-get install mpich libmpich-dev liblapack-dev - name: cache-MOAB id: cache-MOAB uses: actions/cache@v4 @@ -97,7 +97,7 @@ jobs: needs: [build-MOAB, build-Catch2, build-TCLAP, build-ZSTD] runs-on: ubuntu-24.04 steps: - - run: sudo apt-get install mpich doxygen libmpich-dev liblapack-dev libzstd1 libzstd-dev libtclap-dev protobuf-compiler libprotobuf-dev + - run: sudo apt-get update && sudo apt-get install mpich doxygen libmpich-dev liblapack-dev libzstd1 libzstd-dev libtclap-dev protobuf-compiler libprotobuf-dev - name: checkout uses: actions/checkout@v4 - name: fetch-MOAB diff --git a/.github/workflows/format.yml b/.github/workflows/format.yml index 700aaf484..e1ec4efd5 100644 --- a/.github/workflows/format.yml +++ b/.github/workflows/format.yml @@ -7,7 +7,7 @@ jobs: clang-format: runs-on: ubuntu-22.04 steps: - - run: sudo apt-get install clang-format + - run: sudo apt-get update && sudo apt-get install clang-format - uses: actions/checkout@v2.3.4 with: repository: Sarcasm/run-clang-format