Add separate package resource directory to dmf-get

docs/dmf-get-tool.md

@@ -100,6 +100,7 @@ Two variables are automatically substituted in URLs:
 - `DMOD_DMFC_DIR` - Default output directory for DMFC files
 - `DMOD_MANIFEST` - Default manifest path or URL
 - `DMOD_LICENSE_DIR` - Default base directory for license resources (overridden by `--license-dir`)
+- `DMOD_PACKAGE_DIR` - Default root for package resources and metadata (overridden by `--package-dir`)
 - `DMOD_VIEWS_DIR` - Default views directory (same as `--views-dir`)
 
 ### Version Matching
@@ -304,6 +305,24 @@ dmf-get [options] [install] <module_name>[@version]
 
 - `-o, --output-dir <path>` - Output directory for downloaded modules (default: `./dmf` or `DMOD_DMF_DIR`)
 
+#### Package Directory
+
+- `--package-dir <path>` - Use a separate root for installed package resources, unpacked package directories and `.dmd`/`.dmr` metadata. Defaults to the non-empty `DMOD_PACKAGE_DIR` environment variable; the command-line option takes precedence. Without either setting, the existing combined layout is preserved.
+- `.dmf` and `.dmfc` binaries stay directly in `--output-dir` / `DMOD_DMF_DIR`. Binary resources from `.dmr` are installed there by source filename. Non-binary `.dmr` resources resolve `${destination}` against the package root. Packages without `.dmr` keep their complete structure under `<package-dir>/<module>`.
+- Dependencies use the same package root. Configuration copying (`--config`, `--config-dir`, `--config-map`) reads resources from that root, including custom paths declared in the retained `.dmr` file.
+- Default views move to `<package-dir>/views/<module>`. Explicit `--views-dir` / `DMOD_VIEWS_DIR` and `--license-dir` / `DMOD_LICENSE_DIR` retain their precedence. Other explicit destinations declared by `.dmr` remain in effect.
+- Relative paths are resolved from the working directory; missing parent directories are created. ZIP downloads are staged in the package root and removed after extraction; `.dmp` archives are stored there. Temporary ZIP extraction and the download cache keep their existing locations.
+- `--mini` keeps its existing resource filtering, while `.dmd`/`.dmr` metadata stays in the package root. The dedicated `headers`, `docs` and `lib` commands keep their existing output paths.
+- Use a package root outside the module output directory to keep the latter limited to `.dmf`/`.dmfc` files. Existing files from earlier installations are not moved or deleted.
+
+```bash
+dmf-get -o ./dmf --package-dir ./packages mymodule
+DMOD_PACKAGE_DIR=./packages dmf-get -o ./dmf -d dependencies.dmd
+
+# Separate module binaries, package resources and licenses
+dmf-get -o ./dmf --package-dir ./packages --license-dir ./licenses -y mymodule
+```
+
 #### License Options
 
 - `--license-dir <path>` - Install resources whose `.dmr` key is `license` under `<path>/<module>/<source-name>`, overriding their declared destination. Paths may be absolute or relative to the current working directory. Each module (including dependencies) gets its own directory; source filenames are preserved. Directory resources are copied recursively, and reinstalling updates their contents without adding another directory level.
@@ -320,7 +339,7 @@ dmf-get --mini --license-dir ./licenses -y -d dependencies.dmd
 
 #### Views Options
 
-- `--views-dir <path>` - Directory where views (`.dmv`) of the installed applications are placed (default: `DMOD_VIEWS_DIR`, otherwise `views` inside the output directory, e.g. `$DMOD_DMF_DIR/views`). Every module gets its own subdirectory `<path>/<module>`, which is what `${views_dir}` resolves to in `.dmr` files (e.g. `views=./views => ${views_dir}`), so views of different modules never overwrite each other. The contents of the `views` directory are copied into it (a reinstall overwrites them). The `views` resource is installed also with `--mini`.
+- `--views-dir <path>` - Directory where views (`.dmv`) of the installed applications are placed (default: `DMOD_VIEWS_DIR`, otherwise `views` inside the package root when configured, or the output directory, e.g. `$DMOD_DMF_DIR/views`). Every module gets its own subdirectory `<path>/<module>`, which is what `${views_dir}` resolves to in `.dmr` files (e.g. `views=./views => ${views_dir}`), so views of different modules never overwrite each other. The contents of the `views` directory are copied into it (a reinstall overwrites them). The `views` resource is installed also with `--mini`.
 
 #### Configuration Options
 
@@ -351,6 +370,7 @@ dmf-get --mini --license-dir ./licenses -y -d dependencies.dmd
 - `DMOD_DMFC_DIR` - Default DMFC output directory
 - `DMOD_MANIFEST` - Default manifest path or URL
 - `DMOD_LICENSE_DIR` - Default base directory for license resources (overridden by `--license-dir`)
+- `DMOD_PACKAGE_DIR` - Default root for package resources and metadata (overridden by `--package-dir`)
 
 ### Examples
 

tests/integration/test_dmf_get.sh

@@ -490,6 +490,7 @@ rm -rf test_type_both test_type_dmf_only test_type_nodmr typemod.zip dmfonly.zip
 
 echo ""
 bash "$SCRIPT_DIR/test_dmf_get_licenses.sh" "$BUILD_DIR"
+bash "$SCRIPT_DIR/test_dmf_get_packages.sh" "$BUILD_DIR"
 
 echo "=== All dmf-get integration tests passed! ==="
 cd ..

tests/integration/test_dmf_get_packages.sh

@@ -0,0 +1,183 @@
+#!/bin/bash
+# Verify separate package roots using local archives and actual installed files.
+set -euo pipefail
+
+BUILD_DIR="$(cd "${1:-../build}" && pwd)"
+DMF_GET="$BUILD_DIR/bin/tools/dmf-get"
+TEST_DIR="$(mktemp -d)"
+trap 'rm -rf "$TEST_DIR"' EXIT
+cd "$TEST_DIR"
+export DMOD_CACHE_DIR="$TEST_DIR/cache"
+unset DMOD_PACKAGE_DIR DMOD_LICENSE_DIR DMOD_VIEWS_DIR
+
+fail() {
+    echo "FAIL: $*"
+    cat install.log 2>/dev/null || true
+    exit 1
+}
+
+mkdir -p parent/docs parent/settings parent/views parent/tools child/configs
+echo 'parent dmf' > parent/parent.dmf
+echo 'parent dmfc' > parent/parent.dmfc
+echo 'helper dmfc' > parent/tools/helper.dmfc
+echo 'parent docs' > parent/docs/README
+echo 'parent config' > parent/settings/board.ini
+echo 'parent view' > parent/views/main.dmv
+echo 'parent license' > parent/LICENSE
+echo child > parent/parent.dmd
+cat > parent/parent.dmr <<'DMR'
+dmf=./parent.dmf => ${destination}/${module}.dmf
+dmfc=./parent.dmfc => ${destination}/${module}.dmfc
+dmfc=./tools/helper.dmfc => ${destination}/${module}/tools/helper.dmfc
+docs=./docs => ${destination}/${module}/docs
+config=./settings => ${destination}/custom-config/${module}
+views=./views => ${views_dir}
+license=./LICENSE => ${destination}/${module}/LICENSE
+dmd=./parent.dmd => ${destination}/${module}.dmd
+DMR
+# No .dmr: the complete package must move, and config lookup must still work.
+echo 'child dmf' > child/child.dmf
+echo 'child config' > child/configs/board.ini
+(cd parent && zip -qr ../parent.zip .)
+(cd child && zip -qr ../child.zip .)
+echo 'raw module' > raw.dmf
+echo 'packed modules' > bundle.dmp
+cat > manifest.dmm <<MANIFEST
+parent $TEST_DIR/parent.zip
+child $TEST_DIR/child.zip
+raw $TEST_DIR/raw.dmf
+bundle $TEST_DIR/bundle.dmp
+MANIFEST
+args=(-m "$TEST_DIR/manifest.dmm" --no-fallback --skip-arch-check --skip-dmod-ver-check -y)
+run_install() {
+    "$DMF_GET" "${args[@]}" "$@" > install.log 2>&1 || fail "install $*"
+}
+check_binary_dir() {
+    local unexpected
+    unexpected="$(find "$1" -mindepth 1 ! \( -type f \( -name '*.dmf' -o -name '*.dmfc' \) \))"
+    test -z "$unexpected" || fail "non-module entries in $1: $unexpected"
+}
+check_packages() {
+    local packages="$1" binaries="$2"
+    cmp parent/parent.dmfc "$binaries/parent.dmfc"
+    cmp parent/tools/helper.dmfc "$binaries/helper.dmfc"
+    cmp child/child.dmf "$binaries/child.dmf"
+    cmp parent/docs/README "$packages/parent/docs/README"
+    cmp parent/settings/board.ini "$packages/custom-config/parent/board.ini"
+    cmp parent/views/main.dmv "$packages/views/parent/main.dmv"
+    cmp parent/LICENSE "$packages/parent/LICENSE"
+    cmp parent/parent.dmd "$packages/parent.dmd"
+    cmp parent/parent.dmr "$packages/parent.dmr"
+    cmp child/configs/board.ini "$packages/child/configs/board.ini"
+    test ! -e "$packages/parent.zip"
+    test ! -e "$packages/child.zip"
+    check_binary_dir "$binaries"
+}
+
+"$DMF_GET" --help > help.txt
+grep -q -- '--package-dir' help.txt
+grep -q 'DMOD_PACKAGE_DIR' help.txt
+
+# Backward compatibility: without the setting metadata/resources share -o.
+run_install -o legacy parent
+test -f legacy/parent.dmfc
+test -f legacy/parent.dmd
+test -f legacy/parent/docs/README
+test -f legacy/child/configs/board.ini
+test -f legacy/views/parent/main.dmv
+
+# Nested relative path with spaces/trailing slash, recursive deps, custom config mapping.
+run_install -o binaries --package-dir 'nested/unpacked packages/' --config board.ini --config-dir installed-config parent
+check_packages 'nested/unpacked packages' binaries
+cmp parent/settings/board.ini installed-config/parent/board.ini
+
+# Explicit uncompressed preference keeps the binary root flat as well.
+run_install -o dmf-binaries --package-dir dmf-packages --type dmf --no-dependencies parent
+cmp parent/parent.dmf dmf-binaries/parent.dmf
+test ! -e dmf-binaries/parent.dmfc
+test ! -e dmf-binaries/helper.dmfc
+check_binary_dir dmf-binaries
+
+# Reinstall into the same roots; directory resources must not gain another level.
+run_install -o binaries --package-dir 'nested/unpacked packages' parent
+check_packages 'nested/unpacked packages' binaries
+test ! -e 'nested/unpacked packages/parent/docs/docs'
+
+# .dmd installs, tagged configs, no-.dmr fallback, and an absolute root whose
+# final component equals a module name (must not be stripped by normalization).
+printf 'parent driver=board.ini\nchild service=board.ini\n' > dependencies.dmd
+run_install -o dmd-binaries --package-dir "$TEST_DIR/separate/parent" -d dependencies.dmd \
+    --config-map 'driver:drivers;service:services'
+check_packages "$TEST_DIR/separate/parent" dmd-binaries
+cmp parent/settings/board.ini drivers/parent/board.ini
+cmp child/configs/board.ini services/child/board.ini
+
+export DMOD_PACKAGE_DIR="$TEST_DIR/from env"
+run_install -o env-binaries parent
+check_packages "$DMOD_PACKAGE_DIR" env-binaries
+export DMOD_PACKAGE_DIR="$TEST_DIR/unused env"
+run_install -o cli-binaries --package-dir cli-packages parent
+check_packages cli-packages cli-binaries
+test ! -e "$DMOD_PACKAGE_DIR"
+export DMOD_PACKAGE_DIR=''
+run_install -o empty-env --no-dependencies child
+test -f empty-env/child/configs/board.ini
+unset DMOD_PACKAGE_DIR
+
+# Mini mode and explicit view/license destinations keep their existing priorities.
+run_install -o mini-binaries --package-dir mini-packages --mini --no-dependencies parent
+check_binary_dir mini-binaries
+cmp parent/views/main.dmv mini-packages/views/parent/main.dmv
+test -f mini-packages/parent.dmd
+test -f mini-packages/parent.dmr
+test ! -e mini-packages/parent/docs
+test ! -e mini-packages/parent/LICENSE
+run_install -o override-binaries --package-dir override-packages --mini --no-dependencies \
+    --views-dir custom-views --license-dir custom-licenses parent
+check_binary_dir override-binaries
+cmp parent/views/main.dmv custom-views/parent/main.dmv
+cmp parent/LICENSE custom-licenses/parent/LICENSE
+test ! -e override-packages/views
+
+# Raw binaries stay in the module output; DMP archives go to the package root.
+DMOD_DMF_DIR=raw-binaries run_install --package-dir raw-packages --no-dependencies raw
+cmp raw.dmf raw-binaries/raw.dmf
+check_binary_dir raw-binaries
+run_install -o bundle-binaries --package-dir bundle-packages --no-dependencies bundle
+cmp bundle.dmp bundle-packages/bundle.dmp
+check_binary_dir bundle-binaries
+
+# Resource-only commands keep the meaning of their own -o.
+run_install docs parent -o docs-only --package-dir unused-packages
+cmp parent/docs/README docs-only/parent/docs/README
+test ! -e unused-packages
+
+if "$DMF_GET" --package-dir > install.log 2>&1; then
+    fail 'missing argument should fail'
+fi
+grep -q -- '--package-dir requires an argument' install.log
+for path in '' '--mini' 'unsafe;path' "$(printf '%0512d' 0)"; do
+    if "$DMF_GET" "${args[@]}" -o invalid --package-dir "$path" child > install.log 2>&1; then
+        fail "invalid package path should fail: $path"
+    fi
+done
+
+echo 'not a directory' > blocked
+if "$DMF_GET" "${args[@]}" -o blocked-binaries --package-dir blocked child > install.log 2>&1; then
+    fail 'unwritable package root should fail'
+fi
+check_binary_dir blocked-binaries
+
+# Existing blocking paths must fail metadata copies and no-.dmr extraction.
+mkdir -p metadata-blocked/parent.dmd
+if "$DMF_GET" "${args[@]}" -o metadata-binaries --package-dir metadata-blocked --no-dependencies parent > install.log 2>&1; then
+    fail 'metadata destination directory should not be treated as a file'
+fi
+mkdir -p contents-blocked
+echo 'not a directory' > contents-blocked/child
+if "$DMF_GET" "${args[@]}" -o contents-binaries --package-dir contents-blocked --no-dependencies child > install.log 2>&1; then
+    fail 'unwritable unpacked package directory should fail'
+fi
+check_binary_dir contents-binaries
+
+echo 'All dmf-get package directory tests passed!'

tools/system/dmf-get/README.md

@@ -182,6 +182,7 @@ This copies `mymodule`'s configuration to `./config/drivers/mymodule/stm32f746g-
 - `-d, --dependencies <path>` - Path or URL to dependencies (.dmd) file
 - `-m, --manifest <path>` - Path or URL to manifest file
 - `-o, --output-dir <path>` - Output directory for downloaded modules
+- `--package-dir <path>` - Separate root for package resources, unpacked directories and `.dmd`/`.dmr` metadata (default: `DMOD_PACKAGE_DIR`, otherwise the module output directory).
 - `--license-dir <path>` - Install `.dmr` license resources to `<path>/<module>/<source-name>`, including dependencies. Defaults to `DMOD_LICENSE_DIR`; otherwise uses the package destination. Also collects licenses with `--mini`.
 - `--config <path>` - Configuration file to copy from module (for single module only)
 - `--config-dir <path>` - Directory where configuration files should be copied
@@ -198,6 +199,31 @@ This copies `mymodule`'s configuration to `./config/drivers/mymodule/stm32f746g-
 - `-h, --help` - Show help message
 - `-v, --version` - Show version information
 
+### Separate Package Directory
+
+```bash
+dmf-get -o ./dmf --package-dir ./packages mymodule
+DMOD_PACKAGE_DIR=./packages dmf-get -o ./dmf -d dependencies.dmd
+```
+
+With a separate package root, `./dmf` receives only `.dmf`/`.dmfc` binaries;
+package resources and metadata go under `./packages`. `.dmr` resources use that
+root for `${destination}`, while module binaries are installed directly in `-o`
+by source filename. Packages without `.dmr` retain their structure under
+`./packages/<module>`. Dependencies and configuration copying use the new root.
+
+The CLI option overrides a non-empty `DMOD_PACKAGE_DIR`. When neither is set,
+the existing behavior is preserved. Relative paths use the working directory, and missing parent
+directories are created. Existing installations are not moved or cleaned up.
+Choose a package directory outside `-o` to keep the module directory separate.
+
+Default views go to `<package-dir>/views/<module>`. Explicit view and license
+directories still override their destinations, and other explicit `.dmr` paths
+remain effective. Mini mode keeps its resource filtering. ZIP downloads are
+staged in the package root and removed after extraction; `.dmp` archives are
+stored there. Temporary extraction and caching are unchanged. The dedicated
+`headers`, `docs` and `lib` commands retain their existing `-o` behavior.
+
 ## Caching
 
 `dmf-get` caches downloaded manifests and package files to reduce network requests and speed up subsequent downloads. The cache is stored in `~/.cache/dmod/dmf-get/` by default.