Skip to content

Learn the file-IO graph without root, using strace - #2438

Open
sawenzel wants to merge 1 commit into
AliceO2Group:masterfrom
sawenzel:swenzel/filegraph-unprivileged
Open

Learn the file-IO graph without root, using strace#2438
sawenzel wants to merge 1 commit into
AliceO2Group:masterfrom
sawenzel:swenzel/filegraph-unprivileged

Conversation

@sawenzel

Copy link
Copy Markdown
Contributor

This makes the file-IO-graph learning pluggable and adds an strace backend, so a pilot run no longer needs a binary with CAP_SYS_ADMIN. The graph it produces is the one --remove-files-early already reads back.

  • The runner gains --filegraph-backends. Naming several at once runs them side by side, which is how they are compared. O2DPG_PRODUCE_FILEGRAPH still selects fanotify and names its monitor.
  • The strace backend wraps each task command, so a file access is attributed by the trace it lands in rather than by walking /proc after the event. --seccomp-bpf keeps the cost at about 59 us per traced open; the backend probes for it.
  • filegraph_report.py holds the exclusion rules, the ./tfN -> ./tfX templating, the JSON schema and the graphviz rendering, so the two analysers cannot drift. analyse_FileIO_v2.py reproduces its previous output byte for byte.
  • compare_reports.py grades one report against another. A missing edge deletes a file a later task still reads; an extra edge only delays the deletion. The verdicts EXACT, SAFE and UNSAFE follow that asymmetry.
  • tests/equivalence_test.py runs a workflow whose graph is known by construction and grades every backend against it, in seconds and with no ALICE software. strace comes out EXACT.
  • monitor_fileaccess_v2.cpp spun forever on a queue overflow, because the overflow branch skipped FAN_EVENT_NEXT and re-tested the same event.
  • 33 offline tests come with it, plain unittest so they also run on a worker node, and a CI job runs them.

Part of the o2dpg_workflow_runner.py refactoring.

@github-actions

Copy link
Copy Markdown

REQUEST FOR PRODUCTION RELEASES:
To request your PR to be included in production software, please add the corresponding labels called "async-" to your PR. Add the labels directly (if you have the permissions) or add a comment of the form (note that labels are separated by a ",")

+async-label <label1>, <label2>, !<label3> ...

This will add <label1> and <label2> and removes <label3>.

The following labels are available
async-2023-pbpb-apass4
async-2023-pp-apass4
async-2024-pp-apass1
async-2022-pp-apass7
async-2024-pp-cpass0
async-2024-PbPb-apass1
async-2024-ppRef-apass1
async-2024-PbPb-apass2
async-2023-PbPb-apass5

This makes the file-IO-graph learning pluggable and adds an strace backend,
so a pilot run no longer needs a binary with CAP_SYS_ADMIN. The graph it
produces is the one --remove-files-early already reads back.

- The runner gains --filegraph-backends. Naming several at once runs them
  side by side, which is how they are compared. O2DPG_PRODUCE_FILEGRAPH
  still selects fanotify and names its monitor.
- The strace backend wraps each task command, so a file access is
  attributed by the trace it lands in rather than by walking /proc after
  the event. --seccomp-bpf keeps the cost at about 59 us per traced open;
  the backend probes for it.
- filegraph_report.py holds the exclusion rules, the ./tfN -> ./tfX
  templating, the JSON schema and the graphviz rendering, so the two
  analysers cannot drift. analyse_FileIO_v2.py reproduces its previous
  output byte for byte.
- compare_reports.py grades one report against another. A missing edge
  deletes a file a later task still reads; an extra edge only delays the
  deletion. The verdicts EXACT, SAFE and UNSAFE follow that asymmetry.
- tests/equivalence_test.py runs a workflow whose graph is known by
  construction and grades every backend against it, in seconds and with no
  ALICE software. strace comes out EXACT.
- monitor_fileaccess_v2.cpp spun forever on a queue overflow, because the
  overflow branch skipped FAN_EVENT_NEXT and re-tested the same event.
- 33 offline tests come with it, plain unittest so they also run on a
  worker node, and a CI job runs them.

Part of the o2dpg_workflow_runner.py refactoring.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant