Skip to content

Standard: on agent.computer/v1 a failed prepare action answers as a success, and its record keeps keys in clear #235

Open

Description

@pierreleca

Priority

extreme

Components

  • app/nullius (backend)
  • docs and the site
  • catalogue (benchmarks, agents, capabilities)
  • the standard (ports, kits, relay, runs)

Evidence

Three faults in what #224 added, seen on main with the suite's fake desktop:

  1. The prepare record keeps keys in clear. prepare writes its actions and file names to the run's record unchanged. After the action open('agent/sk-xxxxxxxxxxxxxxxxxxxxxxxx.txt', 'w').write('sk-xxxxxxxxxxxxxxxxxxxxxxxx'), the record holds that same text. The relay and the run's readers redact such values.
  2. A failed prepare action answers as a success. With the actions ["WAIT", "print('before')", "raise RuntimeError('installation failed')", "print('after')"] and the file missing, the call answers {"files": {"missing": null}} with no error, and print('after') still runs.
  3. The kit's late reference passes on blank pictures. A reference that read its target from the observation would fail there.

The controls recorded in the kit were also taken before #220 moved the kit's desktop to Pillow 13.0.0.

Screenshot

No response

Reproduction

Before the task is given:

  1. Call prepare with {"actions": ["open('agent/sk-xxxxxxxxxxxxxxxxxxxxxxxx.txt', 'w').write('sk-xxxxxxxxxxxxxxxxxxxxxxxx')", "WAIT"], "files": ["sk-xxxxxxxxxxxxxxxxxxxxxxxx.txt"]} and read the prepare record.
  2. Call prepare with {"actions": ["WAIT", "print('before')", "raise RuntimeError('installation failed')", "print('after')"], "files": ["missing"]} and read the answer.
  3. Run the kit's late reference with blank observations.

Expected

  1. The record holds [REDACTED] in place of each credential-shaped value, by the redaction the relay already uses. Other text is recorded unchanged.
  2. The call answers with an error that names the failed action, and the actions after it are not run.
  3. The late reference finds the button in the observation it was sent and fails on blank pictures.
  4. The kit's record holds controls taken on the desktop that main builds.

Acceptance

  • A credential-shaped value in a prepare action or file name is written as [REDACTED] in the prepare record; an action without one is recorded unchanged.
  • A prepare action that fails answers with an error naming it, and the later actions are not run.
  • The late reference fails when its observations are blank and passes on the kit's desktop.
  • prove passes for the kit on the desktop with Pillow 13.0.0: the null control fails every check, the reference passes all nine, and a script that clicks without looking fails late.
  • The port's page states both changes of prepare.

Dependencies and related

#223 (closed) and #224 (merged): the port's strict rules, where the three faults came in. #220 (merged): Pillow 13.0.0 in the kit's desktop.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions