Skip to content

Standard: on agent.computer/v1 an agent can reach the desktop outside its steps #223

Closed

Description

@pierreleca

Priority

extreme

Components

  • app/nullius (backend)
  • docs and the site
  • catalogue (benchmarks, agents)
  • the standard (ports, kits, relay, runs)

Today

agent.computer/v1 gives the agent tools that reach the desktop without taking a step: python runs code in it, read and screenshot look at it at any time, upload writes to it at any time. A benchmark that counts one step for each decision of the agent, as OSWorld does, cannot hold an agent to that count: between two steps the agent can look, or act through python, as often as it likes. An agent that brings a driver of its own has to install and start it through those same tools, after its task began.

Wanted

The same port under the same name, with stricter rules (nothing was released with the first ones):

  • Once the agent has its task, only acting calls reach the desktop: step (several actions, one step), wait and the single-action tools. python and read are removed.
  • screenshot returns the observation of the last step and takes no new picture. Looking again is wait, which costs a step.
  • A window before the task: upload and prepare (code run in the desktop) work until the run hands the task over, and cost no step. An agent's own driver is installed and started there.
  • A step may name files of the agent's folder in the desktop, and its answer holds them: a driver answers through files.
  • The kit checks the new rules, with a ninth check, late, for an agent that looks again after the screen changed.
  • OSWorld's agent side keeps the upstream's loop and evaluator unmodified.

Acceptance

  • prove passes for the kit: the null control fails every check, the reference passes all nine, and a script that clicks without looking fails late.
  • After the task is given, upload and prepare are refused and take no step, and python and read are not offered.
  • For a unit run by an agent that holds the port, the steps the port counts equal the steps of the upstream's own trace.
  • An agent with no driver of its own passes the kit from the port's own words.
  • OSWorld with its own agents and the null control give the same records as before.
  • The port's page states the window, the tools and the rules.

Dependencies and related

#187 (closed) and #188 (merged): the port's first rules and its kit. #134 (merged): OSWorld2 names the port and has no agent side yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions