Title
Agentic Transfer Bench
Leaders
Roza G. Bayrak (@rgbayrak)
Collaborators
No response
Project description
Can an AI agent make machine-learning workflows easier to build and maintain?
This project will explore the potential of agentic pipelines to automate complex, non-trivial decisions in machine-learning workflows. We’ll use a concrete task as a testbed: adapting an open PPG foundation model to a new dataset.
Key questions include: How can we quantify dataset shift? How should the degree of shift inform the choice of adaptation strategy, from linear probing to full fine-tuning? Can an LLM agent make these decisions reliably? And can it reduce human effort without introducing unacceptable errors?
The final outcome will be a practical understanding of agentic pipelines, supported by our experiments and documented for future use.
Link to project repository/sources
https://github.com/rgbayrak/Agentic-Transfer-Bench
Concerete goals with specific tasks for Brainhack Vanderbilt 2026
We research and plan first; code last. We will write code to test what we learnt as a team.
Document → Implement → Test→ Report
Phase 1 — Understand (research & writing track; everyone)
- Read the primary literature on agentic systems; write
docs/agentic-pipelines.md, a plain-language primer covering the script→agent spectrum, agent anatomy, core vocabulary, when agency helps vs. hurts, patterns, anti-patterns, evaluation criteria, and an annotated reading list.
- Map the real pipeline in
docs/case-study-ppg.md: for each step (fetch → parse → preprocess → embed → decide → adapt → verify), mark it deterministic or agentic, and write a testable hypothesis for why.
Deliverable: primer + decision map, reviewed as a team before any code is written.
Phase 2 — Build
Deterministic backbone (scaffolding):
fetch_dataset — WildPPG and OpenNeuro ds006644parse_dataset, preprocess (resample / window / normalize)load_fm + embed (PaPaGei-S, optionally Pulse-PPG)train_linear_probe with subject-wise cross-validationcompute_shift
Build a minimal eval harness that runs both variants (agent and rule) on ≥2 datasets and records outcome, decision quality, and cost (tokens / $ / wall-clock / step count).
Phase 3 — Report
- Write
docs/findings.md: one section per decision point, structured as what we expected → what we saw → recommendation, ending with a revised, one-sentence definition of "agentic" based on what you actually found.
Deliverable: findings write-up; this is the artifact you'd present or submit.
Good first issues
- Explore and document the vocabulary.
Orchestrator / lead agent: What is an orchestrator? What role does it play in an agentic pipeline? How does it coordinate subtasks, tools, and other agents?
- Specialized sub-agents / workers: What makes a sub-agent “specialized”? How do sub-agents differ from ordinary functions or tools? When is using multiple agents useful?
- Deterministic tools & APIs: What distinguishes a tool or API from an agent? How do agents interact with deterministic tools? What role do tools such as APIs, scripts, and MCP play in an agentic workflow?
- Agentic decision-making: What makes a decision or workflow step “agentic”? How does this differ from a fixed, deterministic pipeline?
- Human-in-the-loop: What does it mean for an agent to remain in the loop without becoming autonomous? Where should humans review, approve, override, or redirect agent decisions?
- Agentic vs. autonomous: Are these actually different concepts? What level of autonomy is necessary for something to be considered agentic?
- Identify and document the tools needed for the adaptation of a PPG time-series foundation model to a new, unseen dataset.
What tools are needed to:
- inspect and understand the new dataset?
- run experiments and evaluate models?
- measure and diagnose domain shift?
- evaluate possible adaptation strategies?
Which decisions should remain human-in-the-loop?
Skills
No prior agents or PPG experience required, the research-and-writing tasks are open to everyone!
"Working" = you've used it on a real task and can debug it. Only a subset of the team needs specialized skills.
- Technical research and writing — working, everyone on Phase 1. Explain a concept plainly, cite sources, state a claim and its evidence.
- Python — working, all Phase-2 contributors. Functions, dataclasses / type hints,
pytest, reading someone else's module.
- Command line / dev environment — working, git (branch / rebase / PR), venv or conda, running scripts,
ssh.
- Working ML knowledge — working, 2–3 people.
- Stats for dataset shift — working, 1–2 people.
- Deep-learning tooling — familiar, 1 person. Load a checkpoint, run a forward pass, batching, CPU/GPU device handling.
- Claude Code specifically — nice to have, lead + 1. Only the orchestration authors need the agent runtime; everyone else uses Copilot / Codex / Cursor / none.
Onboarding documentation
https://github.com/rgbayrak/Agentic-Transfer-Bench see README.md
What will participants learn?
By the end of BrainHack, you should be able to:
- Define "agentic" precisely enough to argue about it, distinguish a script, a DAG, a tool-using LLM call, and an orchestrator, using consistent vocabulary (tool / skill / agent / orchestrator).
- Communicate a technical finding in plain language, with citations and a clear claim/evidence structure.
- Practice real collaborative engineering: forking, branching, pull requests, issues, and pulling only the data you need from a BIDS dataset via DataLad/git-annex.
- Quantify dataset shift (covariate, label, and concept shift) using at least two of: MMD, proxy-A-distance, Population Stability Index, KL divergence.
- Use frozen-embedding linear probing and light fine-tuning (LoRA) to adapt a foundation model to a new dataset.
- Understand how to evaluate an agentic system on five axes: outcome, decision quality, cost, robustness, transparency (decision logging + deterministic fallback).
Public data to use
Number of collaborators
4+
Credit to collaborators
Project collaborators will be included as co-authors on any resulting write-up, preprint, or poster, and acknowledged in the GitHub repository via their commit history and a CONTRIBUTORS file. Anyone who contributes documentation, code, or analysis counts.
Image
in prep
Project Summary
This project will explore when and how to leverage agentic AI in ML pipelines, using PPG foundation-model adaptation to a new dataset as a real-world testbed.
Type
method_development, pipeline_development, documentation
Development status
0_concept_no_content
Topic
machine_learning, deep_learning, physiology
Tools
Pytorch, Datalad, other
Programming language
Python
Modalities
Peripheral physiology
Git skills
0_no_git_skills, 1_commit_push, 2_branches_PRs, 3_continuous_integration
Anything else?
No response
Things to do after the project is submitted and ready to review.
Title
Agentic Transfer Bench
Leaders
Roza G. Bayrak (@rgbayrak)
Collaborators
No response
Project description
Can an AI agent make machine-learning workflows easier to build and maintain?
This project will explore the potential of agentic pipelines to automate complex, non-trivial decisions in machine-learning workflows. We’ll use a concrete task as a testbed: adapting an open PPG foundation model to a new dataset.
Key questions include: How can we quantify dataset shift? How should the degree of shift inform the choice of adaptation strategy, from linear probing to full fine-tuning? Can an LLM agent make these decisions reliably? And can it reduce human effort without introducing unacceptable errors?
The final outcome will be a practical understanding of agentic pipelines, supported by our experiments and documented for future use.
Link to project repository/sources
https://github.com/rgbayrak/Agentic-Transfer-Bench
Concerete goals with specific tasks for Brainhack Vanderbilt 2026
We research and plan first; code last. We will write code to test what we learnt as a team.
Document → Implement → Test→ Report
Phase 1 — Understand (research & writing track; everyone)
docs/agentic-pipelines.md, a plain-language primer covering the script→agent spectrum, agent anatomy, core vocabulary, when agency helps vs. hurts, patterns, anti-patterns, evaluation criteria, and an annotated reading list.docs/case-study-ppg.md: for each step (fetch → parse → preprocess → embed → decide → adapt → verify), mark it deterministic or agentic, and write a testable hypothesis for why.Deliverable: primer + decision map, reviewed as a team before any code is written.
Phase 2 — Build
Deterministic backbone (scaffolding):
fetch_dataset— WildPPG and OpenNeuro ds006644parse_dataset,preprocess(resample / window / normalize)load_fm+embed(PaPaGei-S, optionally Pulse-PPG)train_linear_probewith subject-wise cross-validationcompute_shiftBuild a minimal eval harness that runs both variants (agent and rule) on ≥2 datasets and records outcome, decision quality, and cost (tokens / $ / wall-clock / step count).
Phase 3 — Report
docs/findings.md: one section per decision point, structured as what we expected → what we saw → recommendation, ending with a revised, one-sentence definition of "agentic" based on what you actually found.Deliverable: findings write-up; this is the artifact you'd present or submit.
Good first issues
Skills
No prior agents or PPG experience required, the research-and-writing tasks are open to everyone!
"Working" = you've used it on a real task and can debug it. Only a subset of the team needs specialized skills.
pytest, reading someone else's module.ssh.Onboarding documentation
https://github.com/rgbayrak/Agentic-Transfer-Bench see
README.mdWhat will participants learn?
By the end of BrainHack, you should be able to:
Public data to use
Number of collaborators
4+
Credit to collaborators
Project collaborators will be included as co-authors on any resulting write-up, preprint, or poster, and acknowledged in the GitHub repository via their commit history and a
CONTRIBUTORSfile. Anyone who contributes documentation, code, or analysis counts.Image
in prep
Project Summary
This project will explore when and how to leverage agentic AI in ML pipelines, using PPG foundation-model adaptation to a new dataset as a real-world testbed.
Type
method_development, pipeline_development, documentation
Development status
0_concept_no_content
Topic
machine_learning, deep_learning, physiology
Tools
Pytorch, Datalad, other
Programming language
Python
Modalities
Peripheral physiology
Git skills
0_no_git_skills, 1_commit_push, 2_branches_PRs, 3_continuous_integration
Anything else?
No response
Things to do after the project is submitted and ready to review.
Hi @brainhack-vandy/project-monitors my project is ready!