Failure-Aware Autonomous Continual RL

Implementation for the RIPL Lab Fall 2026 prospective student assignment.

This project investigates whether failure-aware residual reinforcement learning, guided by LLM-generated rewards and failure-focused episode sampling, can improve recovery from novel Push-T failure modes while preserving nominal policy performance.

Pipeline

Stage	Name	Description
T-I	Visual Diffusion Policy	Train a vision-based Diffusion Policy on Push-T using imitation learning.
T-II	Failure Identification	Systematically identify and characterize reproducible failure modes of the trained policy.
T-III	LLM-Driven Reward	Use an LLM to generate dense rewards and episode samplers targeted at the identified failure regimes.
T-IV	Residual RL	Train a residual policy with PPO using the generated rewards and evaluate both targeted failure recovery and nominal performance.

Repository Structure

configs/    Experiment configurations
scripts/    Experiment entry points
src/        Core implementation
docs/       Research notes and debugging documentation

Reproduction

Google Colab

The recommended setup is Google Colab with a T4 GPU.

1. Open RIPL_Full_Colab_v2.ipynb.
2. Select a GPU runtime (T4 GPU, Runtime Version 2026.07).
3. Upload the repository ZIP when prompted.
4. Run the notebook cells sequentially.

The notebook installs the required dependencies, downloads the official ManiSkill Push-T demonstrations, prepares the training data, and runs the experimental pipeline.

The notebook is the recommended method for reproducing the experiments when a compatible local GPU environment is unavailable.

Local Setup

Python 3.12 is recommended.

git clone <REPOSITORY_URL>
cd <REPOSITORY_NAME>

python3.12 -m venv .venv
source .venv/bin/activate

pip install -r requirements.txt

Run the experiment stages using the corresponding scripts in scripts/. Each script documents its required arguments and configuration.

python scripts/train_diffusion.py --help
python scripts/collect_rollouts.py --help
python scripts/analyze_failures.py --help
python scripts/generate_reward.py --help
python scripts/train_residual.py --help
python scripts/evaluate_all.py --help

Reproducibility

• Experiments record the random seed, training configuration, hardware, training time, and evaluation metrics.
• Evaluation is performed over 100 rollouts across 3 random seeds, following the assignment protocol.
• Generated datasets, checkpoints, videos, and experiment outputs are not stored in the repository. They can be regenerated using the provided scripts and configurations.

Results

The experiments are designed around three evaluation questions:

1. Does the failure-discovery procedure identify reproducible failure regimes?
2. Does LLM-generated reward shaping concentrate learning on the identified failures?
3. Can residual PPO improve recovery from those failures without substantially degrading nominal behavior?

The preliminary experiments support the feasibility of the proposed pipeline. Failure analysis identified distribution-shifted initial configurations as the primary regime of interest, particularly changes in object position/orientation and end-effector approach geometry. The T-III reward formulation therefore emphasizes progress toward establishing valid contact, maintaining object contact during the push, and reducing object-to-target distance.

For T-IV, evaluation is reported separately on nominal and failure-focused initial-state distributions. This separation is important because optimizing only the failure distribution can improve recovery while unnecessarily changing behavior that already works on the nominal distribution.

Evaluation Metrics

• Nominal Success: success rate on the original Push-T initial-state distribution.
• Failure-Regime Success: success rate on the perturbed initial-state distribution identified in T-II.
• Object-to-Target Distance: final distance between the manipulated object and target.
• Recovery Improvement: change in failure-regime success relative to the frozen base policy.
• Nominal Retention: change in nominal success after residual adaptation.

The final numerical results are generated by scripts/evaluate_all.py and are stored with the experiment configuration and random seed so that reported values can be reproduced.