Task-Aware Data Validation using Language Models
PrismaDV analyzes downstream task code and sample data to automatically generate validation constraints. SIFTA (Selective Informative Feedback for Task Adaptation) adapts PrismaDV from observed task and test outcomes, using failure precision to focus prompt updates on informative constraint failures.
| Notebook | Reproduces |
|---|---|
workflow_prismadv/results/combined.ipynb |
Table 1; Tables E, H |
workflow_prismadv/results/ablation.ipynb |
Table 2 |
workflow_sifta/scoring.ipynb |
Table 3; Table G |
workflow_sifta/ablation.ipynb |
Table 4; Tables C, D, K |
workflow_prismadv/eid_bench_experiments/results_analysis/cost_runtime_f1_cached.ipynb |
Figure 3 |
workflow_prismadv/results/usage.ipynb |
Table F |
workflow_prismadv/results/per_dataset.ipynb |
Table J |
Performance is evaluated using three comprehensive benchmarks for task-aware data validation:
- ICDBench - Individual Constraint Discovery (63 test cases)
- EIDBench-synth - End-to-End Error Impact Detection (5 datasets, 60 tasks)
- EIDBench-real - End-to-End Error Impact Detection on 3 real repo-derived tasks
See benchmarks overview for detailed comparison and documentation.
-
Install uv (if not already installed):
curl -LsSf https://astral.sh/uv/install.sh | sh -
Install dependencies:
uv sync
-
Create a
.envfile with required API keys:OPENAI_API_KEY=your_openai_api_key GEMINI_API_KEY=your_gemini_api_key SPARK_VERSION=3.5
-
Run tests:
uv run pytest
All task-aware data validation unit tests are saved in the data_processed/ directory. This
directory contains unit tests for all tasks generated using different approaches.
Example: View the data unit tests that PrismaDV (gpt-6-sol) generated for general_task_5.py from the IPL_win_prediction dataset in EIDBench-synth here.
The SIFTA and GEPA optimization runs are in optimization_runs/: one directory per run, with
its settings and scores in summary.json and the program of every round.
