Skip to content

Migrate configuration from Cerberus to Pydantic - #37

Merged
roelofsc merged 18 commits into
developfrom
refactor-configuration
Nov 9, 2026
Merged

roelofsc merged 18 commits into
developfrom
refactor-configuration

Conversation

@roelofsc

@roelofsc roelofsc commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

This PR replaces the Cerberus-based BaseConfig/Config system with a Pydantic Config (BaseModel) model. All configuration sections (train, root_cause_analysis, dtype) have become typed Pydantic models with attribute access (config.train.autoencoder.params) instead of dictionary-style access (config['train']['autoencoder']['params']).

In addition, Config() now generates a complete configuration that can be used directly. It is not optimized to the users data yet however.

Why? The old system just validated a dictionary. Every programmatic change of the configuration meant using nested .get()/.setdefault(), without validation of the input and no IDE support (i.e. tab completion). The configuration class is now a typed object with attribute access. When an attribute is assigned a new value, validation runs automatically. Swapping out a component means setting its name and parameterss directly (i.e. config.train.autoencoder.name = 'lstm', config.train.autoencoder.params = {...}), with autocompletion and inline docs. This will help with development of hyperparameter optimisation workflows.


Core change

energy_fault_detector/config/config.py

The central file. Key changes are:

  • Config is now a Pydantic BaseModel. Section models:
    TrainConfig, AutoencoderConfig, AnomalyScoreConfig, DataPreprocessorConfig,
    ThresholdSelectorConfig, DataClippingConfig, DataSplitterConfig, RootCauseAnalysisConfig.
  • All sections have defaults: Config() with no arguments produces a ready-to-use configuration (MultilayerAutoencoder, RMSE score, quantile threshold, default preprocessing, BlockDataSplitter).
  • config_dict deprecated. It uses model_dump(exclude_none=True) under the hood, emits DeprecationWarning. Prefer model_dump() or direct attribute access.
  • Removed: read_config(). Validation/loading is now done by Pydantic on construction. The constructor also accepts a positional dict (Config({'train': {...}})) in addition to the existing config_filename / config_dict keyword arguments.
  • Removed: inter-field dependency validator on DataSplitterConfig (fields like train_block_size are simply ignored by the splitter if not applicable).

energy_fault_detector/config/base_config.py

Consolidated into config.py. The Cerberus schema and BaseConfig class are gone.

energy_fault_detector/config/__init__.py

Updated exports: Config from config.py, generate_quickstart_config from quickstart_config.py.

energy_fault_detector/config/quickstart_config.py

Migrated to use Config(config_dict=...) instead of the old BaseConfig.update_config() pattern.

pyproject.toml

Dependency change: cerberus → pydantic>=2.0.


Resulting changes

These files changed because the Config API changed. No logic changes. Just dict-style access → attribute access.

energy_fault_detector/fault_detector.py

  • self.config.data_clipping → self.config.train.data_clipping
  • self.config.data_clipping_params.copy() → typed fields from DataClippingConfig model
  • self.config.fit_threshold_on_val → self.config.train.threshold_selector.fit_on_val
  • self.config.arcana_params → self.config.root_cause_analysis.params
  • self.config.protect_conditional_features → self.config.train.protect_conditional_features

energy_fault_detector/core/model_factory.py

  • config["train"] → config.train
  • config.data_preprocessor_steps → train.data_preprocessor.steps
  • ae_params["name"] → ae_config.name, ae_params.get("params", {}) → ae_config.params
  • Added _parse_ts_freq() helper (renamed from _parse_timedelta and moved from config.py to model_factory.py).

energy_fault_detector/core/fault_detection_model.py

  • config.data_split_params → config.train.data_splitter.

energy_fault_detector/evaluation/care2compare.py

  • config['train'].setdefault('data_preprocessor', {}) → config.train.data_preprocessor
  • config.update_config(config.config_dict) → direct attribute mutation on the Pydantic model.
  • Dropped dual old-style/new-style preprocessing branch (only new steps style remains).

energy_fault_detector/quick_fault_detection/configuration.py

  • config['train']['data_preprocessor'].get('params') → config.train.data_preprocessor
  • All dict-style .setdefault() / .get() calls replaced with attribute access and list operations on dp.steps.

energy_fault_detector/quick_fault_detection/optimization.py

  • config.config_dict → config.model_dump(exclude_none=True).

Additional: criticality-based alarm detection (secondary)

This PR also adds an optional, higher-level way to derive alarm events from the predicted anomalies. It is an addition — the existing criticality behaviour is unchanged.

energy_fault_detector/utils/analysis.py

  • New calculate_alarms(...): builds on calculate_criticality and returns (alarm_events, criticality).
    • Raises an alarm once the criticality reaches a threshold and aggregates contiguous alarm timestamps into events with start, end, duration (plus group for MultiIndex data).
    • If criticality_threshold is omitted it defaults to one hour of anomalous behaviour (int(1 hour / time_frequency), or 1 if the frequency exceeds one hour).
    • max_criticality is derived from criticality_threshold + int(alarm_reset_time_minutes / time_frequency), so alarms reset after normal behaviour.
    • Supports MultiIndex (e.g. per-asset) by detecting the datetime level and computing criticality/alarms per group.

energy_fault_detector/core/fault_detection_result.py

  • New FaultDetectionResult.criticality_alarms(...): thin wrapper returning (alarm_events, criticality).
  • FaultDetectionResult.criticality is unchanged and keeps its original signature (normal_idx, init_criticality, max_criticality) and its pd.Series return. The new method is purely additive.

Docs & tests

  • docs/criticality.rst (new): explains the counter and alarm-event derivation; docs/configuration.rst and docs/usage_examples.rst document criticality_alarms().
  • Tests added in tests/utils/test_analysis.py (single- and MultiIndex) and tests/core/test_fault_detection_result.py (legacy criticality and new criticality_alarms).

Note on the criticality config removal

The separate predict config section (config.max_criticality) was removed because it was unused by the criticality code. The criticality functionality itself is not removed — it just continues to be derived from the prediction result, now optionally extended with criticality_alarms().


Tests

tests/test_model_compatibility.py

Add 4 tests verifying that models saved by older EFD versions still load:

  • py312_efd071: Full FaultDetector.load() + predict() (Keras 3.14).
  • py311_efd060: Partial non-Keras component loading (Keras 2.15 models can't be loaded by Keras 3, but config/DataPreprocessor/threshold/score load fine and are run through a partial pipeline).
  • tests/test_data/old_models/: Contains generated model artifacts for two EFD versions (0.6.0/py311/Keras 2.15 and 0.7.1/py312/Keras 3.14), dense + sequence variants.
  • These model files are generated by tests/test_data/old_models/create_legacy_model_fixtures.py. Must be run from the corresponding environment.

Updated existing tests

  • tests/config/test_config.py: Updated for Pydantic validation, added test_empty_config_defaults.
  • tests/config/test_quickstart_config.py: Updated for new Config() constructor.
  • tests/core/test_model_factory.py: Updated config construction, added _parse_ts_freq unit test.
  • tests/test_fault_detector.py: Updated config access patterns.
  • tests/quick_fault_detection/test_configuration.py: Updated.
  • tests/quick_fault_detection/test_quick_fault_detector.py: Updated.

Removed test data

  • tests/test_data/test_bad_early_stopping_config.yaml (early stopping validation is now tested inline).
  • tests/test_data/test_criticality_config.yaml (unused predict section removed from config).

Docs and formatting

Docs

  • docs/configuration.rst: Updated for Pydantic config, BlockDataSplitter default, removed predict section and update_config reference.
  • docs/usage_examples.rst: Migrated to attribute-style access.
  • docs/conf.py: Removed base_config from apidoc skip list.
  • docs/sequence_models.rst: ts_freq string→timedelta64 conversion now happens at model init time, not config-parse time.
  • docs/examples/old_config.yaml (deleted): Showed old params-style preprocessing config.
  • docs/examples/advanced_config.yaml: Removed predict section.
  • README.md: Config() instead of generate_quickstart_config().

Notebooks

  • notebooks/Example - Create new model classes.ipynb: Dict-style → attribute access.
  • notebooks/PreDist/predist_utils.py: Dict-style → attribute access.

Breaking changes

Deprecated

  • config.config_dict → use config.model_dump() (or model_dump(exclude_none=True)).
  • config.update_config({...}) → mutate typed attributes directly (config.train.autoencoder.name = 'lstm') or construct a new Config.

Removed API

Old Replacement
config['train']['autoencoder']['params'] (__getitem__ / __contains__) config.train.autoencoder.params (typed attribute access)
Config.read_config(part) Removed. Pydantic validates on construction; load a section by slicing the dict passed to Config({...})
class BaseConfig (base_config.py) Removed. Consolidated into config.py as the Pydantic Config model
config.data_clipping / config.data_clipping_params config.train.data_clipping.enabled / the typed DataClippingConfig fields
config.fit_threshold_on_val config.train.threshold_selector.fit_on_val
config.protect_conditional_features config.train.protect_conditional_features
config.data_preprocessor_steps() config.train.data_preprocessor.steps
config.arcana_params config.root_cause_analysis.params
config.data_split_params config.train.data_splitter (typed DataSplitterConfig)
config.max_criticality + the predict config section Removed. Criticality config was unused; drop it

@roelofsc
roelofsc force-pushed the refactor-configuration branch from 6f1a6b0 to 5155e4b Compare October 12, 2026 12:06
@roelofsc
roelofsc force-pushed the develop branch 2 times, most recently from a249688 to 500f74d Compare October 12, 2026 12:44
@roelofsc
roelofsc requested a review from cgueck895 November 4, 2026 10:55
@cgueck895
cgueck895 marked this pull request as ready for review November 6, 2026 11:08
@cgueck895
cgueck895 force-pushed the refactor-configuration branch from d2ba305 to 0723f92 Compare November 9, 2026 06:41
roelofsc and others added 18 commits November 9, 2026 11:24
Replace the Cerberus-based BaseConfig/Config with a clean Pydantic design:
- ConfigModel is the top-level Pydantic model with section models for each
  config section (train, predict, root_cause_analysis, etc.)
- Config is a plain wrapper class holding a ConfigModel instance (self._model)
- _SubscriptableModel mixin provides __getitem__/__contains__/get for backward
  compatibility during migration from dict-style to attribute-style access
- Extra validators (_parse_timedelta, _validate_early_stopping,
  _validate_sequence_fit_on_val) run as a @model_validator(mode='after')
- config_dict property rebuilds from the model on every call (model is source
  of truth, not a cached mutable dict)
- Convenience properties now read from self._model instead of config_dict
- Delete base_config.py (consolidated into config.py)
- Fix configuration.py and care2compare.py consumers to use model attributes
  instead of .setdefault() on dict-like objects
- Update pyproject.toml: cerberus -> pydantic>=2.0
…ute access

Remove _SubscriptableModel mixin (__getitem__/__contains__/get) from all
section models and Config. Section models now inherit BaseModel directly.
Config.__getattr__ delegates top-level model fields (train, predict, etc.)
for natural attribute access (config.train.autoencoder.params).

Remove trivial convenience properties that were thin wrappers over model
fields: protect_conditional_features, fit_threshold_on_val, data_clipping
(bool), max_criticality, root_cause_analysis (bool), dtype. The latter two
resolve via __getattr__ delegation to ConfigModel fields.

Keep properties with real logic: data_preprocessor_steps (handles
params->steps migration + deprecation warnings), arcana_params,
data_split_params, data_clipping_params (all return plain dicts via
_model_to_config_dict for **-unpacking consumers).

Migrate all consumers:
- model_factory.py: config['train'] -> config.train, .get('params', {}) -> .params or {}
- fault_detector.py: config.fit_threshold_on_val -> config.train.threshold_selector.fit_on_val,
  config.data_clipping -> config.train.data_clipping is not None, etc.
- configuration.py: config['train']['data_preprocessor'] -> config.train.data_preprocessor,
  dp.get('params') -> dp.params, dp['params']['key'] -> dp.params['key']
- care2compare.py: same pattern
- All test files: conf['train']['autoencoder']['params']['x'] -> conf.train.autoencoder.params['x']
…Config fields, remove dead code

- AutoencoderConfig: default name='default', params={} so Config() works with no args
- DataSplitterConfig: replace params dict with flat fields (train_block_size,
  val_block_size, validation_split, shuffle) matching the YAML format
- DataPreprocessorConfig: default steps=[] instead of None
- Remove update_config (dead code, not called anywhere)
- Remove dead None checks (train, data_preprocessor no longer Optional)
- Simplify DataClipper usage: explicit kwargs instead of model_dump
- Remove unused DataPreprocessorConfig import from care2compare.py
- Clean up or {} guards in model_factory (params default to {})
- Remove orphaned test_criticality_config.yaml
… access

- predist_utils.py: replace config['train']['autoencoder']['params'] with
  config.train.autoencoder.params and config['train']['data_preprocessor']
  with config.train.data_preprocessor.steps
- Example - Create new model classes.ipynb: migrate deprecated
  data_preprocessor.params to steps format
…redict section

- configuration.rst: replace update_config example with direct attribute access
- configuration.rst: remove obsolete 'Prediction options' (predict section removed)
- usage_examples.rst: remove reference to optional predict section
- advanced_config.yaml: remove predict/criticality block
- sequence_models.rst: fix ts_freq attribution (model factory, not config parser)
- Remove orphaned old_config.yaml (deprecated data_preprocessor.params format)
…ware

- Remove from_yaml/from_dict/read_config; the constructor now accepts a
  YAML path or an inline dict (Config('file.yaml'), Config({'train': {...}})).
- Drop the _check_dependencies validator from DataSplitterConfig; the splitter
  ignores irrelevant fields.
- Make _validate_early_stopping type-aware: check val_block_size for block
  splitters, validation_split for sklearn splitters.
- Set DataSplitterConfig default to sklearn with validation_split=0.1 and
  shuffle=False so early stopping and fit_on_val work out of the box.
- Revert DataPrepStepConfig to plain dicts (dict-style access in
  DataPreprocessor and other consumers).
- Add _format_validation_error helper converting Pydantic ValidationError
  into InvalidConfigFile.
- Update README quickstart to Config() and document default split behavior.
- Remove _model_to_config_dict and _dump_value helpers (45 lines);
  Pydantic's model_dump(exclude_none=True) produces identical output
- write_config now uses model_dump instead of config_dict
- config_dict property emits DeprecationWarning, delegates to model_dump
- optimization.py: use config.train.autoencoder.params (attribute access)
  instead of config_dict['train']['autoencoder']['params']
- configuration.py: remove dead dp.params legacy branch (already converted
  to steps during config validation)
- Tests: replace config_dict assertions with model_dump(exclude_none=True)
…rt_config for custom preprocessing

- usage_examples.rst: minimal example uses Config() not generate_quickstart_config
- usage_examples.rst: Configuration section shows Config() first, quickstart second
- configuration.rst: fix data_preprocessor: {} -> omitted/null
- configuration.rst: add Config() snippet before generate_quickstart_config
Reverts the sklearn + validation_split=0.1 default from d005daa7 back to BlockDataSplitter with all optional fields None (splitter falls back to 5040/1680 samples). Re-adds update_config/configuration_file compat shims for old-model loading. Updates docstrings, docs, and test expectations. Adds _parse_ts_freq unit test.
Adds saved-model fixtures for py311_efd060 (Keras 2.15) and py312_efd071 (Keras 3.14), the create_legacy_model_fixtures.py script that generated them, and test_model_compatibility.py which loads and predicts with each fixture on synthetic data. Keras 2 fixtures are marked expectedFailure due to keras.src.engine.functional module path change in Keras 3.
…ibility

The py312_efd071 fixtures were saved with Keras 3.14.1, which adds a
quantization_config key to Dense configs. Python 3.10 caps Keras at
3.12.4 (Keras 3.13+ requires >=3.11), so loading these fixtures failed
with 'Unrecognized keyword arguments passed to Dense'.

Regenerate the fixtures with EFD 0.7.1 on Python 3.10 (Keras 3.12.4),
rename them to py310_efd071, and update the compatibility tests and the
fixture generator docstring. Verified they load and predict on both
Keras 3.12.4 and 3.14.1.
@roelofsc
roelofsc force-pushed the refactor-configuration branch from 7d4391e to bb08d20 Compare November 9, 2026 10:35
@roelofsc
roelofsc merged commit 9243ec6 into develop Nov 9, 2026
3 checks passed
@roelofsc
roelofsc deleted the refactor-configuration branch November 9, 2026 10:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants