Repository navigation
Migrate configuration from Cerberus to Pydantic - #37
Merged
Merged
Conversation
roelofsc
force-pushed
the
refactor-configuration
branch
from
October 12, 2026 12:06
6f1a6b0 to
5155e4b
Compare
roelofsc
force-pushed
the
develop
branch
2 times, most recently
from
October 12, 2026 12:44
a249688 to
500f74d
Compare
cgueck895
approved these changes
Nov 6, 2026
cgueck895
marked this pull request as ready for review
November 6, 2026 11:08
cgueck895
force-pushed
the
refactor-configuration
branch
from
November 9, 2026 06:41
d2ba305 to
0723f92
Compare
Replace the Cerberus-based BaseConfig/Config with a clean Pydantic design: - ConfigModel is the top-level Pydantic model with section models for each config section (train, predict, root_cause_analysis, etc.) - Config is a plain wrapper class holding a ConfigModel instance (self._model) - _SubscriptableModel mixin provides __getitem__/__contains__/get for backward compatibility during migration from dict-style to attribute-style access - Extra validators (_parse_timedelta, _validate_early_stopping, _validate_sequence_fit_on_val) run as a @model_validator(mode='after') - config_dict property rebuilds from the model on every call (model is source of truth, not a cached mutable dict) - Convenience properties now read from self._model instead of config_dict - Delete base_config.py (consolidated into config.py) - Fix configuration.py and care2compare.py consumers to use model attributes instead of .setdefault() on dict-like objects - Update pyproject.toml: cerberus -> pydantic>=2.0
…ute access
Remove _SubscriptableModel mixin (__getitem__/__contains__/get) from all
section models and Config. Section models now inherit BaseModel directly.
Config.__getattr__ delegates top-level model fields (train, predict, etc.)
for natural attribute access (config.train.autoencoder.params).
Remove trivial convenience properties that were thin wrappers over model
fields: protect_conditional_features, fit_threshold_on_val, data_clipping
(bool), max_criticality, root_cause_analysis (bool), dtype. The latter two
resolve via __getattr__ delegation to ConfigModel fields.
Keep properties with real logic: data_preprocessor_steps (handles
params->steps migration + deprecation warnings), arcana_params,
data_split_params, data_clipping_params (all return plain dicts via
_model_to_config_dict for **-unpacking consumers).
Migrate all consumers:
- model_factory.py: config['train'] -> config.train, .get('params', {}) -> .params or {}
- fault_detector.py: config.fit_threshold_on_val -> config.train.threshold_selector.fit_on_val,
config.data_clipping -> config.train.data_clipping is not None, etc.
- configuration.py: config['train']['data_preprocessor'] -> config.train.data_preprocessor,
dp.get('params') -> dp.params, dp['params']['key'] -> dp.params['key']
- care2compare.py: same pattern
- All test files: conf['train']['autoencoder']['params']['x'] -> conf.train.autoencoder.params['x']
…Config fields, remove dead code
- AutoencoderConfig: default name='default', params={} so Config() works with no args
- DataSplitterConfig: replace params dict with flat fields (train_block_size,
val_block_size, validation_split, shuffle) matching the YAML format
- DataPreprocessorConfig: default steps=[] instead of None
- Remove update_config (dead code, not called anywhere)
- Remove dead None checks (train, data_preprocessor no longer Optional)
- Simplify DataClipper usage: explicit kwargs instead of model_dump
- Remove unused DataPreprocessorConfig import from care2compare.py
- Clean up or {} guards in model_factory (params default to {})
- Remove orphaned test_criticality_config.yaml
… access - predist_utils.py: replace config['train']['autoencoder']['params'] with config.train.autoencoder.params and config['train']['data_preprocessor'] with config.train.data_preprocessor.steps - Example - Create new model classes.ipynb: migrate deprecated data_preprocessor.params to steps format
…redict section - configuration.rst: replace update_config example with direct attribute access - configuration.rst: remove obsolete 'Prediction options' (predict section removed) - usage_examples.rst: remove reference to optional predict section - advanced_config.yaml: remove predict/criticality block - sequence_models.rst: fix ts_freq attribution (model factory, not config parser) - Remove orphaned old_config.yaml (deprecated data_preprocessor.params format)
…ware
- Remove from_yaml/from_dict/read_config; the constructor now accepts a
YAML path or an inline dict (Config('file.yaml'), Config({'train': {...}})).
- Drop the _check_dependencies validator from DataSplitterConfig; the splitter
ignores irrelevant fields.
- Make _validate_early_stopping type-aware: check val_block_size for block
splitters, validation_split for sklearn splitters.
- Set DataSplitterConfig default to sklearn with validation_split=0.1 and
shuffle=False so early stopping and fit_on_val work out of the box.
- Revert DataPrepStepConfig to plain dicts (dict-style access in
DataPreprocessor and other consumers).
- Add _format_validation_error helper converting Pydantic ValidationError
into InvalidConfigFile.
- Update README quickstart to Config() and document default split behavior.
- Remove _model_to_config_dict and _dump_value helpers (45 lines); Pydantic's model_dump(exclude_none=True) produces identical output - write_config now uses model_dump instead of config_dict - config_dict property emits DeprecationWarning, delegates to model_dump - optimization.py: use config.train.autoencoder.params (attribute access) instead of config_dict['train']['autoencoder']['params'] - configuration.py: remove dead dp.params legacy branch (already converted to steps during config validation) - Tests: replace config_dict assertions with model_dump(exclude_none=True)
…rt_config for custom preprocessing
- usage_examples.rst: minimal example uses Config() not generate_quickstart_config
- usage_examples.rst: Configuration section shows Config() first, quickstart second
- configuration.rst: fix data_preprocessor: {} -> omitted/null
- configuration.rst: add Config() snippet before generate_quickstart_config
Reverts the sklearn + validation_split=0.1 default from d005daa7 back to BlockDataSplitter with all optional fields None (splitter falls back to 5040/1680 samples). Re-adds update_config/configuration_file compat shims for old-model loading. Updates docstrings, docs, and test expectations. Adds _parse_ts_freq unit test.
Adds saved-model fixtures for py311_efd060 (Keras 2.15) and py312_efd071 (Keras 3.14), the create_legacy_model_fixtures.py script that generated them, and test_model_compatibility.py which loads and predicts with each fixture on synthetic data. Keras 2 fixtures are marked expectedFailure due to keras.src.engine.functional module path change in Keras 3.
…ased alarm function
…ibility The py312_efd071 fixtures were saved with Keras 3.14.1, which adds a quantization_config key to Dense configs. Python 3.10 caps Keras at 3.12.4 (Keras 3.13+ requires >=3.11), so loading these fixtures failed with 'Unrecognized keyword arguments passed to Dense'. Regenerate the fixtures with EFD 0.7.1 on Python 3.10 (Keras 3.12.4), rename them to py310_efd071, and update the compatibility tests and the fixture generator docstring. Verified they load and predict on both Keras 3.12.4 and 3.14.1.
roelofsc
force-pushed
the
refactor-configuration
branch
from
November 9, 2026 10:35
7d4391e to
bb08d20
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR replaces the Cerberus-based
BaseConfig/Configsystem with a PydanticConfig(BaseModel) model. All configuration sections (train,root_cause_analysis,dtype) have become typed Pydantic models with attribute access (config.train.autoencoder.params) instead of dictionary-style access (config['train']['autoencoder']['params']).In addition,
Config()now generates a complete configuration that can be used directly. It is not optimized to the users data yet however.Why? The old system just validated a dictionary. Every programmatic change of the configuration meant using nested
.get()/.setdefault(), without validation of the input and no IDE support (i.e. tab completion). The configuration class is now a typed object with attribute access. When an attribute is assigned a new value, validation runs automatically. Swapping out a component means setting its name and parameterss directly (i.e.config.train.autoencoder.name = 'lstm',config.train.autoencoder.params = {...}), with autocompletion and inline docs. This will help with development of hyperparameter optimisation workflows.Core change
energy_fault_detector/config/config.pyThe central file. Key changes are:
Configis now a PydanticBaseModel. Section models:TrainConfig,AutoencoderConfig,AnomalyScoreConfig,DataPreprocessorConfig,ThresholdSelectorConfig,DataClippingConfig,DataSplitterConfig,RootCauseAnalysisConfig.Config()with no arguments produces a ready-to-use configuration (MultilayerAutoencoder, RMSE score, quantile threshold, default preprocessing, BlockDataSplitter).config_dictdeprecated. It usesmodel_dump(exclude_none=True)under the hood, emitsDeprecationWarning. Prefermodel_dump()or direct attribute access.read_config(). Validation/loading is now done by Pydantic on construction. The constructor also accepts a positional dict (Config({'train': {...}})) in addition to the existingconfig_filename/config_dictkeyword arguments.DataSplitterConfig(fields liketrain_block_sizeare simply ignored by the splitter if not applicable).energy_fault_detector/config/base_config.pyConsolidated into
config.py. The Cerberus schema andBaseConfigclass are gone.energy_fault_detector/config/__init__.pyUpdated exports:
Configfromconfig.py,generate_quickstart_configfromquickstart_config.py.energy_fault_detector/config/quickstart_config.pyMigrated to use
Config(config_dict=...)instead of the oldBaseConfig.update_config()pattern.pyproject.tomlDependency change:
cerberus→pydantic>=2.0.Resulting changes
These files changed because the Config API changed. No logic changes. Just dict-style access → attribute access.
energy_fault_detector/fault_detector.pyself.config.data_clipping→self.config.train.data_clippingself.config.data_clipping_params.copy()→ typed fields fromDataClippingConfigmodelself.config.fit_threshold_on_val→self.config.train.threshold_selector.fit_on_valself.config.arcana_params→self.config.root_cause_analysis.paramsself.config.protect_conditional_features→self.config.train.protect_conditional_featuresenergy_fault_detector/core/model_factory.pyconfig["train"]→config.trainconfig.data_preprocessor_steps→train.data_preprocessor.stepsae_params["name"]→ae_config.name,ae_params.get("params", {})→ae_config.params_parse_ts_freq()helper (renamed from_parse_timedeltaand moved fromconfig.pytomodel_factory.py).energy_fault_detector/core/fault_detection_model.pyconfig.data_split_params→config.train.data_splitter.energy_fault_detector/evaluation/care2compare.pyconfig['train'].setdefault('data_preprocessor', {})→config.train.data_preprocessorconfig.update_config(config.config_dict)→ direct attribute mutation on the Pydantic model.stepsstyle remains).energy_fault_detector/quick_fault_detection/configuration.pyconfig['train']['data_preprocessor'].get('params')→config.train.data_preprocessor.setdefault()/.get()calls replaced with attribute access and list operations ondp.steps.energy_fault_detector/quick_fault_detection/optimization.pyconfig.config_dict→config.model_dump(exclude_none=True).Additional: criticality-based alarm detection (secondary)
This PR also adds an optional, higher-level way to derive alarm events from the predicted anomalies. It is an addition — the existing criticality behaviour is unchanged.
energy_fault_detector/utils/analysis.pycalculate_alarms(...): builds oncalculate_criticalityand returns(alarm_events, criticality).start,end,duration(plusgroupfor MultiIndex data).criticality_thresholdis omitted it defaults to one hour of anomalous behaviour (int(1 hour / time_frequency), or1if the frequency exceeds one hour).max_criticalityis derived fromcriticality_threshold + int(alarm_reset_time_minutes / time_frequency), so alarms reset after normal behaviour.MultiIndex(e.g. per-asset) by detecting the datetime level and computing criticality/alarms per group.energy_fault_detector/core/fault_detection_result.pyFaultDetectionResult.criticality_alarms(...): thin wrapper returning(alarm_events, criticality).FaultDetectionResult.criticalityis unchanged and keeps its original signature (normal_idx,init_criticality,max_criticality) and itspd.Seriesreturn. The new method is purely additive.Docs & tests
docs/criticality.rst(new): explains the counter and alarm-event derivation;docs/configuration.rstanddocs/usage_examples.rstdocumentcriticality_alarms().tests/utils/test_analysis.py(single- and MultiIndex) andtests/core/test_fault_detection_result.py(legacycriticalityand newcriticality_alarms).Note on the criticality config removal
The separate
predictconfig section (config.max_criticality) was removed because it was unused by the criticality code. The criticality functionality itself is not removed — it just continues to be derived from the prediction result, now optionally extended withcriticality_alarms().Tests
tests/test_model_compatibility.pyAdd 4 tests verifying that models saved by older EFD versions still load:
py312_efd071: FullFaultDetector.load()+predict()(Keras 3.14).py311_efd060: Partial non-Keras component loading (Keras 2.15 models can't be loaded by Keras 3, but config/DataPreprocessor/threshold/score load fine and are run through a partial pipeline).tests/test_data/old_models/: Contains generated model artifacts for two EFD versions (0.6.0/py311/Keras 2.15 and 0.7.1/py312/Keras 3.14), dense + sequence variants.tests/test_data/old_models/create_legacy_model_fixtures.py. Must be run from the corresponding environment.Updated existing tests
tests/config/test_config.py: Updated for Pydantic validation, addedtest_empty_config_defaults.tests/config/test_quickstart_config.py: Updated for newConfig()constructor.tests/core/test_model_factory.py: Updated config construction, added_parse_ts_frequnit test.tests/test_fault_detector.py: Updated config access patterns.tests/quick_fault_detection/test_configuration.py: Updated.tests/quick_fault_detection/test_quick_fault_detector.py: Updated.Removed test data
tests/test_data/test_bad_early_stopping_config.yaml(early stopping validation is now tested inline).tests/test_data/test_criticality_config.yaml(unused predict section removed from config).Docs and formatting
Docs
docs/configuration.rst: Updated for Pydantic config,BlockDataSplitterdefault, removedpredictsection andupdate_configreference.docs/usage_examples.rst: Migrated to attribute-style access.docs/conf.py: Removedbase_configfrom apidoc skip list.docs/sequence_models.rst:ts_freqstring→timedelta64conversion now happens at model init time, not config-parse time.docs/examples/old_config.yaml(deleted): Showed oldparams-style preprocessing config.docs/examples/advanced_config.yaml: Removedpredictsection.README.md:Config()instead ofgenerate_quickstart_config().Notebooks
notebooks/Example - Create new model classes.ipynb: Dict-style → attribute access.notebooks/PreDist/predist_utils.py: Dict-style → attribute access.Breaking changes
Deprecated
config.config_dict→ useconfig.model_dump()(ormodel_dump(exclude_none=True)).config.update_config({...})→ mutate typed attributes directly (config.train.autoencoder.name = 'lstm') or construct a newConfig.Removed API
config['train']['autoencoder']['params'](__getitem__/__contains__)config.train.autoencoder.params(typed attribute access)Config.read_config(part)Config({...})class BaseConfig(base_config.py)config.pyas the PydanticConfigmodelconfig.data_clipping/config.data_clipping_paramsconfig.train.data_clipping.enabled/ the typedDataClippingConfigfieldsconfig.fit_threshold_on_valconfig.train.threshold_selector.fit_on_valconfig.protect_conditional_featuresconfig.train.protect_conditional_featuresconfig.data_preprocessor_steps()config.train.data_preprocessor.stepsconfig.arcana_paramsconfig.root_cause_analysis.paramsconfig.data_split_paramsconfig.train.data_splitter(typedDataSplitterConfig)config.max_criticality+ thepredictconfig section