Feat/save persistency

[viktorbeck98]

Task

Closes #140

Description

Adds a full state persistency system to DetectMateLibrary so detectors can checkpoint and resume their internal state across runs.

Core protocol

• dump/load protocol on EventDataStructure, implemented for EventTracker (msgpack), EventDataFrame and ChunkedEventDataFrame (Parquet + msgpack header), and SingleStabilityTracker (to_state/from_state).
• New PersistencyLoadError for load failures.

PersistencySaver (src/detectmatelibrary/utils/persistency/persistency_saver.py)

• fsspec-backed save/load, backend-agnostic (local, in-memory, optional s3/gcs/azure).
• Triggers: background _SaveTimer and events_until_save event-count threshold.
• Idempotent stop() that joins the timer thread; atexit cleanup registered/unregistered correctly.
• Non-serializable event_data_kwargs (e.g. callables like converter_function) are filtered before writing metadata.json.

Config + detector wiring

• New PersistConfig model and persist block on CoreDetectorConfig, integrated into config serialization and the MissingParams warning, and preserved across set_configuration() rebuilds.
• CoreDetector._register_persistency() helper wired into NewValueDetector, NewValueComboDetector, and NewEventDetector.
• EventPersistency gains an ingest callback and events_since_save counter to drive event-count saves.
• Component gains a context-manager protocol and _Stoppable protocol for clean saver shutdown.

Other

• New deps: fsspec, msgpack, pyarrow (with optional s3/gcs/azure extras).
• Docs added under docs/auxiliar/persistency.md, docs/detectors.md, and AGENTS.md (renamed from CLAUDE.md).
• Also folds in from_polars helpers and the TrainBuffer class from the development merge.

How Has This Been Tested?

• New unit tests for dump/load on each structure: tests/test_utils/test_persistency_dump_load.py
• PersistencySaver trigger/save/load tests (timer, events_until_save, idempotent stop): tests/test_utils/test_persistency_saver.py
• PersistConfig config-roundtrip tests: tests/test_common/test_persist_config.py
• End-to-end detector save → reload integration tests across backends: tests/test_detectors/test_persist_integration.py
• from_polars helper tests: tests/test_helper/test_from_to.py
• Full suite green via pytest.

Checklist

• This Pull-Request goes to the development branch.
• I have successfully run prek locally.
• I have added tests to cover my changes.
• I have linked the issue-id to the task-description.
• I have performed a self-review of my own code.

[viktorbeck98]

@ipmach ready for review

[ipmach]

Why there is a protocol, and what is the purpose of this?

[viktorbeck98]

It’s only there to keep the dependency clean. core.py only needs to call .stop() on whatever saver-like object gets attached. So we don't need to make core dependent on smth from the detector persistency

[ipmach]

This file is saturated. It needs some refactor.

All the methods outside the CoreDetector that are not call in the core detector should not be in this file as they are not part of it. I suggest to move this to a persist file.

_register_persistency method should be outside of the class. It takes to much space in the class an does not need to be part of it as it never use the self pointer.

Suggestion:

def init_persistency(config: CoreDetectorConfig, persistency: EventPersistency) -> EventPersistency:
        config = cast(CoreDetectorConfig, self.config)
        if config.persist is None:
            return
        p = config.persist
        saver = PersistencySaver(
            persistency,
            PersistencySaverConfig(
                path=f"{p.path}/{self.name}",
                save_interval_seconds=p.interval_seconds,
                events_until_save=p.events_until_save,
                auto_load=p.auto_load,
                storage_options=p.storage_options,
            ),
        )
        saver.start()
        return saver


class CoreDetector:
            ........
        def _register_persistency(self, persistency: EventPersistency) -> None:
                  self.saver = init_persistency(self.config, persistency=persistency)

[ipmach]

I suggest to treat every persistency method as its own library. (Similar as the schemas)

So, everytime you need a method you do:

import detectmatelibrary.utils.persistency as persistency

persistency.get_configure_variables(.....)

If you follow this syntax then is clear all the methods of each file that is part of the persistency and we have a better separation.

For example in the NewValueDetector:

class NewValueDetector(CoreDetector):
    """Detect new values in log data as anomalies based on learned values."""

    def __init__(
        self,
        name: str = "NewValueDetector",
        config: NewValueDetectorConfig = NewValueDetectorConfig()
    ) -> None:

        if isinstance(config, dict):
            config = NewValueDetectorConfig.from_dict(config, name)

        super().__init__(name=name, buffer_mode=BufferMode.NO_BUF, config=config)
        self.config: NewValueDetectorConfig  # type narrowing for IDE
        self.persistency = persistency.EventPersistency(
            event_data_class=persistency.EventStabilityTracker,
        )
        # auto config checks if individual variables are stable to select combos from
        self.auto_conf_persistency = persistency.EventPersistency(
            event_data_class=persistency.EventStabilityTracker
        )
        self._register_persistency(self.persistency)

So I do not need to look in the imports where each class from persistency comes from. Same as

import numpy as np

np.load()  # I know this load comes from numpy

but if I do:

from numpy import load

load()  # I need to go to the top of the document to know where this comes from

[ipmach]

files that are only use in persistency, should be inside persistency module even if they are generic. Example: RLE_list.py should be inside persistency.

So the dependency lines can be easily draw as:

(DetectMate) <------ (Persistency)

DetectMate dependes on the persistency module, but not in viceversa. It will also help if persistency has a clear API to better split between the two so the module is more modular.

(DetectMate) <---- (Persistency API ) <--- (Persistency)

Same thing as the schemas, where the division is clear:
(DetectMate) <--- (schemas/_classes.py) <--- (schemas methods)

This will make the code more clear and would helps us to later refactor the Detector class or methods to reduce the replicated code

[ipmach]

Lets leave it like this for now, but I think this documentation can be a little hard to follow if you dont know how the persistency works

[ipmach]

The persistency itself looks good, but it should be a more clear separation between the persistency module and the rest of DetectMate.

Otherwise we may run into future problems to maintain the code.

[viktorbeck98]

@ipmach thanks for your comments, much cleaner now. Everything has been addressed :)