# Recovery and Checkpoint Restart (Opt-in)

The recovery layer is **disabled by default**. It lets you detect corrupted training runs and restart from checkpoints automatically using explicit, user-controlled processes.

## Components

- `Tinkex.Recovery.Policy` – configuration (defaults off, conservative backoff, optimizer restore on)
- `Tinkex.Recovery.Monitor` – polls `Rest.get_training_run/2` for `corrupted: true`
- `Tinkex.Recovery.Executor` – bounded worker that restarts runs from checkpoints

Telemetry events: `:detected`, `:started`, `:checkpoint_selected`, `:client_created`, `:completed`, `:failed`, `:exhausted`, `:poll_error` (all under `[:tinkex, :recovery, ...]`).

## Quickstart

```elixir
config = Tinkex.Config.new(api_key: api_key, recovery: %{enabled: true})

policy =
  Tinkex.Recovery.Policy.new(
    enabled: true,
    checkpoint_strategy: :latest,        # or {:specific, "tinker://..."}
    restore_optimizer: true,             # false for weights-only
    poll_interval_ms: 15_000,
    max_attempts: 3,
    backoff_ms: 5_000,
    max_backoff_ms: 60_000,
    on_recovery: fn old_pid, new_pid, cp ->
      Logger.info("Recovered #{cp.tinker_path} -> #{inspect(new_pid)} (old=#{inspect(old_pid)})")
      :ok
    end,
    on_failure: fn run_id, reason ->
      Logger.warning("Recovery failed for #{run_id}: #{inspect(reason)}")
      :ok
    end
  )

{:ok, service} = Tinkex.ServiceClient.start_link(config: config)

{:ok, executor} =
  Tinkex.Recovery.Executor.start_link(
    max_concurrent: 2
  )

{:ok, monitor} =
  Tinkex.Recovery.Monitor.start_link(
    executor: executor,
    policy: policy
  )

:ok = Tinkex.Recovery.Monitor.monitor_run(monitor, "run-123", service, %{training_pid: training_pid})
```

## Checkpoint Strategy

- `:latest` (default): uses `TrainingRun.last_checkpoint`
- `{:specific, path}`: explicit checkpoint path
- `:best`: reserved for future support

If `last_checkpoint` is missing, the executor will fetch the run via REST (requires passing `config`).

## Optimizer Restore

`restore_optimizer: true` uses `ServiceClient.create_training_client_from_state_with_optimizer/3`. Set to `false` for weights-only restarts (fresh optimizer).

## Concurrency and Backoff

- `max_concurrent` (executor option) caps simultaneous recoveries (default: 1).
- `max_attempts`, `backoff_ms`, `max_backoff_ms` control retry behavior; `:exhausted` is emitted when attempts are spent.

## Telemetry and Observability

Attach handlers to `[:tinkex, :recovery, *]` for tracing. `:poll_error` indicates REST polling issues (monitor keeps state).

## Safety Notes

- Nothing runs automatically; you must start monitor/executor and set `policy.enabled: true`.
- Callbacks should be cheap and resilient; errors are swallowed to avoid cascading failures.
- Ensure `config` is available for REST lookups when monitor/executor are used outside the original process.

## Offline Example

Run `mix run examples/recovery_simulated.exs` to see a fully offline flow that seeds a checkpoint, flips a run to `corrupted: true`, and exercises the monitor + executor with stubbed REST/service modules. No network or API key is required.

## Live Example with Injected Corruption

Run `mix run examples/recovery_live_injected.exs` to create a training run, save a real checkpoint, inject a single `corrupted: true` poll response (no server change), and let the monitor + executor restore from that checkpoint. The example then saves a second checkpoint from the recovered client to prove the restart path. Requires `TINKER_API_KEY` (and optional `TINKER_BASE_URL`/`TINKER_BASE_MODEL`).