Skip to main content

Anti-Bootloop Protection, Automatic Rollback & Acknowledgement

A critical threat in any dynamic code delivery system is shipping a patch that crashes on startup, throwing users into an unrecoverable crash loop (bootloop). TkushPatch defends against this hazard using a dual-layer security design:

  1. At the native Rust engine layer (updater/library/src/updater.rs): automatic boot failure detection, immediate local patch revocation, and delivery of a telemetry PatchInstallFailure event.
  2. At the Flutter SDK layer (OtaRollbackTracker in lib/src/security/ota_security_shield.dart): detection of patch version regression (current_patch < last_executed_patch) and execution of the onRollbackDetected callback to purge local caches.

1. Native Anti-Bootloop Mechanism (Rust Engine)​

Rollback Shield Safeguard
  1. Monitored Boot: When the application launches with a new patch, boot attempts are tracked.
  2. Stability Confirmation: Once the UI renders successfully, the patch is marked as healthy.
  3. Automatic Fallback: If an uncaught crash occurs before confirmation, the engine revokes the patch and falls back to the safe base release.

Startup Crash Detection (handle_prior_boot_failure_if_necessary)​

When a patch is booted, the Rust engine records its patch number in state.json as currently_booting_patch with timestamp boot_started_at.

If the mobile app crashes before the Flutter runtime signals completion, the subsequent app launch executes handle_prior_boot_failure_if_necessary() (updater/library/src/updater.rs):

  1. The engine checks that currently_booting_patch is still set.
  2. It executes state.record_boot_failure_for_patch(patch.number): the patch is marked permanently as bad (PatchKnownBad) on this client device so it is never loaded again (UpdateStatus::UpdateIsBadPatch).
  3. It constructs a detailed diagnostic message: crash_recovery: patch <N> failed to boot (detected_at=<ts>,boot_started_at=<ts>,file_ok=<bool>,file_size=<bytes>)
  4. It queues an EventType::PatchInstallFailure (__patch_install_failure__) event which is transmitted to the backend at POST /api/v1/patches/events.

Server-Driven Rollback (rolled_back_patch_numbers)​

In addition to local crash recovery, the server can order fleet revocations through the rolled_back_patch_numbers array returned in POST /api/v1/patches/check. Upon receiving this array, roll_back_patches_if_needed(rolled_back_patches) uninstalls the revoked patches locally.

2. Dart-Side Resilience & Cache Invalidation (OtaRollbackTracker)​

When an app rolls back from Patch #2 to Patch #1 (or baseline Store binary #0), Patch #2 might have written entries to SharedPreferences, Hive, or SQLite using a schema unknown to Patch #1.

The tk_ota_push SDK solves this through OtaRollbackTracker:

final ota = TkOtaPush.init(
  appReleaseVersion: '1.0.0+3',
  // Optional: use FileOtaPatchStateStore to persist last_executed_patch to disk
  rollbackTracker: OtaRollbackTracker(
    store: FileOtaPatchStateStore(File('/path/to/sandbox/ota_patch_state.json')),
  ),
  onRollbackDetected: (OtaRollbackEvent event) async {
    debugPrint(
      'Rollback detected on release ${event.releaseVersion} '
      'at ${event.detectedAt.toIso8601String()}: '
      'Patch #${event.previousPatchNumber} -> Patch #${event.restoredPatchNumber}',
    );
    // Purge potentially incompatible local caches before runApp()
  },
);

Operation of verifyStartupAndHandleRollback()​

  1. On each syncWithRemoteConfig call (Step 0a) or explicit call to verifyStartupAndHandleRollback(), the SDK inspects the running patch (currentPatchNumber() ?? 0).
  2. OtaRollbackTracker.evaluateAndRecord checks this number against last_executed_patch recorded for this releaseVersion in OtaPatchStateStore (InMemoryOtaPatchStateStore by default, or FileOtaPatchStateStore on disk).
  3. It persists last_executed_patch with the current value.
  4. If effectiveCurrent < previous, it returns an OtaRollbackEvent holding:
    • releaseVersion (String)
    • previousPatchNumber (int)
    • restoredPatchNumber (int, 0 if reverted to original Store binary)
    • detectedAt (DateTime in UTC)
  5. The onRollbackDetected callback is awaited before further initialization proceeds.

3. Telemetry Dispatched to Server (POST /api/v1/patches/events)​

The Rust engine reports the following events (declared in updater/library/src/events.rs) to ShorebirdUpdaterController.reportPatchEvent:

Rust Enum (EventType)JSON ID (type)Trigger
PatchDownload__patch_download__Patch binary download has finished.
PatchInstallSuccess__patch_install__Patch booted successfully on subsequent launch.
PatchInstallFailure__patch_install_failure__Patch failed during boot (crash_recovery) or installation.
PatchUpdateFailure__patch_update_failure__Error during update check or staging phase.

Every PatchEvent carries a normalized failure_category (FailureCategory in updater.rs):

  • "network" (retryable)
  • "server" (retryable)
  • "out_of_space" (non-retryable)
  • "permission_denied" (non-retryable)
  • "filesystem" (non-retryable)
  • "corrupt_patch" (non-retryable)
  • "invalid_signature" (non-retryable)
  • "unknown" (retryable cautiously)

4. Rollback Alert Acknowledgement in Support Console (/support)​

When a device submits __patch_install_failure__, its health flag switches to an active Rollback Alert in the Device Fleet & Support dashboard (/support) of the TkushPatch Web Console.

  • A support operator can search for the device (using its UUID client_id or short code #6959347F) and click Acknowledge Rollback Alert.
  • The server records rollback_acknowledged_at on the device entity without mutating or deleting historical audit log events.
  • Device status reverts to healthy as long as no subsequent failure (latestFailureAt > rollbackAcknowledgedAt) occurs.