Skip to main content

Rollback, Kill-Switch, and Acknowledgment

TkushPatch features an end-to-end Rollback and Kill-Switch pipeline combining server-side revocation (manual or via automatic circuit breaker), native Rust engine rollback on the device, the tk_ota_push SDK local cleanup hook, and incident acknowledgment (acknowledgeRollback) for L3 Support and SRE teams.

Rollback Lifecycle Overview​

Rollback Trigger Workflow
  1. Anomaly Detection: Critical defect discovered on an active production patch.
  2. Kill-Switch Execution: Operator runs tk-ota-push rollback or clicks Rollback in the Web Console.
  3. Server Revocation: Patch status changes to ROLLED_BACK and rollout percentage is set to 0%.
  4. Device Rollback: On next check, mobile devices uninstall the revoked patch and restore the base release.

1. Triggering a Patch Kill-Switch (ROLLED_BACK)​

Revoking a patch takes a single click in the Web Console or a single command against the Admin API (OtaAdminController.rollbackPatch):

curl -X POST "$OTA_SERVER_URL/api/admin/v1/patches/$PATCH_ID/rollback" \
  -H "Authorization: Bearer $OTA_API_KEY"

Immediate Server-Side Effects (TkushPatch Server)​

  1. The patch status transitions to ROLLED_BACK and its rolloutPercentage is forced to 0.
  2. The L1/L2 cache for the Release is immediately invalidated (evictAllForRelease).
  3. On subsequent calls to /api/v1/patches/check, the patch number is included in rolled_back_patch_numbers returned to the native Rust engine, which immediately uninstalls the revoked patch.
  4. An immutable PATCH_ROLLBACK_TRIGGERED audit log entry is recorded on the server.

Automatic Circuit Breaker​

In TkushPatch Server, the automatic circuit breaker is enabled by default with:

  • circuitBreakerMinSampleSize = 10 (at least 10 telemetry events received for the patch)
  • circuitBreakerFailureRateThreshold = 0.05 (5% failure rate threshold)

When this threshold is exceeded, the server automatically revokes the patch without waiting for human intervention.

2. Mobile App Resilience (onRollbackDetected)​

When a device rolls back (for example from Patch #2 back to Patch #1 or to the original Store binary null), a local storage schema change (SQLite, Hive, SharedPreferences) introduced in Patch #2 could cause errors in the restored older version.

The verifyStartupAndHandleRollback() method in TkOtaPush (automatically invoked at the start of syncWithRemoteConfig) compares the currently running patch against the last patch recorded by OtaRollbackTracker (current_patch < last_executed_patch):

final ota = TkOtaPush.init(
  appReleaseVersion: '1.0.0+3',
  onRollbackDetected: (OtaRollbackEvent event) async {
    debugPrint(
      'Rollback detected: #${event.previousPatchNumber} -> #${event.restoredPatchNumber}',
    );
    // Purge local cache to guarantee post-rollback stability
  },
);

3. Per-Device Rollback Acknowledgment (acknowledgeRollback)​

In the device fleet inventory (GET /api/admin/v1/devices and the /support Web Console page), any phone that has reported at least one failure or rollback event (failureCount > 0) is flagged with rollbackTriggered: true (Active Rollback).

Once the incident is resolved (for example, after shipping a corrective patch), operators can acknowledge the alert without altering or deleting the immutable telemetry history.

Exact Evaluation Rule (OtaAdminController.listDiscoveredDevices)​

The server stores the rollbackAcknowledgedAt timestamp in ota_device_registry and evaluates:

ackedAfterLastFailure = (rollbackAcknowledgedAt != null)
                        && (latestFailureAt != null)
                        && !rollbackAcknowledgedAt.isBefore(latestFailureAt)

rollbackTriggered     = (failureCount > 0) && !ackedAfterLastFailure
  • As long as no new failure occurs after rollbackAcknowledgedAt, the device is counted as Healthy (rollbackTriggered: false, while rawFailureDetected remains true for auditability).
  • If a new failure occurs on a future patch (latestFailureAt > rollbackAcknowledgedAt), the rollbackTriggered: true alert automatically turns back on.

Acknowledgment and Reset Commands​

You can target a device by its full UUID (client_id) or look it up by its 8-character L3 Short Support Code (#6959347F, retrieved via ota.readShortSupportCode()):

# 1. Acknowledge a device's Rollback alert
curl -X POST "$OTA_SERVER_URL/api/admin/v1/devices/$CLIENT_ID" \
  -H "Authorization: Bearer $OTA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"acknowledgeRollback": true}'

# 2. Reset the acknowledgment (set rollbackAcknowledgedAt back to null)
curl -X POST "$OTA_SERVER_URL/api/admin/v1/devices/$CLIENT_ID" \
  -H "Authorization: Bearer $OTA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"resetRollbackAck": true}'