Imported from jeremylongshore/tons-of-skills-marketplace (plugins/saas-packs/flexport-pack/skills/flexport-incident-runbook/SKILL.md). Install upstream with npx skills add jeremylongshore/tons-of-skills-marketplace --skill flexport-incident-runbook. Copyright stays with the author (MIT).

Flexport Incident Runbook

Overview

Incident response procedures for Flexport logistics API integration failures. Covers shipment tracking outages, customs data sync failures, webhook delivery loss, and API degradation scenarios. Flexport powers real-time supply chain visibility, so incidents directly impact shipment tracking, booking workflows, and customs compliance reporting. Classify severity immediately using the matrix below, then follow the matching playbook.

Prerequisites

• An incident commander, communication owner, secure evidence location, credential-revocation path, and rollback operator.
• Redaction rules for commercial terms, addresses, customs documents, invoice details, and shipment payloads.

Instructions

1. Assign severity and an opaque incident ID; pause unsafe booking, sync, or notification workflows.
2. Classify the event as availability, authorization, delivery, data-integrity, policy, or suspected exposure.
3. Apply the smallest safe mitigation, verify recovery using a sandbox/read-only probe, and test a safe failure path.
4. Rotate/revoke credentials when exposure is possible and resume queues only after idempotency and destination checks pass.
5. Record the timeline, impact, mitigation, owner, and post-incident follow-up.

Output

Produce a redacted incident receipt with severity, opaque ID, impact, mitigation, recovery verification, rollback/revocation decision, and follow-ups. Sensitive evidence stays in the approved incident store.

Examples

For a simulated webhook outage, pause the staging worker and verify no milestone is replayed twice. Restore one synthetic canary event after health recovers, then resume only when the incident commander records the recovery evidence.

Severity Levels

Diagnostic Steps

# Check API health
curl -s -o /dev/null -w "HTTP %{http_code}\n" \
  -H "Authorization: Bearer $FLEXPORT_API_KEY" \
  -H "Flexport-Version: 2" \
  https://api.flexport.com/shipments?per=1

# Check platform status
curl -s https://status.flexport.com/api/v2/status.json | jq -r '.status.description'

# Check rate limit remaining
curl -s -D - -o /dev/null \
  -H "Authorization: Bearer $FLEXPORT_API_KEY" \
  -H "Flexport-Version: 2" \
  https://api.flexport.com/shipments?per=1 2>/dev/null | grep -i "x-ratelimit"

Incident Playbooks

API Outage

1. Confirm via status.flexport.com and diagnostic script above
2. Enable circuit breaker to serve cached shipment data
3. Notify downstream consumers that tracking data is stale
4. Queue failed requests for replay once API recovers
5. Monitor status page for Flexport resolution updates

Authentication Failure

1. Verify API key is set and not expired: check $FLEXPORT_API_KEY
2. Test with a minimal authenticated request (see diagnostics)
3. If 401: rotate API key in Flexport portal, deploy new key
4. If 403: check API key scopes match required permissions
5. Revoke compromised keys after new key is confirmed working

Data Sync Failure

1. Check webhook endpoint health — is your receiver returning 200?
2. Query /webhooks to verify subscription is active
3. Identify missed events by comparing last processed timestamp
4. Trigger manual sync for affected shipments via /shipments polling
5. Replay missed webhook events using Flexport's retry mechanism

Communication Template

**Incident**: Flexport Integration [Outage/Degradation]
**Status**: [Investigating/Identified/Mitigating/Resolved]
**Started**: YYYY-MM-DD HH:MM UTC
**Impact**: [N shipments affected / tracking data stale since HH:MM]
**Current action**: [Circuit breaker active / manual sync running / key rotation in progress]
**Next update**: HH:MM UTC

Post-Incident

• Document timeline from detection to resolution
• Identify root cause (Flexport outage / key expiry / webhook endpoint failure)
• Calculate impact: affected shipments, stale data duration, missed customs deadlines
• Add monitoring for the specific failure mode that was missed
• Implement or verify circuit breaker covers the failed endpoint
• Replay any missed webhook events and reconcile data

Error Handling

Resources

• Flexport Status
• Flexport API Docs

Next Steps

See flexport-observability for monitoring setup and alerting thresholds.