Resilience Program Design
Overview
Planning-level guidance for an organization's resilience program: how to structure policies by tier, and how often to run resilience activities.
Structuring resilience policies across an organization
Recommend a tiered policy model (not one policy per service): classify services by business criticality and set policy targets accordingly.
- Availability SLO — higher as criticality rises. The API accepts only a fixed set of SLO
values and rejects out-of-set ones, so confirm the valid values from the API/docs (e.g. aws resiliencehubv2 create-policy help or the Resilience Hub documentation) rather than relying on a hardcoded list — illustratively, values such as 99.9/99.95/99.99.
- RTO/RPO — tighten as criticality rises (single-digit minutes for critical, hours for low).
- DR approach — match to criticality (more aggressive for more critical), using a value from
the API's DR-approach enum — verify the valid set via the API/docs (e.g. aws resiliencehubv2 create-policy help); illustratively ACTIVE_ACTIVE … BACKUP_AND_RESTORE.
Example (illustrative — resolve the actual enum values against the API before recommending): payments/auth → 99.99 + single-digit-minute RTO + ACTIVE_ACTIVE; internal tools → 99.9 + tens-of-minutes RTO + WARM_STANDBY; dev/test → 99.9 + multi-hour RTO + BACKUP_AND_RESTORE.
Warn against contradictory policies (e.g. the maximum SLO 99.99 with BACKUP_AND_RESTORE, or multi-region RTO shorter than multi-AZ RTO).
How often to run resilience activities (cadence)
Recommend this minimum cadence when asked how often to run resilience activities:
- Continuous: ARC zonal autoshift practice runs (automated)
- Weekly: review the Resilience Hub findings dashboard
- Monthly: run FIS experiments (single-service fault-injection tests)
- Quarterly: cross-service GameDay
- Event-driven: after every production incident and before/after major deployments
Security Considerations
Program-level guidance — bake security into the standards you set:
- Standardize least privilege: require every resilience role (Resilience Hub invoker, FIS execution, ARC operator) in your templates and policies to be least-privilege and resource-scoped, with
aws:SourceArn/aws:SourceAccountcondition keys on their trust policies to prevent confused-deputy access. - Mandate short-lived credentials: require all resilience automation to authenticate as IAM roles with short-lived credentials (role assumption, AWS SSO, instance profiles) — never IAM users with long-lived access keys — as a program standard, since these roles perform privileged and potentially destructive operations.
- Mandate encryption: make SSE-KMS on report/state buckets part of your tier baseline, and enforce encryption in transit (TLS) — e.g. an
aws:SecureTransportdeny-if-false condition on those bucket policies and HTTPS-only API access. - Govern FIS in production: define an authorization / change-management gate for production fault injection as part of the program cadence.
- Limit exposure of resilience outputs: assessment findings, FIS logs, and GameDay reports can contain sensitive architectural detail (resource ARNs, IPs, failure modes) — make restricting their access to authorized personnel part of your program standards.
- Further reading: point teams to the AWS Well-Architected Security Pillar, FIS Security Best Practices, and IAM Best Practices for implementing these standards.
