Surgical Hardening: Closing Gaps Without Breaking Production

The most common reason hardening recommendations go unimplemented is not cost or prioritization -- it is fear of breaking production. WAF rule changes can cause false positives. TLS configuration changes can break older clients. Connection timeout changes can affect legitimate long-running requests. These are real risks, and the fear of them is rational. Surgical hardening is the doctrine that addresses this fear: change one thing at a time, monitor for impact, and roll back immediately if something breaks.

DDactic's hardening recommendations are structured to enable surgical application. Each recommendation specifies the change, the expected impact on OPI score, the traffic classes that could be affected, and the rollback procedure. This structure enables an engineer to implement one recommendation, monitor for 24-48 hours, confirm no false positives, and then proceed to the next -- rather than applying ten changes simultaneously and debugging which one caused the problem.

The Log-Before-Block Principle

For any new WAF rule, begin in log-only mode (Cloudflare calls this "Log" action, AWS WAF calls it "Count" mode). Run the rule in log mode for 48-72 hours and review what traffic it would have blocked. If the log shows legitimate traffic that would have been blocked (for example, a monitoring system polling an endpoint faster than the rate limit threshold), adjust the threshold or add an exception for the monitoring IP range before switching the rule to block mode. This approach eliminates the most common source of false positives from rate limit rules: legitimate automated traffic that exceeds the configured threshold.

# Cloudflare: start new rate limit rule in Log mode
# Action: Log (not Block)
# Monitor: Security > WAF > Events, filter by rule name
# After 48h review: if no legitimate traffic flagged, switch to Block

# AWS WAF: start in Count mode
aws wafv2 update-rule-group --name "ApiRateLimits" \
  --rules '[{"Action": {"Count": {}}, ...}]'  # use Count, not Block
# After monitoring period:
aws wafv2 update-rule-group --name "ApiRateLimits" \
  --rules '[{"Action": {"Block": {"CustomResponse": {"ResponseCode": 429}}}, ...}]'

Change Isolation

Apply changes one at a time, with a 24-48 hour observation window between changes. A single change affects a small number of traffic patterns, making it straightforward to attribute any anomaly to the change. Multiple simultaneous changes create attribution ambiguity: if traffic suddenly drops 15% after applying five WAF rules, you have to test each rule in isolation to identify the cause. The surgical approach is slower but produces fewer production incidents and faster diagnosis when something goes wrong.

Traffic Baselining Before Changes

Before applying any hardening change, capture a traffic baseline: the normal request rate, error rate, and response time percentiles for the affected endpoint over a 7-day period. This baseline is the reference for anomaly detection after the change is applied. A WAF rule change on a Monday that is followed by a 5% increase in error rate on Wednesday is a production incident if the baseline shows no such pattern -- and it is irrelevant if the baseline shows a recurring Wednesday spike that predates the change.

Rollback Procedures Must Be Pre-Written

Every hardening change should have a rollback procedure written before the change is applied. For WAF rules, the rollback is straightforward: disable the rule or switch it to log-only mode. For TLS configuration changes (disabling renegotiation, changing cipher suites), the rollback requires reverting the server configuration and reloading the service -- which should be a documented runbook step, not something the engineer improvises under pressure. For connection timeout changes in Nginx, the rollback requires reverting the nginx.conf change and running nginx -s reload. Writing these procedures before the change ensures they are correct and executable, not reconstructed from memory during an incident.