Kent Nguyen's Lab #indatawetrust
← All posts

Our servers were attacked on Vietnamese New Year 2026 by a Vietnamese Botnet - 18th Feb 2026.

Our servers were attacked on Vietnamese New Year 2026 by a Vietnamese Botnet - 18th Feb 2026.

On the morning of February 18, 2026, the second day of the week-long Tết (Lunar New Year) holiday in Vietnam, our Samoa eCommerce operations team reported urgent payment failures. Customers could not complete checkout, and transactions had stopped processing.

The investigation

While the broader team was offline for the holidays, as the lead technical architect I connected to our production servers from a hotel balcony and ran Claude Code directly in the terminal to inspect system logs and diagnose the failure.

Server Log Analysis

The logs revealed that an automated botnet scanner had discovered an open port on February 16 and attempted to attach the instance as a replica. While the remote connection failed, the unauthorized attempt triggered internal safety mechanisms that placed the database into read-only mode.

Because our payment gateway rate-limiters write incoming request counters directly to this cache database, every subsequent checkout API call returned a write error, effectively stalling transactions.

The attack vector was a variant of the known 'i love u' Redis replication exploit script. Using terminal-based AI assistance, I quickly identified the modified keys, purged the exploit artifacts, and restored the instance to normal read-write operations.

Removing Traces

The root cause

The automated script was only a symptom. The underlying issue originated weeks earlier during a pre-holiday deployment rush: a configuration update for a new payment gateway had omitted strict security group firewall rules, leaving the Redis port accessible to the public internet.

Key incident response takeaways

Automated scanning scripts continuously target holiday periods when response times are typically delayed.

Core operational practices to prevent similar incidents:

  • Strict firewall rules: verify all security groups and binding addresses before holiday freeze windows.
  • Isolate internal caching layers: ensure Redis and intermediate state stores are never directly routable over public IP interfaces.
  • Terminal-based AI for incident triage: using AI directly in the command line accelerates log parsing, regex searching, and artifact remediation during off-hours emergencies.