4 minute read Security

You Can’t Respond to What You Don’t See

In security, knowing something happened is often more important than preventing it entirely. This is where many systems fall flat — they log too little, log the wrong things, or don’t monitor logs at all.

Today, we’re talking about logging and monitoring failures: the cracks attackers slip through unnoticed, and the reason many companies take months to discover breaches.

What Are Logging and Monitoring Failures?

This category is about visibility and detection:

  • Are security-relevant events being logged?
  • Are logs monitored or alerting anyone?
  • Is there enough context in the logs to investigate?
  • Are logs protected from tampering?
  • Can attackers cover their tracks?

Security incidents don’t always start with a red flag — they often look like failed logins, unusual requests, or background noise.

Without good logs, you can’t detect, investigate, or respond to threats.

What Does It Look Like in Practice?

Missing Logs

APIs and backend services often don’t log 4xx/5xx errors, failed auth attempts, or unusual access patterns.

No Centralised Logging

Logs spread across containers, servers, or regions with no aggregation — making incident response a nightmare.

No Alerting

Critical anomalies like a dozen failed login attempts or a password reset flood go unnoticed because no one’s watching.

Logs Can Be Tampered With

Logs stored as plaintext, writable by the same app user, or not immutable — letting attackers cover their tracks.

How Attackers Exploit This

Let’s say an attacker:

  1. Probes your login endpoint to find usernames.
  2. Brute-forces a few passwords slowly, over time.
  3. Logs in, exfiltrates data, and covers their tracks.

If:

  • You don’t log failed logins…
  • You don’t alert on unusual login volume…
  • You don’t track session creation or IP changes…

…you’ll have no idea it happened. Until it’s too late.

Breaches often go undiscovered for months because there were no breadcrumbs to follow.

How Engineers Can Defend Against This

  1. Log Security-Relevant Events

At a minimum, you should log:

  • Failed/successful login attempts
  • Access to sensitive endpoints (e.g. admin, finance)
  • Permission changes or role escalations
  • Data exports or downloads
  • Auth token/session creation and revocation

Include:

  • Timestamp
  • IP/user agent
  • User ID (if known)
  • HTTP status code or result
  1. Centralise and Correlate Logs

Use a centralised logging stack (e.g. ELK, Loki/Grafana, Splunk) so you can:

  • Query logs across services
  • Correlate events between microservices
  • Investigate incidents quickly
  1. Protect Logs from Tampering
    • Write logs to an external system or append-only storage
    • Use root-only permissions or write-only log users
    • Consider hashing log entries for integrity
  2. Implement Alerting and Monitoring

Set up alerts for:

  • Repeated login failures
  • Abnormal user behavior (IP, time, frequency)
  • Elevated permissions or role changes
  • New devices or access locations

Use anomaly detection and dashboards to visualise threats in real time.

  1. Test Detection

Simulate attacks (e.g. login floods, privilege abuse) to verify your system logs, alerts, and escalates appropriately.

Great call — a simple, actionable table can go a long way in helping teams build better logging practices. Here’s a “What to Log (and Why)” table tailored for developers and engineers:

What to Log (and Why)

Event Type What to Log Why It Matters
🔐 Login (success & failure) Username/email, timestamp, IP, user agent, status (success/fail) Detect brute force, credential stuffing, and suspicious access
🔑 Password resets User ID, IP, method (email/SMS), success/failure, timestamp Track misuse of account recovery
🧪 Auth token/session events Token issued, revoked, expired; IP, user agent, user ID Monitor session hijacking and misuse
🎭 Role/permission changes Who made the change, what changed, when, target user Spot privilege escalation and unauthorised admin activity
📥 Data access/download Endpoint, user ID, IP, file/data accessed, timestamp Flag excessive data export or scraping
⚠️ Access denied/403 errors User ID (if any), attempted resource, IP, timestamp Detect probing for unauthorised access
📦 API errors (4xx/5xx) Endpoint, user ID, request params (censored), response code, timestamp Catch broken logic, fuzzing attempts, and misuse
🌍 New device/IP login Geolocation info, device fingerprint, user ID, timestamp Alert on logins from new regions or devices
🔄 Configuration changes Who changed what, old value → new value, when Critical for post-incident investigations
🛠️ Admin actions Who did what, affected resources/users, timestamp Visibility into high-risk changes

Tips for All Log Events:

  • Always include timestamp, user identifier, IP address, and action performed
  • Avoid logging sensitive data (passwords, auth tokens, raw credit card numbers)
  • Ensure logs are structured (JSON or key-value) for parsing and searching
  • Keep logs immutable and centralised

Real-World Case: The Silent Breach

In a well-known breach, attackers accessed cloud storage using compromised credentials. They operated for 8 months before anyone noticed — because:

  • There were no alerts for large data transfers
  • Logs rotated every 24h and were never reviewed
  • No one was watching for logins from new countries

The attack wasn’t sophisticated. The lack of monitoring was.

Final Thoughts

Logging isn’t just a dev tool. It’s your black box recorder. And monitoring isn’t optional — it’s your early warning system.

If something bad happens and no one logs it… did it even happen? (Yes. And it probably cost you a lot of money.)

Next up: A10: Server-Side Request Forgery (SSRF) → We’ll look at how attackers use your app to make internal network requests — and what to do about it.

Leave a comment