DEADHAND

← All posts · · 4 min read

I Pointed an AI Pentest Agent at My Own SaaS Before Launch: Here's Everything It Found

I built areyouesoteric.cc and, before opening it to free accounts and paid memberships

Screenshot took on https://areyouesoteric.cc/

I pointed Artex, an AI pentesting tool, at it as a pure outside attacker: no account, no inside knowledge. It scored 7/10 with zero critical and zero high findings. Payments, auth, injection resistance and secrets all held. The 8 findings it did report were medium and below, and they're the classic indie SaaS gaps: unsigned binaries, soft email authentication, missing rate limits and missing security headers. The biggest lesson was that getting the core right first matters more than the checklist, and that the checklist is still an afternoon of work you shouldn't skip.

This was an unauthenticated, external assessment of my own app. It's a point-in-time snapshot, not a certification.

The scope and the method

Artex worked the way a real outside attacker would: recon the infrastructure, map the APIs, throw injection payloads, probe the payment and auth logic, check email and DNS, and inspect the desktop app I ship.

  • In scope: the public web app, the APIs (portal, license verification, checkout), the edge and TLS (Cloudflare), email + DNS, and the desktop installer.

  • Rules: no account, no inside info.

  • Method: OWASP WSTG, the OWASP API Security Top 10 and the standard PTES phases (recon → scan → exploit → report).

What held

The areas that cause the breaches you read about were tested and held up:

Area Score What held
Payments & auth 8.5 Paid access is validated server-side, the Stripe webhook signature is verified, and there's no payment or auth bypass and no IDOR (you can't swap an ID in a request and see someone else's data).
Injection resistance 8.5 21+ crafted payloads across checkout, the portal and license verification (SQL, NoSQL, prototype pollution, mass-assignment, type-confusion) produced zero injection.
Infrastructure & TLS 9.0 The real origin server isn't discoverable, TLS is locked to modern versions and strong ciphers, and the WAF blocks the usual sensitive paths.
Secrets 9.0 No API keys or secrets leaked in the web app or the installer. .git, .env and source maps all return 404.

The money and the data were solid, which is the part that counts most.

What it found

None of the 8 findings are emergencies, but a few are mistakes almost every indie SaaS makes. Grouped by fix:

Finding Why it matters Fix
Unsigned installer and executable If file hosting were ever tampered with, users would download a malicious build with nothing to catch it. Unsigned installers also trigger more antivirus and SmartScreen warnings. Code-sign both, enforce signing in the release pipeline so an unsigned build can't ship, and publish a SHA-256 checksum for every release.
DMARC p=none and SPF soft-fail Mail forged from my own domain would still get delivered. A phishing gift for a brand that will email members about billing. Move DMARC to quarantine, then reject. Tighten SPF, confirm DKIM on every sending service, add MTA-STS.
Founders seats reservable repeatedly by unauthenticated users Availability could be temporarily exhausted. It self-heals, so it's an annoyance, but it's an abuse path. One active reservation per verified identity and short reservation timeouts.
No rate limiting on a couple of API endpoints Opens the door to brute-force and automated abuse. Per-IP plus per-identity limits with backoff. Cloudflare rules are the fastest path.
Missing HTTP security headers Pages could be framed for clickjacking, with weaker defense-in-depth (HSTS, CSP, X-Frame-Options and friends). Add the standard headers: report-only first, then enforce.
No CAA record or DNSSEC Weaker DNS and certificate-issuance hygiene. Add a CAA record and enable DNSSEC.
Hardcoded default value in the desktop app A baked-in default that shouldn't ship. Strip it from the app.

Related: I'm also registering obvious typo and look-alike domains defensively and pointing them at the real one, so nobody else can grab them for brand-accurate phishing.

None of the header, DNS or default-value items are exploitable on their own. They're the difference between "fine" and "tight."

The order I'm fixing them in

  1. Sign the binaries, since every user of the desktop app touches this.
  2. Fix email auth and register the look-alike domains.
  3. Rate limits and reservation gating.
  4. Headers, then CAA and DNSSEC.

That should move the external posture to roughly 9/10.

💡 Get the core right first: payments, auth, injection and secrets. That's where the real breaches live, and it's the hardest to bolt on later. Signing, email auth, headers and rate limits are real but mostly checklist work.

A few limits:

  • Point-in-time: one assessment on one day. New code means new risk.
  • Unauthenticated only: this tested what a stranger can reach. It says less about what a logged-in user could abuse.
  • External only: nothing here covers my own machines, accounts or internal tooling.
  • An AI agent isn't a certification: it's a fast, cheap way to find the obvious things before launch, not a substitute for a formal audit if you handle regulated data.

Try it yourself

If you're about to launch something, this checklist covers most of what Artex flagged:

  • Sign your installers and publish checksums
  • DMARC at reject, SPF strict, DKIM on every sender
  • Rate limits on every unauthenticated endpoint
  • One reservation or hold per verified identity, with timeouts
  • HSTS, CSP and X-Frame-Options, deployed report-only first
  • CAA record and DNSSEC
  • No hardcoded defaults baked into shipped apps

I'll post a follow-up once it's all closed out, including what it's actually like to get a code-signing certificate and wrestle DMARC into reject. Have you pentested your own app before launch? Leave a comment with what it caught.

Sources

Originally published on Hashnode. Comments and reactions live there.