Spam reports & failure scenarios

Two tools for the parts of email, SMS and SSO that usually only break in production: whether your mail lands in the inbox, and how your app behaves when a provider is down.

#Spam score and inbox placement

Open a message and choose the Spam & placement tab. The report has four parts:

• Score and placement estimate. The spam score against its threshold (lower is better) and an estimate: Inbox likely, At risk (spam or promotions folder) or Spam folder likely, with the reasons. It is an estimate from the message itself; real placement also depends on your domain and IP reputation.
• SpamAssassin. When the server runs SpamAssassin, its score and every rule that fired, with points, description and a plain-English fix for the common ones.
• Testlie heuristics. Built-in rules that work without SpamAssassin, always shown and labelled separately: missing or invalid Message-ID and Date, ALL CAPS subject, HTML without a text part, image-heavy mail, URL shorteners, links to IP addresses or with misleading text, scripts and forms, missing List-Unsubscribe or one-click unsubscribe on bulk mail, missing DKIM, Gmail clipping size. Once the authentication check has run, SPF, DKIM and DMARC failures are added as rules too.
• Gmail and Yahoo sender requirements. A checklist of the 2024 bulk-sender rules: SPF or DKIM, both for bulk senders, a DMARC record and alignment, one-click unsubscribe (RFC 8058) with the headers covered by DKIM, a well-formed message, not sending as @gmail.com, TLS, PTR records, a spam rate under 0.3% and honouring unsubscribes within two days. Items Testlie cannot see from a captured message (spam rate, PTR) are marked Your side with what to check.

#One-click unsubscribe

Marketing and other bulk mail needs both headers, and the URL must unsubscribe on a POST without a login or confirmation page:

List-Unsubscribe: <https://example.com/u/TOKEN>, <mailto:unsubscribe@example.com>
List-Unsubscribe-Post: List-Unsubscribe=One-Click

Point the URL at a request bin in your test environment to check that your app sends it and that the endpoint answers the POST.

#Score every message automatically

Tick Score every new message in this inbox automatically in the report. New mail is scored by the worker, and message.received webhooks get a message.spam field:

{ "event": "message.received",
  "message": { "id": "…", "subject": "…",
    "spam": { "score": 2.1, "threshold": 5, "verdict": "ok", "engine": "builtin", "placement": "inbox", "heuristicsScore": 2.1 } } }

#From tests

curl -H "Authorization: Bearer $TESTLIE_API_KEY" "https://testlie.com/api/messages/$MESSAGE_ID/spam"

The answer is { "report", "cached" }; ?refresh=1 analyses the message again. While report.auth.status is running the authentication check is still running in the background: ask again a moment later for the complete checklist. From a Node.js test:

const { report } = await (await fetch(`${TESTLIE}/api/messages/${id}/spam`, { headers })).json();
expect(report.verdict).not.toBe("spam");
expect(report.checklist.find((c) => c.id === "one_click_unsubscribe").status).toBe("pass");

#Failure scenarios

Failure scenarios (in the sidebar) are one-click presets that make the sandboxes fail together, so you can test retries, timeouts and error pages without changing any inbox, IdP or bin settings. Turning a scenario off restores normal behaviour within a few seconds. While one is active, a banner across the app says so and has a Turn all off button.

Each preset has parameters (rates, delays, error codes) shown on its card. A scenario applies to the whole team or to one project (the inboxes, SMS inboxes and bins in it; mock IdPs belong to the team, so IdP presets need a team-wide activation). It switches itself off after the chosen time (one hour by default). Activations and deactivations are recorded in the audit log.

#From CI

Owners, admins and write API keys can switch scenarios on and off:

curl -X POST "https://testlie.com/api/scenarios/email-provider-outage/activate" \
  -H "Authorization: Bearer $TESTLIE_API_KEY" -H "Content-Type: application/json" \
  -d '{"params":{"mode":"tempfail","failCount":2},"durationMinutes":15}'
# run the tests that expect your app to retry
curl -X POST "https://testlie.com/api/scenarios/email-provider-outage/deactivate" -H "Authorization: Bearer $TESTLIE_API_KEY"

From a test, turn a scenario on, run the test and always turn it off again:

const api = (path, body) =>
  fetch(`${TESTLIE_URL}/api/scenarios/${path}`, {
    method: "POST",
    headers: { Authorization: `Bearer ${process.env.TESTLIE_API_KEY}`, "Content-Type": "application/json" },
    body: JSON.stringify(body ?? {}),
  });

await api("webhook-receiver-down/activate", { params: { mode: "503" }, durationMinutes: 15 });
try {
  await placeOrder();
  await expectRetryScheduled();
} finally {
  await api("webhook-receiver-down/deactivate");
}