Testing in CI

The pattern: your app under test sends mail over SMTP as usual, and your test asks the Testlie API for the message it expects, then asserts on its subject, recipients, links or codes.

Setup

Create an inbox for CI (or a disposable inbox per run).
Create an API key on the API keys page. A read-only key is enough for waiting and reading messages; give it an expiry if your policy requires one.
Store the inbox's SMTP username and password, the inbox ID and the API key as CI secrets.
Point the app under test at smtp.testlie.com:587 with those credentials.

Waiting for a message

GET /api/inboxes/:id/messages/wait?to=<address>&subject=<text>&since=<ISO time>&timeout=<seconds>

The request blocks until a matching message is in the inbox, then answers 200 with { "message": { ... } } (the newest match). If nothing matches before the timeout it answers 408.

Parameter	Matches
`to`	An envelope recipient, exactly (case-insensitive). Includes Cc and Bcc recipients.
`subject`	Subject contains this text (case-insensitive).
`since`	Received at or after this ISO 8601 time.
`timeout`	Seconds to wait, 1 to 120. Default 30.

It wakes up the moment mail arrives (no polling interval to tune), so a test waits only as long as the email takes.

GitHub Actions

yaml

- name: Sign-up sends a welcome email
  env:
    SMTP_HOST: smtp.testlie.com
    SMTP_PORT: "587"
    SMTP_USER: ${{ secrets.SANDBOX_SMTP_USER }}
    SMTP_PASS: ${{ secrets.SANDBOX_SMTP_PASS }}
    API_KEY: ${{ secrets.SANDBOX_API_KEY }}
    INBOX_ID: ${{ secrets.SANDBOX_INBOX_ID }}
  run: |
    since=$(date -u +%Y-%m-%dT%H:%M:%SZ)
    email="ci-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}@example.com"
    npm run test:signup -- --email "$email"   # your test; the app sends mail over SMTP
    curl -fsS -H "Authorization: Bearer $API_KEY" \
      "https://testlie.com/api/inboxes/$INBOX_ID/messages/wait?to=$email&subject=Welcome&since=$since&timeout=60" \
      > message.json
    jq -e '.message.subject | test("Welcome")' message.json

curl -f fails the step on the 408 timeout.

Playwright (TypeScript)

import { test, expect } from "@playwright/test";

const api = (path: string) =>
  fetch(`https://testlie.com${path}`, { headers: { Authorization: `Bearer ${process.env.API_KEY}` } });

test("password reset email has a working link", async ({ page }) => {
  const email = `reset-${Date.now()}@example.com`;
  const since = new Date().toISOString();
  await page.goto("/forgot-password");
  await page.getByLabel("Email").fill(email);
  await page.getByRole("button", { name: "Send reset link" }).click();

  const res = await api(`/api/inboxes/${process.env.INBOX_ID}/messages/wait?to=${encodeURIComponent(email)}&since=${since}&timeout=30`);
  expect(res.status).toBe(200);
  const { message } = await res.json();

  const full = await (await api(`/api/messages/${message.id}`)).json();
  const link = full.message.html.match(/href="([^"]*reset[^"]*)"/)[1];
  await page.goto(link);
  await expect(page.getByText("Choose a new password")).toBeVisible();
});

pytest

python

import os, time, requests

API = "https://testlie.com"
H = {"Authorization": f"Bearer {os.environ['API_KEY']}"}

def wait_for(to, subject=None, since=None, timeout=30):
    params = {"to": to, "timeout": timeout}
    if subject: params["subject"] = subject
    if since: params["since"] = since
    r = requests.get(f"{API}/api/inboxes/{os.environ['INBOX_ID']}/messages/wait", headers=H, params=params, timeout=timeout + 5)
    assert r.status_code == 200, f"no email for {to}"
    return requests.get(f"{API}/api/messages/{r.json()['message']['id']}", headers=H).json()["message"]

def test_welcome_email(app_client):
    email = f"user-{time.time_ns()}@example.com"
    since = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
    app_client.post("/signup", data={"email": email})
    msg = wait_for(email, subject="Welcome", since=since)
    assert "Confirm your email" in msg["html"]

Unique recipients

Parallel jobs share an inbox safely when every test sends to its own address. Build the recipient from something unique to the run (ci-${GITHUB_RUN_ID}-${shard}@example.com, a timestamp or a UUID) and pass it as to, plus since set to just before the action. Then a test can never pick up a message from another job or from an earlier attempt.

The recipient's domain does not matter: nothing is delivered, so example.com addresses are fine.

Disposable inboxes

For full isolation, create an inbox per run and let it delete itself:

bash

inbox=$(curl -fsS -X POST -H "Authorization: Bearer $API_KEY" -H "Content-Type: application/json" \
  -d "{\"name\":\"ci-$GITHUB_RUN_ID\",\"expiresInMinutes\":60}" https://testlie.com/api/inboxes)
export SMTP_USER=$(echo "$inbox" | jq -r .inbox.smtp.username)
export SMTP_PASS=$(echo "$inbox" | jq -r .inbox.smtp.password)
export INBOX_ID=$(echo "$inbox" | jq -r .inbox.id)

Creating inboxes needs a key with write access. The inbox and its messages are deleted after expiresInMinutes (up to 7 days), or delete it yourself at the end with DELETE /api/inboxes/$INBOX_ID.

Reading the message

GET /api/messages/:id returns everything you might assert on: subject, from, to, cc, bcc, envelope, html, text, headers (in order), size and attachments (download each with GET /api/messages/:id/attachments/:attachmentId). GET /api/messages/:id/raw returns the original MIME.

To test SSO in the same pipeline, use a mock identity provider with an auto-login user: your app's "Sign in with SSO" button then completes without any login screen.

Testing in CI

#Setup

#Waiting for a message

#GitHub Actions

#Playwright (TypeScript)

#pytest

#Unique recipients

#Disposable inboxes

#Reading the message

#Testing sign-in