MailHub Delivery Webhooks Design

Goal

Let each user push terminal email delivery results (sent, bounced, failed) to external business systems via signed HTTPS callbacks, with an observable SQLite-backed delivery queue (retry, logs, manual replay). Account-level endpoints are the default; domains may define override endpoints.

Product Decisions

Item	Choice
Primary use case	Notify business systems of delivery outcomes
Events (MVP)	Terminal only: `sent`, `bounced`, `failed`
Scope	Account default + optional domain override
Reliability	In-process worker + SQLite queue, exponential backoff, delivery log, manual replay
Endpoints	Multiple URLs per account and per domain; each subscribes to a subset of events
Stack fit	Node app + SQLite; no Redis / external broker

Non-Goals (MVP)

Intermediate events (queued, deferred)
Per-recipient split payloads beyond fields already on send_events
Workflow builders / conditional routing UI
Guaranteed exactly-once delivery (at-least-once with idempotent delivery keys)
Redis / BullMQ / separate worker service
Webhook for DNS health or admin audit events

Architecture

send_event reaches terminal status
        │
        ▼
 enqueueWebhookDeliveries(sendEvent)
   resolve targets (domain override vs account)
   insert webhook_deliveries (pending), idempotent
        │
        ▼
 webhook worker (interval inside app process)
   claim due rows → HTTP POST signed body
   success | schedule retry | mark dead

Trigger points

Concrete call sites in the current codebase (plan should wire both):

updateSendEventDelivery (Postfix tracker path) — when computed nextStatus is terminal and differs from the previous row status.
logSendEvent (and any wrapper that inserts a terminal status, typically failed on immediate send failure) — when the inserted status is already terminal.

Other mailer/submission/API paths only matter if they write through these functions; do not add parallel enqueue call sites without going through DB helpers.

Enqueue is synchronous DB insert only (must not await outbound HTTP). Worker performs HTTP.

Process model

Worker starts with HTTP server lifecycle (e.g. after listen).
Poll interval: ~5–15s (constant).
Concurrency: small fixed pool (e.g. 2–3 in-flight POSTs) to protect local Node and remote endpoints.
On process restart, pending/retryable rows remain in SQLite and resume via next_attempt_at.

Delivery status machine

webhook_deliveries.status values:

Status	Meaning
`pending`	Not yet successfully delivered; may be due when `next_attempt_at <= now`
`processing`	Claimed by worker for an in-flight HTTP attempt (short-lived lease)
`success`	Last attempt received HTTP 2xx
`dead`	Exhausted max attempts without 2xx

There is no long-lived failed status. Transient HTTP/network failures stay pending with an updated next_attempt_at, attempt_count, error, and response_* fields for the UI (“last attempt failed” is derived from error / response_status while status=pending or dead).

Transitions:

Enqueue → pending, attempt_count=0, next_attempt_at=now.
Worker claims due pending rows → processing (see claim protocol).
Complete HTTP attempt: always increment attempt_count (including 2xx).
HTTP 2xx → success.
HTTP non-2xx / timeout / network error:
- If attempt_count < 8 → pending + backoff next_attempt_at.
- Else → dead.
Manual replay (from success or dead, or pending with errors) → pending, attempt_count=0, next_attempt_at=now, clear error (MVP may clear response preview). Reject or no-op if status is processing (wait for lease expiry) to avoid racing an in-flight POST.

Worker claim protocol

To avoid double POST within one process (and reduce multi-instance races):

In a single SQLite transaction, SELECT up to N rows where status='pending' AND next_attempt_at <= now ordered by next_attempt_at, then UPDATE those ids to status='processing', set last_attempt_at=now, and set next_attempt_at to a lease deadline (e.g. now + 2 minutes) so a crashed worker does not leave rows stuck forever.
Only claimed rows may perform HTTP POST.
On completion, update to success or back to pending/dead as above; clear lease by writing the final status.
Reaper (same worker loop): if status='processing' and next_attempt_at < now (lease expired), reset to pending with next_attempt_at=now so the row can be retried.

MVP assumes a single app instance (current Docker deploy). Claim still required for in-process concurrency.

Resolution Rules

Given userId, domainId, eventType ∈ {sent, bounced, failed}:

Load enabled webhooks for this user where domain_id = domainId and events includes eventType.
If any such domain-scoped webhooks exist → use only those (override account).
Else load enabled account-scoped webhooks (domain_id IS NULL) that subscribe to eventType.
Skip disabled or unsubscribed endpoints.
For each selected webhook, insert a delivery row if not already present for
(webhook_id, send_event_id, event_type).

Data Model

`webhooks`

Column	Type	Notes
id	INTEGER PK
user_id	INTEGER NOT NULL	FK users
domain_id	INTEGER NULL	NULL = account-level; else domain override
name	TEXT NOT NULL	Display label
url	TEXT NOT NULL	HTTPS endpoint
secret_ciphertext	TEXT NOT NULL	Encrypted, same pattern as DNS/SMTP secrets
secret_prefix	TEXT NOT NULL	For UI display
events_json	TEXT NOT NULL	JSON array subset of sent/bounced/failed
enabled	TEXT/INTEGER	boolean
created_at / updated_at	TEXT	ISO

Indexes: (user_id), (user_id, domain_id).

`webhook_deliveries`

Column	Type	Notes
id	INTEGER PK	Public id may be exposed as `whd_{id}`
webhook_id	INTEGER NOT NULL
user_id	INTEGER NOT NULL	Denormalized for isolation queries
send_event_id	INTEGER NOT NULL	Real event id, or `0` for synthetic tests; no FK to send_events
event_type	TEXT NOT NULL	sent \| bounced \| failed
payload_json	TEXT NOT NULL	Exact body bytes basis (JSON text)
status	TEXT NOT NULL	pending \| processing \| success \| dead
attempt_count	INTEGER NOT NULL DEFAULT 0
next_attempt_at	TEXT NOT NULL
last_attempt_at	TEXT
response_status	INTEGER
response_body_preview	TEXT	Truncated (~1–2KB)
error	TEXT
created_at	TEXT

Constraints / indexes:

UNIQUE (webhook_id, send_event_id, event_type) for enqueue idempotency
(status, next_attempt_at) for worker
(user_id, created_at DESC) for UI logs

On webhook delete: CASCADE or mark deliveries orphaned—prefer ON DELETE CASCADE for simplicity.

Retry Policy

Attempt after first failure	Delay (approx.)
1	1 minute
2	5 minutes
3	30 minutes
4	2 hours
5	6 hours
6+	12 hours

Max attempts: 8 (including first try). Then status = dead.
Success: HTTP 2xx. All non-2xx and transport errors use the same backoff path (no special-case 4xx in MVP).
Manual replay: mutate the same delivery row — status=pending, attempt_count=0, next_attempt_at=now, clear error / response preview; do not change payload_json so the signed body stays stable for that delivery id.

HTTP Contract

Request

Method: POST
Headers:
- Content-Type: application/json
- User-Agent: MailHub-Webhook/1.0
- X-MailHub-Signature: t=<unix_seconds>,v1=<hex_hmac>
- X-MailHub-Event: email.sent|email.bounced|email.failed
- X-MailHub-Delivery: whd_<id>
Timeout: 10 seconds
Redirects: do not follow

Body

{
  "id": "whd_42",
  "type": "email.sent",
  "created_at": "2026-07-09T12:00:00.000Z",
  "data": {
    "message_id": "mh-42",
    "send_event_id": 42,
    "queue_id": "A1B2C3D4E5",
    "status": "sent",
    "domain": "example.com",
    "from": "noreply@example.com",
    "to": ["user@example.com"],
    "subject": "Hello",
    "detail": "optional status detail",
    "delivered_at": "2026-07-09T12:00:01.000Z"
  }
}

Internal status	`type`
sent	`email.sent`
bounced	`email.bounced`
failed	`email.failed`

Signature

Payload id construction: In one SQLite transaction, insert the delivery row (placeholder payload_json if needed), read lastInsertRowid, then set final payload_json with "id": "whd_<id>" and stable created_at before commit, so the worker never claims a row without the signed body. Signature always uses that stored payload_json as raw_body.

Signed string: {t}.{raw_body} where raw_body is the exact stored JSON string POSTed.

v1 = hex(HMAC_SHA256(secret, signed_string))

Receivers should reject if |now - t| > 300 seconds (document 5-minute skew).

Secret: generated on create / rotate; returned once in API response; stored encrypted; list UI shows prefix only.

SSRF protections

Default: only https: URLs.
Optional env flag (e.g. WEBHOOK_ALLOW_HTTP_LOCAL=1) for http://127.0.0.1 / localhost in development.
After DNS resolve, block private, loopback, link-local, and metadata IPs (document exceptions for local flag).
Reject non-HTTP(S) schemes.

Admin API

All routes require authenticated session (or existing API auth pattern used by other user-scoped CRUD). Resources always filtered by user_id.

Method	Path	Behavior
GET	`/api/webhooks`	List; query `domainId` optional
POST	`/api/webhooks`	Create; body: name, url, events[], domainId?, enabled; response includes `secret` once
PATCH	`/api/webhooks/:id`	Update name, url, events, enabled, domainId (not secret)
POST	`/api/webhooks/:id/rotate-secret`	New secret, return once
DELETE	`/api/webhooks/:id`	Delete
POST	`/api/webhooks/:id/test`	Enqueue a synthetic test delivery (see below)
GET	`/api/webhook-deliveries`	List with filters: status, webhookId, eventType, limit
POST	`/api/webhook-deliveries/:id/replay`	Replay dead/success/pending (re-queue per replay rules)

Test delivery: Does not require a real send_events row. Use send_event_id = 0 (allowed only for test deliveries; document as sentinel). Payload shape matches production with:

data.test: true
data.message_id: "mh-test"
data.send_event_id: 0
data.status / type from the webhook’s first subscribed event, or sent / email.sent if all three are subscribed
Placeholder from/to/subject/domain from the user’s first domain when available, else fixed examples

Unique key for test rows: still (webhook_id, send_event_id, event_type) — concurrent tests of the same event on the same webhook may hit the unique constraint; API should reuse the existing test delivery row and reset it to pending (replay semantics) instead of failing.

Validation: URL format + SSRF check on create/update; events non-empty subset of allowed three; domainId must belong to user when set.

UI

Global Webhooks page (replace Placeholder)

Table: name, scope (Account / domain name), URL (truncated), events tags, enabled switch, last delivery status snippet
Create / edit drawer
After create / rotate: modal with full secret + copy (warning: shown once)
Actions: test, open deliveries filtered by webhook, delete
Deliveries panel or sub-view: status, event, attempts, HTTP code, error, replay button

Domain detail → Webhooks tab

Same CRUD scoped to current domainId (override endpoints)
Short help text: “If this domain has any enabled endpoints for an event, account-level endpoints for that event are skipped.”

Reuse existing design system: PageHeader, SectionCard, StatusPill, CodeBlock for secret / sample payload docs.

Code Modules

Module	Responsibility
`src/webhook-model.js`	Pure: event map, resolve list, payload build, sign, backoff, URL validation helpers
`src/webhook-dispatcher.js`	Worker loop, fetch, SSRF resolve checks, status updates
`src/db.js`	Schema, CRUD, enqueue helper
`src/server.js`	Routes; call enqueue after terminal status transitions
`src/pages/Webhooks.tsx` (+ small components)	Global UI
Domain detail tab	Wire domain-scoped list

Testing

Unit: resolve rules (domain override vs account), signature stable, backoff schedule, idempotent enqueue key, event type mapping
DB: CRUD isolation by user; unique delivery constraint
Dispatcher: mock fetch—2xx success; 500 retries; timeout; SSRF rejected URL never fetched
Server API: auth boundary; secret not returned on list

Delivery Order

Schema + db CRUD + model pure functions + tests
Enqueue hooks on terminal status
Dispatcher worker + SSRF
REST API
Global Webhooks UI
Domain detail tab
i18n + build gate

Success Criteria

Terminal status change creates at most one delivery row per webhook endpoint
Domain override suppresses account endpoints for that event
Failed POSTs remain pending (or become dead) with backoff and appear in UI
Manual replay works
Worker claim uses processing lease to avoid double POST
Secrets encrypted at rest; shown once on create/rotate
npm test / npm run build pass

Open Follow-ups (out of MVP)

queued / deferred events
Multiple concurrent endpoints without override (always merge account+domain)
Dead-letter alerts / email to owner
Signing key versioning (v2=)
OpenAPI doc page for receivers

2026-07-09-mailhub-webhooks-design.md 14 KB Vēsture Neapstrādāts