Async processing in Sophia

Sophia offloads work that is too slow, too fragile, or too expensive to run inside an HTTP request. The pattern is Amazon SQS queues consumed by AWS Lambda functions in apps/functions, with business logic implemented in apps/api services so the same code can run in Fargate, Vitest, or Lambda without an SQS event.

Infrastructure is defined in infra/queues.ts and wired from sst.config.ts. Deploy outputs expose queue URLs for the API and operators:

Output	Source
interviewCompletedQueue	sst.config.ts → interviewCompletedQueue.url
interviewCompletedDlq	sst.config.ts → interviewCompletedDlq.url
reportGenerateQueue	sst.config.ts → reportGenerateQueue.url
reportGenerateDlq	sst.config.ts → reportGenerateDlq.url

The Fargate API receives INTERVIEW_COMPLETED_QUEUE_URL and REPORT_GENERATE_QUEUE_URL via infra/api.ts.

Why SQS + Lambda (not inline)

1. Duration — Credit settlement touches MongoDB transactions, ledger entries, and lot allocation. Report generation calls OpenAI with structured output and can run up to five minutes (infra/queues.ts). Neither belongs on the bot’s complete/fail HTTP path.
2. Survival across deploys — Messages stay in the queue if a consumer is redeployed mid-flight; the work is retried when Lambda comes back.
3. Isolation of failure — A poison message or LLM outage must not block interview completion in the API. The API updates interview status synchronously, then enqueues settlement and (after settlement) report generation.
4. Operational safety — Every queue has a dead-letter queue (DLQ) with 14-day retention so exhausted retries are visible and replayable, not silently dropped (infra/queues.ts, .cursor/rules/async-jobs.mdc).

Comments in infra/queues.ts and interview-completed.ts refer historically to “transcription, scoring, and settlement.” As implemented today, the interview-completed consumer runs credit settlement only; the bot writes proctoring/transcript data via the API before publishing completion events (apps/api/src/modules/interviews/bot-interview.service.ts).

Architecture at a glance

Bot/API (Fargate)
  completeInterview / failInterview
       → InterviewCompleted queue
            → Lambda: interview-completed
                 → settleCompletedInterview / settleFailedInterview
                      → (on first successful completion settle) ReportGenerate queue
                           → Lambda: report-generate
                                → generateReport (OpenAI)

See queues.md for per-queue detail and the architecture diagram for a visual overview.

Consumer wrapper: createSqsConsumer

All Lambda handlers use createSqsConsumer from apps/functions/src/platform/sqs-consumer.ts. Do not implement raw SQS handlers (see .cursor/rules/async-jobs.mdc).

Situation	Behaviour
Message handled successfully	Handler returns { batchItemFailures: [] }; SQS deletes the message
One message throws in handle	That message’s id is added to batchItemFailures → SQS redrives only that message
JSON body fails Zod validation	Logged and dropped (not in batchItemFailures; retry would never succeed)
connectMongo() throws	Entire invocation throws → whole batch retried

Infrastructure must set partialResponses: true on the event source mapping so AWS honors batchItemFailures (infra/queues.ts, infra.mdc).

Correlation: producers set a requestId message attribute; the consumer restores it via runWithRequestContext (sqs-consumer.ts, sqs.service.ts).

Idempotency

SQS is at-least-once. Duplicate delivery must not double-charge credits or double-call the LLM.

Interview settlement (apps/api/src/modules/credits/settlement.service.ts):

• Claims work with InterviewModel.findOneAndUpdate({ settledAt: null }, { $set: { settledAt, credits.consumed, retentionExpiresAt } }).
• If the claim fails, processing returns without side effects.
• settleCreditsInSession runs only after a successful claim inside the same transaction.
• Report enqueue runs only when settled === true after completion settlement, so redeliveries do not republish report.generate.

Report generation (apps/api/src/modules/reports/report-generation.service.ts):

• Skips if report status is already COMPLETED.
• Atomically claims NOT_STARTED or FAILED → PROCESSING with $inc: { attemptCount } and unique index on interviewId.
• After MAX_ATTEMPTS (3) at the document level, marks report FAILED without throwing (message consumed successfully).

Never catch errors inside handle and return normally — that deletes the message without retry (the CleverHire incident documented in sqs-consumer.ts).

Failure handling strategy

1. Per-message retry — Transient errors in handle → batchItemFailures → visibility timeout → redelivery.
2. DLQ after max receives — SST configures dlq.retry: 3 on both primary queues (infra/queues.ts) (three receives on the primary queue, then message moves to the DLQ).
3. Non-retryable payloads — Schema mismatches are deleted after logging.
4. Batch-level retry — Mongo unreachable at startup retries all messages in the batch.
5. Producer gaps — If queue URLs are unset locally, sqs.service.ts logs a warning and skips publish (no throw).
6. Downstream publish errors — Failed publishReportGenerate after settlement is logged only; settlement is already committed (settlement.service.ts).

Operational expectation from .cursor/rules/async-jobs.mdc: alarm on DLQ depth > 0.

Handler layout

Lambda entry	Thin handler	Business logic
interview-completed.handler	apps/functions/src/handlers/interview-completed.ts	@sophia/api/settlement via settlement.ts re-export
report-generate.handler	apps/functions/src/handlers/report-generate.ts	@sophia/api/report-generation via report-generation.ts re-export

interview-failed.ts exports a standalone consumer schema/handler for a possible future dedicated queue; production subscribes only interview-completed.ts, which unions completed and failed bodies.

Only the two queues above are deployed today. Expiry sweepers, notification delivery, retention purge, and ledger reconciliation are backlog — see Remaining work and Implementation tracker.

Further reading

• queues.md — schemas, IAM links, timeouts, batch settings
• Architecture diagram