Non-functional requirements & quality gates
As of: 2026-09-22
| Symbol | Meaning |
|---|---|
| 🟢 Met | Evidence in code, infra, or qualification |
| 🟡 Partially Met | Building blocks; target not demonstrated |
| 🔴 Not Measured / Not Started | No harness or not implemented |
| 🔵 N/A | Dropped or out of Phase 1 |
1. Performance & Latency
| Metric | Target | Status | Evidence / gap |
|---|---|---|---|
| Core app p95 response | < 2 s (excl. uploads/jobs) | 🔴 Not Measured | durationMs logged; no p95 gate |
| Stalled live-service detection | ≤ 60 s | 🔴 Not Started | FR17 / AC18 |
| Report readiness after end | p95 ≤ 10 min; p99 ≤ 30 min | 🔴 Not Measured | 5 min Lambda timeout; no SLO tracking |
| Recording availability after end | p95 ≤ 30 min | 🔴 Not Measured | No age metrics |
| End-of-turn to bot audio | p95 ≤ 3 s | 🔴 Not Measured | Bot pipeline; not in Sophia repo |
| Live incident availability (proctoring) | p95 ≤ 5 s | 🔴 Not Started | FR19 not productized |
| Detector failure recognition | ≤ 10 s | 🔴 Not Started | PRO05–PRO10 |
| Scheduling usability | ≤ 3 min | 🔴 Not Measured | — |
| Candidate preflight usability | ≤ 3 min | 🔴 Not Measured | — |
| Calendar sync | ≤ 2 / 5 min | 🔵 N/A | Dropped |
2. Scale & Load
| Target | Requirement | Status | Evidence / gap |
|---|---|---|---|
| Platform concurrent live | 10, ≥4 workspaces | 🔴 Not Met | Bot default 3 |
| Per-workspace cap | 3 | 🔴 Not Started | No enforcement in API |
| Soak test | 90 min | 🔴 Not Started | — |
| Daily workload qual | 100 sessions/day | 🔴 Not Started | — |
| 100-session mix composition | Durations, shortened, failures, reviewed normals | 🔴 Not Started | — |
| Bulk import | 100 rows | 🔵 N/A / 🔴 | Bulk OPEN |
| Code reality 3 bots cluster-wide | Documented | 🟢 Met | Contradicts release 10 |
| 11th / 4th reject without harming confirmed | — | 🔴 Not Started | HTTP 429 cluster only |
| API scale ECS min 2 max 6 | CPU/memory targets | 🟡 Partially Met | infra/api.ts |
| Overload blocks new, not confirmed | FR17 | 🔴 Not Started | — |
3. Quality Gates
| Gate | Target | Status | Notes |
|---|---|---|---|
| Phone detection | ≥90% recall, ≥95% precision | 🔴 Not Started | FR19 |
| Person detection | ≥90% / ≥95% per category | 🔴 Not Started | Held-out ≥100 events |
| Eye/gaze | ≥85% sens/spec; macro-F1 ≥0.80 | 🔴 Not Started | Credibility redesign OPEN |
| Report quality | ≥20 human reviews; zero fabrication | 🔴 Not Started | Pipeline exists |
| Usability | 5 users; ≥4/5 tasks | 🔴 Not Started | — |
| Wallet sequences | ≥1,000 randomized + invariants | 🔴 Not Started | §15.1 |
| False escalation | ≤5% clean sessions | 🔴 Not Started | §15.2 |
| WAL01–WAL24 matrix | Per scenario | 🔴 Not Started | — |
| PRO05–PRO10 failures | ≤10 s recognize; block/pause | 🔴 Not Started | — |
| Held-out evaluation set | Phone/person/gaze/failures/clean | 🔴 Not Started | §15.2 |
| 10 simultaneous + detectors | Pass audio/recording/cost | 🔴 Not Started | — |
| Eye calibration usability | ≥95% without engineering | 🔴 Not Started | — |
| AC03 isolation | Zero leakage | 🔴 Not Started | §16.1 |
4. Cost & Commercial
| Metric | Target | Status | Evidence / gap |
|---|---|---|---|
| Planning variable cost | $0.15/paid minute | 🔴 Not Measured | — |
| Stress variable cost | $0.25 p95 session | 🔴 Not Measured | — |
| Release gate blended | ≤ $0.15 or exception | 🔴 Not Started | — |
| Fixed infra launch budget | $500/month proposed | 🟡 Partially Met | ~$88.50 Fargate ref |
| QA provider budget | $500 proposed | 🔴 Not Started | Process |
| Alert rolling > $0.15/min | — | 🔴 Not Started | §8.3 |
| Alert p95 > $0.25/min | — | 🔴 Not Started | — |
| Incident spend protection | Pause bookings | 🔴 Not Started | WAL23 |
| Reference component costs | Planning table | 🔵 N/A | §17.4 |
| Break-even illustrations | Finance | 🔵 N/A | §17.4 |
5. Reliability & Consistency
| Requirement | Status | Evidence / gap |
|---|---|---|
| Money ops atomic + idempotent; balances ≥ 0 | 🟡 Partially Met | credit.service.ts, transactions |
| WAL01–WAL24 replay/concurrency | 🟡 Partially Met | Webhook idempotency; matrix not qual |
| Retries + DLQ review | 🟡 Partially Met | SQS DLQ; no alarms/replay tooling |
| Replay no duplicate charges | 🟡 Partially Met | Not fault-injection proven |
| Backup/rollback runbooks | 🔴 Not Started | SST retain; no runbooks |
| Reconciliation cache vs ledger | 🔴 Not Started | Comment in credit.repo.ts |
| AC03 zero cross-tenant | 🟡 Partially Met | Repo discipline; no suite |
| Global DB rules (integer money, tenant indexes) | 🟡 Partially Met | Models/repos |
| Partial SQS batch failures | 🟢 Met | partialResponses: true |
| Production Mongo required at startup | 🟢 Met | /health degraded when down |
6. Availability & Support
| Requirement | Status | Evidence / gap |
|---|---|---|
| Support Mon–Fri 09:00–18:00 IST | 🔴 Not Started | Process |
| Acknowledgement 1 business day | 🔴 Not Started | — |
| No 24/7 SLA | 🔵 N/A | Policy |
| Named on-call roster | 🔴 Not Started | FR17 |
| Candidate self-service recovery | 🟡 Partially Met | Join flows partial |
| Contract tests + real-payment smoke | 🟡 Partially Met | Vitest; smoke not documented |
| Stripe test mode replay paths | 🟡 Partially Met | Webhook verify |
ALB /health | 🟢 Met | infra/api.ts |
| Production min 2 Fargate tasks | 🟢 Met | infra/api.ts |
7. Security (§16)
| Requirement | Status | Implementation notes |
|---|---|---|
| Auth Cognito + Google + email | 🟡 Partially Met | Firebase in code |
| Verification, reset, logout | 🟡 Partially Met | Firebase routes |
| Candidate expiring scoped credentials | 🟡 Partially Met | Join token, session token |
| Authorization every request; immediate role change | 🟡 Partially Met | requireAuth / requireAdmin |
| Member own / admin workspace / candidate surfaces | 🟡 Partially Met | AC03 pending |
| No cross-workspace access | 🟡 Partially Met | Tenant repos |
| Tenant isolation indexes | 🟡 Partially Met | data-access rules |
| Phone encrypted; scoped URLs ≤5 min | 🟡 Partially Met | Export flow not started |
| Operator audit; ledger corrections | 🔴 Not Started | Ledger append-only 🟡 |
| Data lifecycle purge / legal hold | 🔴 Not Started | retentionExpiresAt only |
| AC03 isolation test | 🔴 Not Started | — |
| Logs exclude PII content | 🟡 Partially Met | Pino redaction |
| Security headers + CORS | 🟢 Met | secureHeaders() |
| Stripe webhook HMAC | 🟢 Met | — |
| Bot internal token | 🟢 Met | x-internal-token |
| Rate limiting beyond password verify | 🔴 Not Started | — |
| Admin 2FA / IP allow-list | 🔵 N/A | OPEN in spec |
8. Notifications (FR15)
Email + in-product only; no SMS; candidates never get scores.
| Notification type | Status |
|---|---|
| Email verification | 🔴 Not Started |
| Password reset | 🔴 Not Started |
| Team invitation | 🟡 Partially Met |
| Candidate booking / change / cancel | 🔴 Not Started |
| Reminder 24h / 1h before start | 🔴 Not Started |
| Unbooked invitation 24h before expiry | 🔴 Not Started |
| Candidate completion (no scores) | 🔴 Not Started |
| Report ready / failed | 🔴 Not Started |
| Calendar sync issues | 🔵 N/A |
| Admin payment / funding / expiry | 🔴 Not Started |
| Admin alerts (<30 min, zero, blocked, expiry) | 🔴 Not Started |
| Member funding visibility | 🔴 Not Started |
| ICS in email | 🔴 Not Started |
| Bounce / retry without duplicates | 🔴 Not Started |
| Timezone / deactivated creator fallback | 🔴 Not Started |
Queues: Only InterviewCompleted, ReportGenerate — no notification consumer.
9. Retention (FR16)
| Policy element | Spec | Status | Evidence / gap |
|---|---|---|---|
| Default 60 days evidence | — | 🟡 Partially Met | retentionExpiresAt at settlement |
| Admin 30 days option | — | 🔴 Not Started | — |
| Anchor expiry terminal / unbooked | — | 🟡 Partially Met | — |
| Draft files 30 days | — | 🔴 Not Started | — |
| Snapshot + deletion date UI | — | 🔴 Not Started | — |
| Cannot shorten promised availability | — | 🔴 Not Started | — |
| Purge notices 7d / 1d | — | 🔴 Not Started | Field only |
| PDF + evidence export | — | 🔴 Not Started | — |
| Export page 1h fresh auth | — | 🔴 Not Started | — |
| Scoped URLs ≤ 5 min | — | 🟡 Partially Met | Files module |
| Export copies purge 24h | — | 🔴 Not Started | — |
| Active purge 72h | — | 🔴 Not Started | — |
| Backups +30d tombstone | — | 🔴 Not Started | — |
| Legal hold | — | 🔴 Not Started | — |
| Financial records 7 years | — | 🔴 Not Started | — |
| Operational logs 30 days | — | 🟡 Partially Met | CloudWatch default |
| Audit 1 year | — | 🔴 Not Started | — |
| Primary region us-east-1 | — | 🟡 Partially Met | ACM/ALB |
| No indefinite customer storage | — | 🟢 Met (policy) | — |
10. Operations (FR17)
Capacity & signup
| Requirement | Status |
|---|---|
| 10 concurrent / 3 per workspace | 🔴 Not Met |
| 100 completed/day qual | 🔴 Not Started |
| 100 rows/import | 🔵 N/A / 🔴 |
| Public signup within capacity | 🔴 Not Started |
| Overload blocks new not confirmed | 🔴 Not Started |
Monitoring
| Signal | Status |
|---|---|
| Ledger invariants | 🔴 Not Started |
| Session/worker/room health | 🔴 Not Started |
| Provider errors | 🔴 Not Measured |
| Report/recording age | 🔴 Not Started |
| Email queues | 🔵 N/A |
| Stripe reconciliation | 🔴 Not Started |
| SQS consumer health | 🟡 Partially Met |
Alerting
| Requirement | Status |
|---|---|
| CloudWatch critical alarms | 🔴 Not Started |
| DLQ depth > 0 | 🔴 Not Started |
Log-based request.failed | 🔴 Not Started |
| Cost alerts | 🔴 Not Started |
| Stalled runtime ≤60s | 🔴 Not Started |
Operator console & runbooks
| Requirement | Status |
|---|---|
| Audited operator console | 🔴 Not Started |
| Backup restore | 🔴 Not Started |
| Rollback / deploy failure | 🟡 Partially Met |
| DLQ replay | 🟡 Partially Met |
| Credit adjustment without duplicate | 🔴 Not Started |
Observability
| Capability | Status |
|---|---|
| Structured JSON logging (Pino) | 🟢 Met |
Correlation x-request-id API ↔ SQS | 🟢 Met |
| PII redaction | 🟢 Met |
| Per-request access log | 🟢 Met |
| Error envelope | 🟢 Met |
GET /health + smoke | 🟢 Met |
| ALS org/user on all logs | 🟡 Partially Met |
| Metrics / APM / tracing | 🔴 Not Started |
Proctoring ops (FR19)
| Requirement | Status |
|---|---|
| Preflight all detector paths before paid start | 🔴 Not Started |
| Independent detector heartbeats | 🔴 Not Started |
| Missing capture block/pause meter | 🟡 Partially Met |
| Evidence incidents; no default 100% integrity | 🔴 Not Started |
| Neutral integrity reminders ≤2 | 🔴 Not Started |