# Blob Backend

Ultra-low latency (<200ms) Voice AI backend for the **Blob** kindergarten learning platform.
Built on **FastAPI + uvloop**, streaming async pipelines, hybrid Firebase/JWT authentication,
a server-rendered Admin Console, and a hierarchical curriculum seeder that ingests
`blob-seeder-v2` lesson families (canonical lessons + variation subcollections with embedded
steps) into Firestore in chunked, idempotent, atomic batches.

---

## Table of Contents

1. [Project Overview](#1-project-overview)
2. [Tech Stack](#2-tech-stack)
3. [Architecture Design](#3-architecture-design)
4. [Sequence Diagram — Voice Session Flow](#4-sequence-diagram--voice-session-flow)
5. [Admin Console & Curriculum Seeder](#5-admin-console--curriculum-seeder)
6. [Repository Layout](#6-repository-layout)
7. [Environment Variables](#7-environment-variables)
8. [Run Locally — Without Docker](#8-run-locally--without-docker)
9. [Run Locally — With Docker](#9-run-locally--with-docker)
10. [Test Suite](#10-test-suite)
11. [Infrastructure Design](#11-infrastructure-design)
12. [Health & Observability](#12-health--observability)
13. [API Surface](#13-api-surface)
14. [Blob Console — Prompt Playground](#14-blob-console--prompt-playground)

---

## 1. Project Overview

Blob is a real-time, voice-first learning companion for kindergarten children. The backend
serves three concurrent surfaces:

| Surface | Audience | Protocol | Purpose |
|---|---|---|---|
| **Voice WebSocket** | Flutter mobile client | WS (binary frames) | Streaming STT → LLM → TTS round-trip |
| **Auth REST API** | Mobile + Admin | HTTPS/JSON | Firebase ID-token exchange, hybrid JWT issuance |
| **Admin Console** | Internal admins | HTTPS/HTML + WS | Login, dashboard, curriculum seeding with live monitor |

Design goals: sub-200 ms first-audio latency, zero-cost free-tier providers, single-binary
deploy, and a hierarchical curriculum pipeline (canonical lesson families → variation
subcollections) with full audit trail and runtime variation selection.

---

## 2. Tech Stack

**Runtime**
- Python 3.12 · FastAPI · `uvloop` event loop · `uvicorn[standard]`

**Voice pipeline**
- STT — `faster-whisper` (local, free) · `webrtcvad` for VAD · `numpy` / `soundfile`
- LLM — `google-genai` (Gemini 2.0 Flash, free tier)
- TTS — `edge-tts` (free) · ElevenLabs SDK (optional fallback)
- gRPC stubs — `grpcio` / `grpcio-tools` for future microservice extraction

**Auth & Identity**
- `firebase-admin` (ID-token verification, Firestore SDK)
- `python-jose[cryptography]` (backend session JWTs)
- `httpx` (Firebase REST `signInWithPassword` for admin login)

**Admin Console**
- `Jinja2` server-rendered HTML · vanilla JS · static CSS dark theme
- `itsdangerous` (signed httpOnly session cookies, timed)
- `python-multipart` (login form parsing)

**State & I/O**
- Redis 7 (`redis[hiredis]`) — async session cache, future job pub/sub
- Cloud Firestore — admin/curriculum collections, audit logs

**Observability**
- `structlog` JSON logs · per-stage latency metrics
- `opentelemetry-api/sdk/instrumentation-fastapi` (ready, opt-in)

**Testing**
- `pytest` · `pytest-asyncio` · `httpx` TestClient

---

## 3. Architecture Design

```mermaid
graph TD
    subgraph Clients
        Flutter[Flutter Mobile Client]
        AdminBrowser[Admin Browser]
    end

    subgraph FastAPI["FastAPI Application (uvloop)"]
        WS[/"/ws/voice — Voice Gateway"/]
        AuthAPI[/"/auth/* — REST Auth"/]
        AdminUI[/"/admin/* — Console + WS Monitor"/]
        Health[/"/health, /health/services"/]
    end

    subgraph Core
        Session[VoiceSession Manager]
        Pipeline[Streaming Pipeline Orchestrator]
        AdminAuth[Admin Auth + Cookie Sessions]
        Seeder["Curriculum Seeder Worker<br/>(flat + family modes)"]
        CurrRepo[Curriculum Repository]
        Broadcaster[In-process Job Broadcaster]
        Audit[Audit Service]
    end

    subgraph Workers["Async Pipeline Workers"]
        STT[STT Worker — faster-whisper]
        LLM[LLM Worker — Gemini 2.0]
        TTS[TTS Worker — Edge-TTS]
    end

    subgraph External
        Firebase[(Firebase Auth + Firestore)]
        Redis[(Redis 7)]
        GeminiAPI[Google Gemini API]
        EdgeTTS[Edge-TTS Cloud]
    end

    Flutter <-->|WS binary frames| WS
    Flutter <-->|HTTPS JSON| AuthAPI
    AdminBrowser <-->|HTTPS HTML + WS| AdminUI

    WS --> Session --> Pipeline
    Pipeline --> STT --> LLM --> TTS
    AuthAPI --> Firebase
    AdminUI --> AdminAuth --> Firebase
    AdminUI --> Seeder --> Firebase
    AdminUI --> CurrRepo --> Firebase
    Seeder --> Broadcaster --> AdminUI
    Seeder --> Audit --> Firebase
    Session --> Redis

    STT -.local.-> STT
    LLM -->|REST stream| GeminiAPI
    TTS -->|WS stream| EdgeTTS
```

**Key design properties**
- **Streaming-first**: STT/LLM/TTS run as overlapping async workers; TTS begins synthesising on the first ~40 chars of LLM output.
- **Bounded queues + drop strategy**: stable under producer/consumer skew.
- **Hybrid auth**: `AUTH_PROVIDER=jwt` (legacy) or `firebase` (Firebase ID-token → backend JWT exchange). Switch via env var, no code change.
- **Hierarchical curriculum model**: canonical lesson families at root, variation subcollections with embedded steps. Runtime selects among variations by weight + eligibility rules.
- **Idempotent seeder**: SHA-256 of canonical payload short-circuits duplicate batches; atomic `db.batch()` writes per family with `set(merge=True)`. Auto-detects `blob-seeder-v2` format vs legacy flat JSON.
- **Two-pass relation resolution**: prerequisite/next/remediation lesson codes resolved to UUIDs before writes using batch Firestore lookups.
- **Per-job in-process broadcaster**: WebSocket fan-out for live seeding monitor (single-process; Redis pub/sub is the documented scale-out path).

---

## 4. Sequence Diagram — Voice Session Flow

```mermaid
sequenceDiagram
    autonumber
    participant C as Flutter Client
    participant A as /auth (REST)
    participant FB as Firebase Auth
    participant W as /ws/voice
    participant S as VoiceSession
    participant P as Pipeline Orchestrator
    participant STT as STT Worker
    participant LLM as LLM Worker (Gemini)
    participant TTS as TTS Worker (Edge-TTS)
    participant R as Redis

    C->>FB: signInWithPassword(email, password)
    FB-->>C: idToken
    C->>A: POST /auth/firebase-exchange  (idToken)
    A->>FB: verify_id_token(idToken)
    FB-->>A: uid + claims
    A-->>C: backend JWT (TTL=1h)

    C->>W: WS upgrade  ?token=backendJWT
    W->>W: verify backend JWT
    W->>S: open VoiceSession(child_profile_id)
    S->>R: SET session:{id} (state, lesson ctx)

    loop Realtime audio loop
        C-->>W: binary audio frame (40ms PCM)
        W->>P: enqueue audio chunk
        P->>STT: feed bytes → VAD-gated batches
        STT-->>P: partial / final transcript
        P->>LLM: stream transcript + lesson prompt
        LLM-->>P: token stream
        P->>TTS: buffer ≥40 chars → synthesise
        TTS-->>P: audio chunks (Opus/PCM)
        P-->>W: stream audio frames
        W-->>C: binary TTS frames
    end

    C->>W: close
    W->>S: finalize, persist transcript
    S->>R: DEL session:{id}
```

---

## 5. Admin Console & Curriculum Seeder

A server-rendered console at **`/admin/*`** for internal operators. No SPA build step.

### 5.1 Pages

| Route | Purpose |
|---|---|
| `GET /admin/login` · `POST /admin/login` | **ADC mode**: one-click login using gcloud identity (auto-bootstraps admin doc). **Password mode**: email+password via Firebase REST + Firestore role check. |
| `POST /admin/logout` | Invalidate session, clear cookie |
| `GET /admin/dashboard` | Live metrics: lesson/step/subject counts, latest seed job, Redis & Firebase health |
| `GET /admin/seeding` | 4-panel curriculum seeding console (Raw → Preview → Execute → Live Monitor) |

### 5.2 JSON / WebSocket APIs (admin-only)

| Endpoint | Purpose |
|---|---|
| `POST /admin/serialize-lesson` | Validate + canonicalise raw lesson JSON (no Firestore writes) |
| `POST /admin/start-seed` | Spawn async seeder job, returns `job_id` |
| `POST /admin/cancel-seed` | Cooperative cancel of an active job |
| `GET  /admin/seed-status/{job_id}` | Snapshot of current job state |
| `WS   /admin/ws/seed-monitor?job_id=…` | Live counter / progress stream from `JobBroadcaster` |

### 5.3 Curriculum data model (Firestore)

The curriculum is stored using a **hierarchical canonical family model** (Research-04 +
blob-seeder-v2 refactor). Each canonical lesson is a family root; variations live in a
subcollection with steps embedded inside each variation document.

```
admins/{uid}                                    # role: admin|super_admin, is_active
admin_sessions/{session_id}                     # signed cookie sessions, TTL + idle timeout
admin_audit_logs/{auto_id}                      # LOGIN, SERIALIZE, SEED_START, …
curriculum_source_versions/{sha256}             # idempotency dedup of identical batches
curriculum_source_manifests/{sha256}            # provenance tracking per seeded batch
curriculum_subjects/{uuid}                      # reference_code: MATH, LA
curriculum_categories/{uuid}                    # reference_code: MATH-COUNT, LA-PD, …
curriculum_lessons/{lesson_code}                # canonical lesson (family root)
    └── variations/{variation_code}             # variation doc (embedded steps array)
seed_jobs/{job_id}                              # status, counters, started_by, timestamps
seed_job_logs/{auto_id}                         # per-event log lines
```

**ID strategy**: `curriculum_subjects` and `curriculum_categories` use UUID4 document IDs.
`curriculum_lessons` and `variations` use **human-readable codes** as document IDs
(e.g., `K-MATH-008`, `K-MATH-008-CK-01`). The codes are also stored in
`canonical_lesson_code` and `variation_code` fields for querying and display.
Legacy flat-mode lessons write steps to a `steps/` subcollection with the step code as document ID.

**Relationship hierarchy**:
```
curriculum_subjects/{uuid}  (MATH)
    └── curriculum_categories/{uuid}  (MATH-COUNT)
            └── curriculum_lessons/{K-MATH-008}
                    ├── variations/{K-MATH-008-CK-01}   ← default, 4 steps
                    ├── variations/{K-MATH-008-CDA-01}  ← drill, 4 steps
                    ├── variations/{K-MATH-008-CSMP-01} ← narrative, 4 steps
                    └── variations/{K-MATH-008-MANS-01} ← assessment, 4 steps
```

See [`Documents/Implementation-Docs/LessonSeederDocs/FIRESTORE_COLLECTION_MODELS.md`](Documents/Implementation-Docs/LessonSeederDocs/FIRESTORE_COLLECTION_MODELS.md)
for full schema documentation with sample data.

### 5.4 Validation rules (serializer)

**Legacy flat path** (non-seeder-v2):
- `lesson_id` regex: `^K-[A-Z]+-(R)?\d{2,3}(-[A-Z0-9]+-\d{2})?$`
- Required: `subject_id`, `category_id`, `lesson_order > 0`, `agent_flow`, `ml_expectation`
- `validation_type=stt_semantic_match` → `expected_keywords` must be non-empty
- No self-prerequisite; duplicate `(subject_id, category_id, lesson_order)` rejected
- Unknown subject/category refs (when Firestore checks enabled) rejected

**Family path** (blob-seeder-v2 via `validate_families()`):
- Each family must have at least one variation
- Exactly one variation must be `is_default: true`
- Every variation must have at least one `interactive_question` step
- No duplicate `canonical_lesson_code` or `variation_code` within a batch
- `selection_weight > 0` required
- Subject/category UUID FK verified against Firestore
- Kindergarten pedagogical warnings (duration, emotion states)

### 5.5 Seeder semantics

Two ingestion modes, auto-detected by the `/admin/start-seed` endpoint:

**Family mode** (blob-seeder-v2 payloads):
- `parse_for_ingestion()` → `List[SeederFamilyDTO]` preserving hierarchy
- Two-pass relation resolution: codes → UUIDs for prerequisite/next/remediation refs
- SHA-256 dedup → if `curriculum_source_versions/{hash}` exists, marks as duplicate-skipped
- Atomic batch writes per family: `curriculum_lessons/{uuid}` + `variations/{uuid}` in single `db.batch()`
- Steps embedded inside variation documents (not a separate subcollection)
- Per-family fallback on batch failure; orphan cleanup via `delete()`
- Redis cache warmup: `lesson:{uuid}`, `lesson_variations:{uuid}`, `lesson_steps:{uuid}`

**Legacy flat mode** (non-seeder-v2 payloads):
- `serialize_payload_async()` → flat `CanonicalLesson` dicts
- Writes `curriculum_lessons/{lesson_id}` + `steps/` subcollection
- Same dedup, chunked batching, and rollback semantics as family mode

**Common properties**:
- Cooperative `asyncio.Event` cancel → final status `cancelled`
- Exponential backoff on retryable errors (429, 503, UNAVAILABLE)
- Final status: `completed` · `partial_failed` · `failed` · `cancelled`

---

## 6. Repository Layout

```
pyproject.toml                  # uv / PEP 517 project metadata

app/
├── main.py                     # Production FastAPI app, lifespan, router wiring
├── server.py                   # Alternative create_app() factory (lighter config)
├── api/                        # voice WS + auth REST routes
│   └── routes.py               # Conversation REST + WS routes (agent_flow)
├── core/                       # config, logging, redis, auth, firebase, pipeline, session
│   └── settings.py             # Alternate settings module (CORS, logging helpers)
├── handlers/                   # audio_stream handler
├── grpc/                       # generated stubs + clients (future microservices)
├── prompts/                    # Blob system prompt
├── services/                   # stt_service, llm_service, tts_service
├── models/                     # pydantic v2 models (auth, conversation, websocket_messages, …)
├── middleware/                 # error_handler (admin-aware redirects)
├── utils/                      # audio, auth, fallback_manager, message_builder
└── admin/                      # ── Admin Console & Seeder ──
    ├── routes.py               # all /admin/* endpoints (auto-detects family mode)
    ├── auth_service.py         # Firebase password sign-in, session cookies
    ├── deps.py                 # get_current_admin dependency
    ├── middleware.py           # IP allowlist (opt-in)
    ├── audit_service.py        # admin_audit_logs writer
    ├── dashboard_service.py    # metrics aggregation (cached 30s)
    ├── serializer_service.py   # parse → structure → enrich → validate + parse_for_ingestion()
    ├── serializer_validator.py # CanonicalLesson + family-level + cross-batch checks
    ├── seed_service.py         # JobHandle registry, start/cancel/start_family_job
    ├── seed_worker.py          # chunked idempotent writes (flat run() + hierarchical run_families())
    ├── seed_reference_data.py  # Seeds subjects/categories (MATH, LA + subcategories)
    ├── curriculum_repository.py # Query helpers for hierarchical model
    ├── websocket_monitor.py    # in-process JobBroadcaster
    ├── firestore_collections.py # Collection/subcollection name constants
    └── models.py               # CanonicalLesson, CanonicalLessonDocument, VariationDocument,
                                #   StepEmbedded, SeederFamilyDTO, SeedJob, AdminContext, …

templates/admin/                # base.html, login.html, dashboard.html, seeding.html, partials/
static/admin/                   # admin.css, admin.js, toast.js, seeding.js
proto/                          # llm.proto, stt.proto, tts.proto
tests/                          # pytest suite (see §10)
Documents/                      # architecture, plans, V4 implementation docs
docker-compose.yml · Dockerfile · requirements.txt · .env.example
```

> **Entrypoint**: run the backend with `python -m app.main` (or `app/main.py`) for the full production server with Firebase auth, Redis, Admin Console, and the complete voice pipeline.

---

## 7. Environment Variables

Copy `.env.example` → `.env` and fill in. Grouped highlights below; see `.env.example` for the full list with defaults.

### Core app
| Var | Default | Purpose |
|---|---|---|
| `APP_ENV` | `development` | `development` enables uvicorn `--reload` |
| `APP_HOST` / `APP_PORT` | `0.0.0.0` / `8000` | Bind address |
| `APP_LOG_LEVEL` | `DEBUG` | structlog level |

### Auth (hybrid)
| Var | Default | Purpose |
|---|---|---|
| `AUTH_PROVIDER` | `jwt` | `jwt` = legacy, `firebase` = Firebase ID-token exchange |
| `JWT_SECRET_KEY` | — | Legacy HS256 secret (when `AUTH_PROVIDER=jwt`) |
| `FIREBASE_PROJECT_ID` | — | Firebase project (when `AUTH_PROVIDER=firebase`) |
| `FIREBASE_SERVICE_ACCOUNT_PATH` | _(empty)_ | **Leave empty for ADC** — only set to a JSON key path for legacy/CI environments |
| `BACKEND_JWT_SECRET` / `BACKEND_JWT_TTL_SECONDS` | — / `3600` | Backend session JWTs minted after Firebase exchange |

### Voice pipeline
| Var | Default | Purpose |
|---|---|---|
| `GOOGLE_API_KEY` | — | Gemini API key |
| `GEMINI_MODEL` | `gemini-2.0-flash` | Gemini model id |
| `EDGE_TTS_VOICE` | `en-US-AriaNeural` | Edge-TTS voice |
| `STT_MODEL_SIZE` | `base` | faster-whisper model |
| `STT_CONFIDENCE_THRESHOLD` | `0.6` | Reject low-confidence transcripts |
| `VAD_ENERGY_THRESHOLD` / `VAD_SILENCE_MS` | `500` / `800` | VAD tuning |
| `SAMPLE_RATE` / `AUDIO_CHUNK_MS` | `16000` / `40` | Audio framing |
| `QUEUE_MAX_SIZE` / `TTS_BUFFER_CHARS` | `100` / `40` | Pipeline backpressure |
| `BYPASSLLM` | `False` | Echo mode for STT/TTS testing |

### Admin Console
| Var | Default | Purpose |
|---|---|---|
| `ADMIN_AUTH_MODE` | `firebase_password` | `adc` = one-click login using gcloud ADC identity (**development only** — auto-bootstraps super_admin); `firebase_password` = email+password form |
| `FIREBASE_WEB_API_KEY` | — | Required only when `ADMIN_AUTH_MODE=firebase_password` |
| `ADMIN_SESSION_SECRET` | placeholder — **must change** | Signs httpOnly admin cookies |
| `ADMIN_SESSION_TTL_SECONDS` | `28800` | Absolute cookie lifetime (8h) |
| `ADMIN_INACTIVITY_TIMEOUT_SECONDS` | `1800` | Idle timeout (30m) |
| `ADMIN_COOKIE_SECURE` | `False` | **Set `True` in production** (HTTPS) |
| `ADMIN_SEED_BATCH_SIZE` | `200` | Firestore commit chunk size |
| `ADMIN_ALLOWED_IPS` | _(unset)_ | CSV allowlist gating `/admin/*` |

### Infrastructure
| Var | Default | Purpose |
|---|---|---|
| `REDIS_URL` | `redis://localhost:6379/0` | Redis connection |

> **Firebase credentials**: the backend uses **Application Default Credentials (ADC) first** and only falls back to `FIREBASE_SERVICE_ACCOUNT_PATH` if ADC is unavailable. On GCP/Cloud Run, no key file is needed.

---

## 8. Run Locally — Without Docker

### Prerequisites
- Python **3.12+**
- Redis 7 running locally (`redis-server` or `brew services start redis`)
- **gcloud CLI** installed + ADC configured (see §9.1 for full Firebase setup)
- (For admin console) A Firebase project + Web API Key + a Firestore `admins/{uid}` doc with `is_active: true` and `role: "admin"` or `"super_admin"`

### Steps

```bash
# 1. Clone & enter
cd blob-backend

# 2. Virtual environment
python3.12 -m venv venv
source venv/bin/activate

# 3. Install dependencies
pip install -r requirements.txt

# 4. Generate gRPC stubs (one-time)
python -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. proto/*.proto

# 5. Configure
cp .env.example .env
# Edit .env:
#   - AUTH_PROVIDER=firebase
#   - FIREBASE_PROJECT_ID=<your-project>
#   - FIREBASE_SERVICE_ACCOUNT_PATH=     (leave empty for ADC)
#   - FIREBASE_WEB_API_KEY=<from Firebase console>
#   - GOOGLE_API_KEY=<Gemini key>
#   - ADMIN_SESSION_SECRET=$(openssl rand -hex 32)

# 6. Setup ADC (one-time per machine)
gcloud auth application-default login --project=<your-project>

# 7. Ensure Redis is up
redis-cli ping     # → PONG

# 8. Run
export PYTHONPATH=$PYTHONPATH:.
python -m app.main
# Look for: "Firebase credential: Application Default Credentials (ADC)"
# and: "Firebase Admin SDK initialized  credential_mode=application_default"
```

App is now live:
- API root: <http://localhost:8000>
- Health: <http://localhost:8000/health>
- Admin Console: <http://localhost:8000/admin/login>
- Voice WebSocket: `ws://localhost:8000/ws/voice?token=<backend-jwt>`

---

## 9. Run Locally — With Docker

```bash
# Build + start backend + Redis
docker-compose up --build

# Background mode
docker-compose up -d

# Tail logs
docker-compose logs -f backend

# Stop
docker-compose down
```

The compose file mounts the repo into `/app` so source edits hot-reload (when `APP_ENV=development`). `.env` in the repo root is loaded automatically.

The `docker-compose.yml` automatically mounts your local `~/.config/gcloud` directory
(read-only) into the container so ADC credentials are available without any extra setup.
If the directory doesn't exist, Docker will create an empty mount — the backend will still
start but Firebase features will fail until ADC is configured.

---

## 9.1. Firebase Authentication — ADC Setup

The backend uses **Application Default Credentials (ADC)** as the primary Firebase
authentication strategy. No service-account JSON key files need to be distributed.

### How it works

```
┌─────────────────────────────────────────────────────────────────────┐
│  Startup: initialize_firebase()                                     │
│                                                                     │
│  1. Is FIREBASE_SERVICE_ACCOUNT_PATH set and non-empty?             │
│     YES → credentials.Certificate(path)  [legacy fallback]         │
│     NO  ↓                                                           │
│  2. Use credentials.ApplicationDefault()                            │
│     • Checks GOOGLE_APPLICATION_CREDENTIALS env var first           │
│     • Falls back to ~/.config/gcloud/application_default_creds.json │
│     • On GCP (Cloud Run/GKE): uses metadata server automatically   │
│                                                                     │
│  3. firebase_admin.initialize_app(cred, options={projectId: ...})   │
└─────────────────────────────────────────────────────────────────────┘
```

### Local development (your workstation)

```bash
# 1. Install gcloud CLI (if not already)
# https://cloud.google.com/sdk/docs/install

# 2. Login and set your project
gcloud auth login
gcloud config set project gyfsoo-dev

# 3. Create ADC credentials (writes ~/.config/gcloud/application_default_credentials.json)
gcloud auth application-default login --project=gyfsoo-dev

# 4. Verify (optional)
cat ~/.config/gcloud/application_default_credentials.json | python3 -c "
import json, sys; d=json.load(sys.stdin); print(f'type={d[\"type\"]}, project={d.get(\"quota_project_id\",\"N/A\")}')"
# → type=authorized_user, project=gyfsoo-dev
```

Your `.env` should have:
```env
AUTH_PROVIDER=firebase
FIREBASE_SERVICE_ACCOUNT_PATH=        # empty → ADC
FIREBASE_PROJECT_ID=gyfsoo-dev
ADMIN_AUTH_MODE=adc                   # one-click login using gcloud identity
```

When `ADMIN_AUTH_MODE=adc`, the login page shows a **"Sign in as {email}"** button.
On first login, the backend automatically:
1. Creates a Firebase Auth user for your gcloud email (if one doesn't exist)
2. Creates a Firestore `admins/{uid}` doc with `role: super_admin`
3. Creates a session and redirects to the dashboard

No email/password setup in Firebase Console required for local development.

### Docker (local containers)

The `docker-compose.yml` mounts `~/.config/gcloud:/root/.config/gcloud:ro` automatically.
No additional setup needed after running `gcloud auth application-default login` on the host.

### Production (Cloud Run / GKE / GCE)

ADC is automatic — the platform injects credentials via the metadata server using the
compute service account. Just ensure:
1. The service account has **Firebase Admin SDK** and **Cloud Firestore** IAM roles.
2. `FIREBASE_PROJECT_ID` is set in the environment.
3. `FIREBASE_SERVICE_ACCOUNT_PATH` is **not set** (or empty).

### Troubleshooting

| Symptom | Cause | Fix |
|---|---|---|
| `DefaultCredentialsError: Could not automatically determine credentials` | No ADC file and no `GOOGLE_APPLICATION_CREDENTIALS` | Run `gcloud auth application-default login` |
| `PermissionDenied: Missing or insufficient permissions` | ADC user lacks Firestore/Auth IAM roles | Grant roles via `gcloud projects add-iam-policy-binding` |
| `Firebase Admin SDK initialized` with `credential_mode=application_default` in logs | ✅ Working correctly | — |
| `credential_mode=service_account_file` in logs | Using key file, not ADC | Remove/empty `FIREBASE_SERVICE_ACCOUNT_PATH` |

---

## 10. Test Suite

`pytest` + `pytest-asyncio`. Lives in [tests/](tests/).

```bash
# Full suite
pytest -v

# Specific module
pytest tests/test_admin_serializer.py -v

# Single test
pytest tests/test_admin_seed_worker.py::test_worker_cancel_stops_seeding -v

# Just the admin subsystem (15 tests)
pytest tests/test_admin_*.py -v
```

### Coverage map

| Area | Files |
|---|---|
| Health & boot | [test_health.py](tests/test_health.py), [test_config.py](tests/test_config.py) |
| Voice pipeline | [test_pipeline.py](tests/test_pipeline.py), [test_audio.py](tests/test_audio.py), [test_audio_stream_handler.py](tests/test_audio_stream_handler.py) |
| Services (STT/LLM/TTS) | [test_services.py](tests/test_services.py), [test_stt_confidence.py](tests/test_stt_confidence.py), [test_fallback_manager.py](tests/test_fallback_manager.py) |
| Conversation & message routing | [test_conversation_manager.py](tests/test_conversation_manager.py), [test_message_router.py](tests/test_message_router.py), [test_message_protocol.py](tests/test_message_protocol.py) |
| Auth (legacy + hybrid) | [test_auth.py](tests/test_auth.py), [test_auth_hybrid.py](tests/test_auth_hybrid.py), [test_token_exchange.py](tests/test_token_exchange.py) |
| WebSocket integration | [test_websocket_auth.py](tests/test_websocket_auth.py), [test_websocket_integration.py](tests/test_websocket_integration.py) |
| **Admin Console** | [test_admin_auth.py](tests/test_admin_auth.py) — cookie sign/verify · [test_admin_serializer.py](tests/test_admin_serializer.py) — 7 validator cases · [test_admin_seed_worker.py](tests/test_admin_seed_worker.py) — duplicate / partial-fail / cancel · [test_admin_routes_smoke.py](tests/test_admin_routes_smoke.py) — render + auth redirect |

**Current counts**: 99 tests passing, 3 pre-existing failures (config default assertion, 2 websocket integration JWT-related).

Manual matrix runner: [tests/manual_websocket_standard_matrix.py](tests/manual_websocket_standard_matrix.py).

---

## 11. Infrastructure Design

```mermaid
graph LR
    subgraph Edge
        LB[HTTPS LB / Reverse Proxy<br/>nginx · Cloud Load Balancer]
    end

    subgraph Compute["Backend (containerised)"]
        API[FastAPI + uvloop<br/>Dockerfile multi-stage<br/>EXPOSE 8000]
    end

    subgraph DataPlane
        Redis[(Redis 7<br/>session cache)]
        FS[(Cloud Firestore<br/>admins / curriculum / audit)]
        FBA[(Firebase Auth)]
    end

    subgraph ExternalAPIs
        Gemini[Google Gemini 2.0 Flash]
        Edge[Edge-TTS]
    end

    Mobile[Flutter Client] -->|HTTPS / WSS| LB --> API
    AdminBrowser[Admin Browser] -->|HTTPS / WSS| LB
    API <--> Redis
    API <--> FS
    API <--> FBA
    API -->|REST stream| Gemini
    API -->|WS stream| Edge
```

**Container build** is a 2-stage Dockerfile: builder stage installs deps + generates gRPC stubs, runtime stage carries only the slim Python image + `libgomp1` (faster-whisper requirement).

**Scale-out path**
- The voice pipeline is stateless per-WebSocket → horizontally scalable behind a sticky-session LB.
- Redis backs cross-process session state (today: lightweight; tomorrow: pub/sub for the seeder broadcaster — see R-2 in the V4 implementation plan).
- gRPC stubs are pre-generated for STT/LLM/TTS so each can be lifted into its own service when needed.

---

## 12. Health & Observability

- `GET /health` → `{ "status": "ok", "env": "..." }` (liveness probe)
- `GET /health/services` → upstream provider status (Firebase, Firestore, Redis, configured providers, effective auth mode)
- **Logging**: `structlog` JSON to stdout, level via `APP_LOG_LEVEL`
- **Latency metrics** emitted per stage:
  - `stt_result.latency_ms`
  - `llm_first_token.latency_ms`
  - `tts_audio_chunk.latency_ms`
- **Audit log** for every admin action in Firestore `admin_audit_logs/`
- OpenTelemetry packages are installed and ready — wire an exporter to enable traces.

---

## 13. API Surface

| Method | Path | Auth | Purpose |
|---|---|---|---|
| GET | `/health` | none | Liveness |
| GET | `/health/services` | none | Upstream readiness |
| POST | `/auth/firebase-exchange` | Firebase ID token | Mint backend JWT |
| POST | `/auth/legacy-login` | password | Legacy JWT path |
| WS | `/ws/voice?token=…` | backend JWT | Streaming voice session |
| GET | `/admin/login` | none | Login form |
| POST | `/admin/login` | form (email+password) | Establish admin session cookie |
| POST | `/admin/logout` | admin cookie | Invalidate session |
| GET | `/admin/dashboard` | admin cookie | Metrics dashboard |
| GET | `/admin/seeding` | admin cookie | Seeding console |
| POST | `/admin/serialize-lesson` | admin cookie | Validate/preview lessons |
| POST | `/admin/start-seed` | admin cookie | Start seed job |
| POST | `/admin/cancel-seed` | admin cookie | Cancel job |
| GET | `/admin/seed-status/{job_id}` | admin cookie | Job snapshot |
| WS | `/admin/ws/seed-monitor?job_id=…` | admin cookie | Live job stream |

### Demo / Development endpoints (no auth)

Defined in `app/api/demo.py` for local demos when mounted by `app.main`; `app/api/routes.py` is used by the alternative `app.server` app.

| Method | Path | Purpose |
|---|---|---|
| POST | `/transcribe` | Browser audio → transcribed text |
| POST | `/tts` | Text → base64 audio |
| WS | `/ws/{session_id}` | No-auth text-based AI conversation loop with per-session system prompt override |
| POST | `/conversation` | Text → Blob AI response + base64 audio (`app.server` only) |

#### Demo WebSocket message types (`/ws/{session_id}`)

| `type` field | Direction | Payload | Purpose |
|---|---|---|---|
| `set_system_prompt` | client → server | `{ "prompt": "<string or null>" }` | Override the system prompt for this session; send before or during a conversation. Pass `null` to revert to the default. |
| `end_session` | client → server | — | Close the session cleanly |
| `user_message` | client → server | `{ "text": "<string>" }` | Send a user chat turn |
| `blob_response` | server → client | `{ "text": "<string>" }` | Blob AI reply for the turn |
| `session_ended` | server → client | — | Acknowledgement of `end_session` |

---

## 14. Blob Console — Prompt Playground

`blob-console.html` is a self-contained, single-file browser console for iterating on Blob's system prompts and testing the demo WebSocket without any build step.

### Opening the console

```bash
# 1. Start the backend (see §8)
export PYTHONPATH=$PYTHONPATH:.
python -m app.main

# 2. Open the console in a browser — no server needed for the file itself
#    Either open it directly:
open blob-console.html          # macOS
xdg-open blob-console.html     # Linux

#    Or serve it from the repo root for cleaner URLs:
python3 -m http.server 5500
# then visit http://localhost:5500/blob-console.html
```

### First-time setup

No setup is required by default. The console assumes the backend is reachable at `http://localhost:8000` and derives the WebSocket URL automatically.
Use the **WebSocket mode** toggle in the **Transcript Test** panel to switch between REST (`POST /api/admin/test/chat`) and the demo WebSocket (`/ws/{session_id}`).
### API modes

| Mode | Description | Requires |
|---|---|---|
| **Backend REST** | Posts to `POST /api/admin/test/chat` on the backend for transcript testing and prompt refinement. | Backend running + endpoint enabled |
### Conversation modes

| Mode | Sub-mode | Purpose |
|---|---|---|
| **Child** | Companion | Free-play conversation as Blob the companion |
| **Child** | Learning | Lesson-guided conversation (select a lesson from the Lessons panel) |
| **Child** | Assessment | Assessment-style interaction |
| **Parent** | Onboarding | Simulates the parent onboarding / consent flow |

### Panel overview

| Panel | Purpose |
|---|---|
| **Overview** | Quick stats: lesson count, saved sessions, active mode |
| **Chat** | Live conversation window; connects/reconnects the demo WebSocket automatically |
| **Prompts** | Edit the system prompt for each sub-mode inline; changes take effect on the next connection |
| **Lessons** | Browse curriculum lessons fetched from the backend or Firestore; select a lesson to inject into the Learning prompt |
| **Sessions** | Saved test conversations; load a past session to replay or annotate |
| **Feedback** | Annotate each turn (👍 / 👎 + note), add overall feedback, then click **Refine Prompt** to generate an improved prompt via the AI |

### Testing the system prompt override

The backend now supports per-session system prompt overrides via the `set_system_prompt` WebSocket message. The console sends this automatically when you edit the prompt in the **Prompts** panel and start a new chat. You can also test it manually:

```javascript
// In the browser DevTools console while the chat WebSocket is open:
const ws = new WebSocket("ws://localhost:8000/ws/test-session");
ws.onopen = () => {
  // 1. Override the system prompt
  ws.send(JSON.stringify({
    type: "set_system_prompt",
    prompt: "You are a pirate who loves maths. Keep answers to one sentence."
  }));
  // 2. Send a message
  ws.send(JSON.stringify({ type: "user_message", text: "What is 2 + 2?" }));
};
ws.onmessage = e => console.log(JSON.parse(e.data));

// Revert to the default system prompt:
ws.send(JSON.stringify({ type: "set_system_prompt", prompt: null }));
```

### Prompt refinement workflow

1. Run a test conversation in the **Chat** panel.
2. Click **Save Session**.
3. Open the session in the **Feedback** panel; annotate turns and add overall notes.
4. Click **Refine Prompt** — the console calls `POST /api/admin/test/chat` with a meta-prompt that incorporates your feedback.
5. Review the diff between the old and refined prompt.
6. Click **Try refined prompt** to test it in a new chat session without permanently saving.
7. Click **Keep this prompt** to adopt it, or discard.

---

## License & Ownership

Internal to **Blob / Gyfsoo**. See `Documents/` for architecture decisions, V4 plans, and the
admin-console implementation blueprint (`Documents/Implementation-Docs/V4-06-IMPLEMENTATION-PLAN-ADMIN-CONSOLE-CURRICULUM-SEEDER.md`).