| |
Smarty the Fox Greeting

CDPS AI Scaling & Data Residency
The Chief Data and Policy Strategy (CDPS) branch architecture for analyzing 10,000+ public consultation entries while adhering to Protected B data sovereignty.

1. High-Volume Batch Processing

N8n workflows configured with internal looping and queue management to digest thousands of records over time without hitting API timeouts.

2. "Sliding Window" Context Management

A Python-based chunking_utility.py securely structures long PDF/Word documents into overlapping RAG-optimized pieces while perfectly preserving root ECCC metadata (consultation ID, entry ID).

3. Automated Metadata Validator

An integrated pre-processor (metadata_validator.py) verifies incoming data against standard ECCC Controlled Vocabularies (e.g., verified Regions and Themes) before executing expensive GenAI operations.

4. Bilingual Parity Auditor

  1. Mandatory Translation Parity:
    Triggers parallel analysis for French and English sources using Azure OpenAI endpoints based strictly in Canada East.
  2. Mathematical Drift Detection:
    Analyzes sentiment outputs from both streams, calculating divergence. If the 'linguistic drift' exceeds the tolerance threshold, it flags the entry for human CDPS review via a Teams bot webhook.
📄 Read the full technical blueprint: CDPS-Scaling.md

Step-by-step process (proposed)
  1. Author scenarios
    Write each decision point as a node with: id, prompt, choices[], and optional effects (score, flags, inventory, time, detection risk).
  2. Compile to a normalized graph
    Convert markdown / JSON / YAML into a single canonical graph model (nodes + edges + metadata).
  3. Validate
    Fail fast on broken content: missing node ids, dead links, unreachable endings, cycles (if disallowed), and invalid effects.
  4. Instrument & balance
    Compute helpful analytics: path lengths, expected risk, “dominant choice” detection, and story coverage.
  5. Render a playable experience
    Use the normalized graph to drive a UI: show prompt, show choices, apply effects, track state, and record a run log.
  6. Export outcomes
    Output the run transcript (JSON + human summary) for learning, audits, or classroom review.

Minimal node format

 
{
  "id": "start",
  "prompt": "You receive an email claiming to be from IT...",
  "choices": [
    {"label": "Click the link", "to": "phished", "effects": {"risk": "+=20"}},
    {"label": "Report to security", "to": "good_call", "effects": {"risk": "-=5", "badge": "reporter"}}
  ]
}

Pipeline sketch

Step-by-step process (proposed)
  1. Author scenarios
    Write each decision point as a node with: id, prompt, choices[], and optional effects (score, flags, inventory, time, detection risk).
  2. Compile to a normalized graph
    Convert markdown / JSON / YAML into a single canonical graph model (nodes + edges + metadata).
  3. Validate
    Fail fast on broken content: missing node ids, dead links, unreachable endings, cycles (if disallowed), and invalid effects.
  4. Instrument & balance
    Compute helpful analytics: path lengths, expected risk, “dominant choice” detection, and story coverage.
  5. Render a playable experience
    Use the normalized graph to drive a UI: show prompt, show choices, apply effects, track state, and record a run log.
  6. Export outcomes
    Output the run transcript (JSON + human summary) for learning, audits, or classroom review.

Minimal node format

 
{
  "id": "start",
  "prompt": "You receive an email claiming to be from IT...",
  "choices": [
    {"label": "Click the link", "to": "phished", "effects": {"risk": "+=20"}},
    {"label": "Report to security", "to": "good_call", "effects": {"risk": "-=5", "badge": "reporter"}}
  ]
}

Pipeline sketch

Design goals (philosophy lab vibes)
✂️
Citizens survive by default: the world is coherent and self-consistent.
Player interventions create tradeoffs (security vs speed, stealth vs certainty, etc.).
Bad endings are teachable and explain why (not just “you lose”).

n8n Gamification Engine
The current site is a Minimum Viable Product. The n8n process will augment this by bridging Enterprise Content Management with Generative AI and Gamification.
View Architecture Proposal

Suggested next files
If this repo stays tiny, these are the pieces that make it feel “real” fast.
  1. scenarios/ — sample adventures (phishing, USB drop, password reset, insider risk).
  2. schema.json — JSON Schema for node/graph validation.
  3. compiler.js — parse → normalize graph.
  4. validator.js — structural + semantic checks.
  5. runner.js — applies effects, records transcript.

Open questions
  • Primary authoring format: markdown, yaml, or json?
  • Do we allow cycles/loops ("investigate more")?
  • Do we track state as numeric risk, flags, inventory… or all three?

Refactoring reality one quanta at a time

Ishikawa (Fishbone) Discovery Phase

Vulnerability mapping phase: Add or remove root causes under each of the 6M categories to customize the diagram.