BlueprintParser — Construction Blueprint Intelligence

From the dashboard at /home, drag a drawing set onto the upload card. BP accepts a normal multi-page PDF; individual pages up to roughly 10,000 px on either axis work fine at 300 DPI, which covers the common 24×36 and 30×42 sheet sizes. You get a progress bar; when it finishes, the project appears in the project list.

Behind the scenes, the file is uploaded to S3 and a processing job is kicked off. You do not have to wait on the page; you can close the tab and come back later.

For each page, BP runs OCR, detects CSI MasterFormat codes (the industry-standard classification scheme — "08 14 00 = Wood Doors"), extracts drawing numbers from title blocks, detects schedules and keynotes, and classifies what's on the sheet. This takes roughly one minute per ten pages. A 200-page set is usually ready in five to ten minutes on the default Fargate tier.

You don't have to do anything during this step. When the project card shows Ready, click in.

The viewer lives at /project/[id]. It looks like a drawing review tool: a page sidebar on the left, a big canvas in the middle, a toolbar across the top, and a stack of panels you can flip open from the right edge. Pan with V, click with A, scroll to zoom (hold ⌘/ Ctrl if you're on a trackpad). The panels on the right — Text, CSI, LLM Chat, QTO, Schedules, Keynotes — are the feature surface. You only open the ones you need.

BP's text pipeline already knows where the schedules and keynotes are. What it doesn't know, until you ask, is where every door and window physically ison the floor plans. That's a YOLO run (an admin kicks it off — see Section 5). Once it's done, open the Schedules/Tables panel, point at the door schedule, and click Auto Parse. Then pick which YOLO class the tags are drawn inside (usually circle) and run Map Tags. BP binds each schedule row to every matching shape in the drawings and gives you a count per row.

Auto-QTO (the QTO → Auto tab) does all of that on autopilot: pick a material type, confirm the schedule, run the mapping, review the counts, export.

Everything in the QTO panel exports to CSV or Excel through the Export CSVbutton at the bottom of the panel. One row per tag or area item, with counts, pages, annotations, and notes. Paste it into the bid spreadsheet and you're done.

Most of the complexity in these docs is the answer to one question: what if the happy path doesn't work? If bucket fill leaks through an open doorway, Section 8 covers barriers and the four tuning knobs. If Auto-QTO blocks you at the start, Section 7 covers the YOLO class requirements and how to fix them from Admin. If chat runs out of context room on a big project, Section 9 explains the budget and the presets that trade structure for OCR. And if you want to know how the whole thing runs on AWS, Section 11 is the tour.

BP is organized as a set of engines that produce structured data and a single Viewer that consumes it, with a Graph/Output layerthat feeds everything back to the LLM and the Admin dashboard. Data flows upload → Preprocessing Engine → (optional) YOLO Post-Pipeline → Viewer surfaces (display + user parsing) → ParsedRegions → Graph/Output → LLM chat and downstream tools. Every stage persists to pageIntelligence or projectIntelligence; nothing is ephemeral.

• Preprocessing Engine(upload-time, always runs per page): rasterize at 300 DPI → Textract OCR (LAYOUT + TABLES) → drawing-number extraction → CSI code detection (3-tier matching) → text annotations (phones, equipment tags, abbreviations, 37+ types) → shape parse (keynote symbols via Python/OpenCV) → page intelligence analyze (classification, cross-refs, noteBlocks) → text-region classify (6-stage composite: LINE consumption, column-aware proposal, whitespace-rect discovery, Union-Find merge, per-region analysis, decision tree) → heuristic engine (9 rules, text-only mode) → table classifier → CSI spatial map (9×9 grid with title-block + right-margin zones).
• YOLO Post-Pipeline(admin-triggered, optional): SageMaker Processing job on g4dn.xlarge → YOLO annotations ingested → re-run heuristic engine with YOLO data → re-classify tables → composite region classifier (classifiedRegions) → YOLO density heatmap (text_box + vertical_area + horizontal_area aggregated on a 16×16 grid) → ensemble reducer (cross-signal agreement, suppresses keyword-only false positives) → auto-table-detector (emits AutoTableProposal[], read-only until user commits).
• Viewer (user surface, /project/[id]): canvas with pdf.js rasterizer + nine overlay layers, a dense toolbar, three mutually-exclusive modes (pointer/move/markup), a stack of toggleable right-side panels (Text, CSI, LLM Chat, QTO, Schedules/Tables, Keynotes, Specs/Notes, Page Intelligence, View All), a bottom Annotation Panel, and user-driven parsing tools (Table Parse, Keynote Parse, Notes Parse, Spec Parse [planned], Symbol Search, Bucket Fill, Shape Parse, Split Area, Scale Calibration). Section 02 enumerates the full tree.
• Graph / Output Layer (downstream consumers): every user-committed ParsedRegion promotes via /api/regions/promote into pageIntelligence.parsedRegions; CSI tags merge into pages.csiCodes via idempotent mergeCsiCodes; computeProjectSummaries rebuilds projectIntelligence.summaries (schedules, notesRegions, specRegions, parsedTables, yoloTags). The context-builder assembles a budget-allocated LLM payload from all of the above; the CSI network graph + hub pages are derived once per project and surfaced in chat and the View All panel.
• Admin Dashboard: Pipeline config (toggle stages, concurrency, per-company heuristic overrides), Heuristics tab (DSL editor for rules), AI Models tab (register YOLO models, trigger runs), LLM Config (provider + context-budget allocations across 19 sections), Overview (reprocess controls + Lambda CV job status). Every viewer feature has a corresponding admin tuning surface.

How they connect: Preprocessing runs once per upload and populates JSONB blobs on pages. YOLO Post-Pipeline augments those blobs on admin trigger. The Viewer reads them into a Zustand store (17 slice hooks) and renders overlays; user parsing tools write back to the same blobs via the generic /api/regions/promote commit route. The Graph/Output layer re-derives summaries on every commit and serves them to Chat and the Admin dashboard. The whole stack is one database shape with one write path per mutation, which is why every number in the UI traces back to a pixel on a page.

Everything in BP ultimately fits into two axes. Horizontally, a project is a list of pages; each page carries OCR text, a classification, detected text annotations, detected tables, CSI codes, and (optionally) YOLO detections. Vertically, a project is a bundle of cross-cutting data: annotations (user markups + YOLO + takeoff), pageIntelligence (per-page structured analysis), and projectIntelligence (a project-wide summary including the CSI network graph, hub pages, and discipline breakdown).

The preprocessing pipeline, the LLM context builder, the takeoff engine, and the viewer all read and write through those two shapes. Section 03 walks through how the data actually arrives; Section 11 walks through where it is stored and why.

BP is the same codebase in every deployment tier — the difference is purely which external services are configured. A development machine with Docker and a Groq free-tier API key can run the full viewer against a locally-hosted PostgreSQL instance, parse tables with img2table and Camelot, and chat with an LLM, all without a single AWS credential. Add an S3 bucket and page images become durable; add the full Terraform stack and you get CloudFront, Textract, Step Functions, and Label Studio; add a SageMaker Processing role and a YOLO ECR image and YOLO object detection becomes available on-demand.

BP is a single Next.js 16 application (App Router, React 19, TypeScript) backed by PostgreSQL 16 via drizzle-orm. State in the viewer lives in a single zustand store with slice selectors. LLM access goes through a thin adapter layer over the Anthropic, OpenAI, and Groq SDKs, plus a generic OpenAI-compatible endpoint for Ollama and self-hosted models. The CSI network graph is rendered with d3-force. Python sidecars (pdfplumber, Camelot, img2table, TATR, OpenCV, Tesseract, and the YOLO inference container) are spawned from TypeScript via stdin/stdout JSON. AWS deployment is codified in infrastructure/terraform/— 13 files totaling the full stack including ECS, RDS, S3, Step Functions, IAM, Secrets Manager, and CloudFront.

Every feature under the Viewer, nested by the DOM/panel hierarchy it renders into. One-line description under each. If a feature has sub-modes or tabs, those are indented under the parent. This is deliberately exhaustive; skim for the shape, read for the specifics.

• Canvas core
  • PDFPage.tsx pdf.js rasterizer
    Renders the current page as a bitmap at the user's zoom scale; caches the last 8 rendered pages as ImageBitmaps for instant tab-back.
  • Zoom / Fit / Pan controls
    +/− buttons, Fit-to-window, wheel-zoom in Move mode, drag-to-pan in Move mode.
  • Thumbnail sidebar
    Collapsible left-side page list with page-name + drawing-number labels; click to jump, scrolls synchronously with the main canvas.
• Modes (mutually exclusive, keyboard-bound)
  • Pointer (A)
    Click-to-select overlays; double-click opens edit dialogs for markups, annotations, parsed regions.
  • Move / Pan (V)
    Click-drag pans the canvas, wheel zooms. No overlay interaction.
  • Markup
    Draw rectangle, polygon, or freehand stroke. Opens MarkupDialog on finish for name + note + color pick from 20-color palette.
  • Group / multi-select
    Shift-click-add plus empty-canvas lasso; applies bulk ops (delete, recolor, category change) across selection.
• Canvas overlay layers(stable z-order, normalized 0–1 coordinates)
  • SearchHighlightOverlay
    Yellow boxes around tsvector search hits from the toolbar text-search.
  • TextAnnotationOverlay
    Boxes around detected phones, equipment tags, room names, abbreviations (37+ annotation types).
  • KeynoteOverlay
    Keynote shape detections (circles, hexagons, diamonds) with inner-text OCR; gated by the showKeynotes toggle.
  • AnnotationOverlay
    The master layer — YOLO detections, user markups, takeoff items, shape-parse output, symbol-search results. Click-to-select, drag-to-move, vertex-edit on polygons. Also hosts the draw-rect state machine for Parse flows.
  • ParseRegionLayer
    Saved ParsedRegion outlines + grid preview, color-coded by type (keynote amber, notes blue, spec violet, schedule pink). Also renders the shared parseDraftRegion dashed preview while a user is actively parsing.
  • GuidedParseOverlay
    Draggable row + column boundary lines rendered during a Guided Parse (keynote and notes share this via a prop-based API).
  • FastManualParseOverlay
    Stage 4 Notes primitive — double-click snaps to Textract LINE, derives columns from line margins. Pending rework into ParagraphOverlay (paragraph-level hit-test + adjustable BB + Cmd+C/V template paste).
  • DrawingPreviewLayer
    Rubber-band preview while the user is dragging to draw a new markup or bbox.
  • ParsedTableCellOverlay
    TATR cell-structure overlay for parsed tables; click-a-cell to search by its text, double-click to toggle highlight.
• Toolbar
  • Back-to-dashboard + click-to-rename project name
    Inline edit on the project name, persists to projects.name.
  • Zoom controls (− / % / +) + Fit
    Symmetric bracketed zoom; Fit recalculates for the current page dimensions.
  • Mode toggle (Pointer / Pan / Markup)
    3-state button; keyboard shortcuts A, V.
  • Symbol Search button
    Draw a template bbox to find all instances; exposes Lite / Power / Custom presets for confidence thresholds.
  • Menu dropdown
    Labeling wizard (YOLO training export), Settings, Page Intelligence toggle, Admin link, Help tips toggle, Export PDF (disabled placeholder).
  • Text search
    Full-text search over OCR via Postgres tsvector; highlights on canvas + lists hits in Text panel.
  • Trade filter / CSI code filter
    Filter the CSI Network Graph, View All, and QTO lists by trade or specific CSI division.
  • YOLO toggle (+ per-model confidence sliders)
    Shows when any YOLO annotation is loaded. Dropdown chevron opens per-model sliders (yolo_medium / yolo_primitive / yolo_precise).
  • Six panel toggles
    Text / CSI / LLM Chat / QTO / Schedules/Tables / Keynotes (+ Specs/Notes in the D2 panel orchestrator).
• Right-side panels (toggleable, stackable)
  • Text Panel (TextPanel.tsx)
    OCR text viewer, searchable, per-word Textract confidence, click-to-jump-to-canvas-position.
  • CSI Panel (CsiPanel.tsx)
    Detected CSI MasterFormat codes grouped by division; toggle between page-scope and project-scope; click a code to highlight triggers on canvas.
  • LLM Chat Panel (ChatPanel.tsx)
    Project- or page-scoped chat, streams via SSE; has 20 tools (search, read-page, highlight, zoom, list-schedules, count-takeoff, etc.).
  • QTO Panel (TakeoffPanel.tsx) — quantity takeoff
    • Count tab
      Click-to-count with color-coded markers; auto-deduplicates via YOLO tag bindings where available.
    • Area tab (+ Scale Calibration + Bucket Fill + Split Area)
      Polygon draw or bucket-fill flood with text-as-wall barrier detection; Scale Calibration is a 2-point known-dimension flow; Split Area slices a saved polygon.
    • Linear tab
      Polyline length; same scale-calibration model as Area.
    • Auto-QTO tab
      Suggests pages with likely schedules (ensemble-driven after Stage 2b); “Find & Parse Doors Schedule” style shortcuts auto-trigger Table Parse.
    • All tab
      Flat list of every committed takeoff item, exportable to CSV.
  • Schedules/Tables Panel (TableParsePanel.tsx)
    • Auto Parse
      Multi-method merger: OCR-positions, Textract TABLES, OpenCV lines, img2table. Returns a consolidated grid + confidence.
    • Guided Parse
      User draws region, server proposes row/col boundaries, user drags to adjust, client extracts cells.
    • Manual Parse
      User draws column BBs + row BBs; grid extraction runs client-side via word-center hit-test.
    • Compare / Edit
      Side-by-side method outputs; edit cell text and re-save.
    • Map Tags section
      Bind tag column of a parsed table to YOLO tag instances; auto-infers scope + pattern.
  • Specs/Notes Panel (SpecsNotesPanel.tsx) — D2 orchestrator
    • Spec Parse tab
      Stage 5 scope, currently stubbed. Will target full-page vertical-column spec layouts (PART / SECTION / GENERAL NOTES dense prose).
    • Notes Parse tab (NotesPanel.tsx)
      • Index
        Project-wide table of detected note regions from summaries.notesRegions; row click jumps to page and opens Parser pre-filled with the region bbox.
      • Classifier
        Per-page Accept / Edit / Reject cards for Layer-1 classified textRegions (notes-numbered + notes-key-value). Accept one-click-promotes via /api/regions/promote; Reject persists to rejectedTextRegionIds with stale-ID cleanup.
      • Parser — Auto sub-mode
        Server runs parseNotesFromRegion (numbered-first, K:V fallback) + CSI detection; client shows dashed preview on canvas.
      • Parser — Guided sub-mode
        Propose row/col boundaries, user drags on GuidedParseOverlay, client extracts grid.
      • Parser — Fast-manual sub-mode (pending rework)
        Double-click Textract LINE to snap columns. Known-broken on dense multi-line paragraphs; scheduled for redesign as ParagraphOverlay primitive.
      • Parser — Manual sub-mode
        Draw column BBs + row BBs; grid extracted client-side via word-center hit-test. The always-works fallback.
    • Keynotes tab (KeynotePanel.tsx)
      • All Keynotes
        Flat list of every parsed keynote table across the project; CSV export.
      • Auto / Guided / Manual / Compare
        Same sub-mode taxonomy as Table Parse but scoped to bubble-keyed keynote grids.
  • Page Intelligence Panel (PageIntelligencePanel.tsx)
    Read-only dump of pageIntelligence for the current page — classification, crossRefs, textRegions, noteBlocks, heuristicInferences, ensembleRegions. Debug/inspection surface.
  • View All Panel (ViewAllPanel.tsx)
    Project-wide list with per-entity eye toggles (master-eye memento); surfaces schedules, parsed tables, keynotes, notes, specs, YOLO tags, CSI codes. Clickable-graph substrate for future LLM-side reasoning.
• Bottom Annotation Panel
  Horizontal summary row grouping Markups, YOLO detections, and takeoff items by source; filter chips per category.
• Dialogs / modals
  • Markup dialog
    Name + note + color on markup save.
  • Bucket Fill Assign dialog
    Assign a filled region to an Area item + color; surfaces HTTP errors inline.
  • Scale Calibration dialog
    Two-point calibration with known real-world distance + unit selector (ft, in, m, mm).
  • Symbol Search config
    Confidence presets (Lite / Power / Custom) + per-project defaults.
  • Export CSV modal
    Keynote / Schedule / Notes export with column selection.
• Standalone tools (trigger from toolbar or panels)
  • Symbol Search
    Draw a bbox around any symbol on the page; CV matcher finds every other instance across the project.
  • Bucket Fill
    Flood-fill area computation from a click point; text-as-wall paradigm with 1k/2k/3k/4k resolution slider; assigns to Area item with error surfacing.
  • Split Area
    Slice a saved polygon with a user-drawn line into two children.
  • Shape Parse
    Python/OpenCV keynote-shape detector (circles, hexagons, diamonds, pills, squares). Runs at upload; results live in pages.keynotes.
  • Scale Calibration
    Per-page; stored in scaleCalibrations[pageNumber]. Required before any Area/Linear takeoff produces real-world units.
• ParsedRegion outputs (write path from Viewer into the graph)
  • type: "schedule"
    Tabular grid from Schedules/Tables panel.
  • type: "keynote"
    Key → Description grid from the Keynotes tab.
  • type: "notes"
    Notes-numbered or notes-key-value grid from Notes Parse.
  • type: "spec" (Stage 5 planned)
    Section-header → body list from Spec Parse.
  • type: "legend"
    Symbol legend variant; shares NotesData shape.
  All types commit through the generic POST /api/regions/promote route. Server merges CSI tags into pages.csiCodes via mergeCsiCodes and refreshes projectIntelligence.summaries after the transaction commits.

Top: the toolbar. Left: a collapsible page sidebar with thumbnails. Center: the canvas, which renders the current page and its overlays. Right: a stack of toggleable panels — Text, CSI, LLM Chat, QTO, Schedules/Tables, Keynotes, Page Intelligence — which fly in from the right edge when activated. Bottom: the Annotation Panel, a summary row grouping markups, YOLO detections, and takeoff items by source.

The toolbar is dense by design — a working estimator needs every mode and every panel within one click of the canvas. Below is a live static rendition with fake data so you can see the exact layout and control styling without loading a real project.

From left to right: the back arrow returns to the project dashboard; the project name is click-to-rename; the - and + buttons bracket the current zoom percentage and the Fit button auto-fits the page; the 3-state mode toggle selects Pointer / Pan / Markup; the Symbol button opens a draw-a-bbox-to-find-all-instances workflow; the Menu button opens the dropdown (shown below). The right half of the toolbar carries the text search, the trade filter, the CSI code filter, the YOLO toggle (with per-model dropdown when multiple models are loaded), and the six panel toggles.

The canvas has three mutually-exclusive modes, controlled by setMode() in the viewer store. The internal mode values are "pointer", "move", and "markup". Keyboard shortcuts are A (pointer), V (pan/move), and switching to Markup mode activates the drawing tools. Pointer mode clicks on overlays to select them; move mode click-drags the canvas to pan and mouse-wheel zooms; markup mode lets you draw rectangles, polygons, or freehand strokes.

Markup annotations are user-authored overlays: a rectangle, polygon, or freehand stroke with an associated name and optional multi-line note. Each markup gets one of twenty colors drawn from the TWENTY_COLORS palette (src/types/index.ts), and the markup dialog captures a name and note on save. Markups show up in the Annotation Panel at the bottom of the viewer under the MARKUPScategory, and they're persisted to the annotations table with source = "user".

The menu collects operations that don't belong on the main toolbar: a labeling wizard for building YOLO training sets, a settings modal, a toggle for the Page Intelligence panel, a link to the admin dashboard, and a help tips toggle that reveals contextual tooltips across the UI.Export PDFis present but disabled — it's the obvious future feature.

When a project has any YOLO annotations loaded, the purple YOLObutton appears. It toggles the canvas overlay and opens the Detection Panel. When multiple models are loaded, the dropdown chevron reveals a per-model panel with independent confidence sliders — useful for tuning the output on a project where yolo_medium is noisy but yolo_precise is conservative.

The right half of the toolbar holds six panel toggles. Panels slide in from the right edge and can be stacked. Each is independently toggleable and each keeps its own internal state.

The canvas mounts several overlay layers on top of the rendered page. They stack in a stable z-order and each can be toggled or filtered independently. All overlays operate in normalized 0–1 page coordinates so they stay aligned when the user zooms or the page dimensions change across pages.

• SearchHighlightOverlay— yellow boxes around tsvector search hits.
• TextAnnotationOverlay— boxes around detected phone numbers, equipment tags, room names, and other text-annotation matches.
• KeynoteOverlay— detected keynote shapes (circles, hexagons, diamonds) with their inner text.
• AnnotationOverlay— the main YOLO + user markup layer. Click-to-select, click-to-edit.
• ParseRegionLayer— saved table parse regions, click to jump to the parsed data.
• GuidedParseOverlay— the live grid lines rendered while tuning a Guided Parse.
• DrawingPreviewLayer— the rubber-band preview while the user is drawing a new markup.

The viewer's state lives in a single Zustand store at src/stores/viewerStore.ts (1,986 lines). The store is large but access is scoped through seventeen slice hooks— each hook returns a narrow set of fields memoized by useShallow, so components only re-render on changes to their own slice.

src/components/viewer/AnnotationOverlay.tsx is the center of the drawing logic — 2,581 lines that handle every canvas mode, hit testing, bucket fill commit, split-area, vertex edit, polygon drawing, symbol search, markup, calibration, and keynote/table parse region selection. The file has one structural trap that bit the group-tool fix on 2026-04-19 and keeps coming back: adding a new mode requires touching four places.

The companion architecture doc at featureRoadMap/BPArchitecture_422.md contains the full mode table and exact line numbers if you're about to add a new tool.

Before any area or linear takeoff will produce real-world numbers, the user has to calibrate the page scale. You click Set Scale in the Area tab, click two points on a known dimension (a grid line, a labeled wall), and enter the real-world distance plus a unit. Calibration is stored per page in scaleCalibrations[pageNumber]— reusing a polygon on a new page requires recalibrating unless the pages share the same scale.

The pipeline is triggered when the projects route creates a project row. On local development, it's invoked inline via processProject(projectId) in src/lib/processing.ts. On AWS, the same function runs inside the cpu-pipeline ECS task, launched by an AWS Step Functions state machine (infrastructure/terraform/stepfunctions.tf). In both cases the public-ID lookup, the processing body, and the post- processing project analysis are identical — the state machine just gives you durable retries, CloudWatch logging, and isolation from the web task.

export async function processProject(projectId: number): Promise<{
  pagesProcessed: number;
  pageErrors: number;
  processingTime: number;
}> {
  // ... fetch project, download PDF, count pages ...
  // ... mapConcurrent(pageNums, pageConcurrency, processOnePage) ...
  // ... analyzeProject + computeProjectSummaries + warmCloudFrontCache
}

The diagram below shows the exact order each page moves through. Every stage is individually wrapped in a try/catch: a failure in Textract doesn't prevent text annotation detection from running on the (possibly empty) output, a failure in CSI detection doesn't prevent heuristics from firing, and so on. Per-page errors are written to pages.error so you can spot partial results in the admin dashboard without the whole project being marked as failed.

1. Rasterize at 300 DPI (rasterizePage()) — the full-resolution PNG for display. This is what the viewer's canvas eventually renders.
2. Upload PNG + 72 DPI thumbnail to S3 — both get Cache-Control: public, max-age=31536000, immutable so CloudFront can cache forever. The thumbnail backs the sidebar.
3. Re-rasterize at a safe DPI if the 300 DPI image exceeds 9500 px in either dimension— Textract rejects images above 10000 px. A 24×36" sheet at 300 DPI is 10800 px; the pipeline re-rasterizes at roughly 263 DPI in that case. The re-rasterized buffer is only used for OCR; the display image stays at 300 DPI.
4. OCR via Textract with Tesseract fallback (analyzePageImageWithFallback()) — produces a structured TextractPageData with per-word bounding boxes and confidence scores. If Textract is unreachable or credentials are missing, it falls through to Tesseract.
5. Flatten OCR into raw text (extractRawText()) — the concatenation used by the PostgreSQL search_vector column for /api/search.
6. Extract the drawing number from the title block (extractDrawingNumber()). This is what becomespages.name— e.g., "A-101".
7. Detect CSI codes (detectCsiCodes()) — the 3-tier matcher. Output is written to pages.csi_codes. Section 04 explains the algorithm.
8. Detect text annotations (detectTextAnnotations()) — runs the 10 detector modules from src/lib/detectors/registry.ts: contact, codes, dimensions, equipment, references, trade, abbreviations, notes, rooms, csi-annotations. Produces a grouped annotation list with sub-categories.
9. Analyze page intelligence (analyzePageIntelligence()) — discipline and drawing-type classification, cross-references to other sheets, note blocks. This is the first place the pipeline produces a structured summary of the page.
10. Classify text regions (classifyTextRegions()) — OCR-based identification of where the tables, schedules, legends, and note blocks live on the page. Produces textRegions[] with confidence scores.
11. Run the heuristic engine in text-only mode (runHeuristicEngine()). Rules that do not require YOLO classes fire here. Section 05 explains how YOLO-augmented heuristics re-run later, after a YOLO job completes.
12. Classify tables (classifyTables()) — combines text regions and heuristic inferences into classified table candidates (door schedule, finish schedule, keynote table, etc.).
13. Compute CSI spatial heatmap (computeCsiSpatialMap()) — divides the page into a 9×9 grid plus title-block and right-margin special zones and tallies CSI instances per zone. Initial pass is OCR-only; a YOLO pass can refresh later.
14. Upsert the pages row and rebuild the search_vector via a raw SQL to_tsvector('english', rawText). This is the single write-point for the whole per-page pipeline.

Once every page has finished (or errored), the pipeline switches gears. It reads all processed pages back, passes them to analyzeProject() which computes the discipline breakdown, hub pages, cross-reference graph, and the CSI network graph via buildCsiGraph(). The result — a structured projectIntelligence blob and a short text projectSummary — is written back to the projects row. A separate computeProjectSummaries() pass then builds the per-index lookup tables (CSI → pages, trade → pages, keynote → pages, text-annotation → pages) that lookupPagesByIndex() reads at O(1) from LLM tool calls.

The final step is a best-effort CloudFront cache warm: each page PNG gets a HEAD request so CloudFront edge locations pull it ahead of the first viewer hit. Failures are logged and ignored.

Pages run in parallel via a small mapConcurrent() helper with a default limit of 8. The limit is per-company configurable through companies.pipelineConfig.pipeline.pageConcurrency and the Admin → Pipelinetab — raise it on a beefy Fargate task, lower it if Textract throttles you. The spatial grid size is also configurable via pipelineConfig.pipeline.csiSpatialGrid.

CSI MasterFormat is an industry-standard classification system maintained by the Construction Specifications Institute. Every specification section has a code like 08 14 00 — Division 08 (Openings), section 14 (Wood Doors), subsection 00 (general). Division is the most useful unit: it maps to trade, it's stable across projects, and it's dense enough that 25 divisions can meaningfully describe any project while being small enough to fit the whole project's division breakdown into a single paragraph of LLM context.

Because CSI is a closed vocabulary, BP can turn a full page of OCR (easily 4–10k characters) into a single short tag list like [22 00 00, 23 05 00, 26 05 00] plus confidence scores and then look up detailed division data on demand through tool calls. That pattern is what lets BP scale LLM chat to 200-page projects without ever exceeding a Sonnet-sized context budget.

src/lib/csi-detect.ts implements a rule-based matcher against a MasterFormat database. The matcher runs three tiers in order of specificity; a code can be tagged by any tier it passes, and the tier with the highest confidence wins. Defaults:

The matcher keeps only codes whose final score beats matchingConfidenceThreshold (default 0.40). All defaults are overridable per-company through companies.pipelineConfig.csi and the Admin → CSI tab, which also lets admins upload a custom CSI database TSV (useful for trades like fire alarm that benefit from an expanded vocabulary).

const DEFAULT_CONFIG: CsiDetectConfig = {
  matchingConfidenceThreshold: 0.4,
  tier2MinWords: 3,
  tier3MinWords: 5,
  tier2Weight: 0.75,
  tier3Weight: 0.50,
};

After detection, computeCsiSpatialMap()bins every CSI-tagged text annotation (and, after a YOLO pass, every YOLO-inferred region) into a 9×9 grid plus two special zones: title-block (y > 0.85) and right-margin(x > 0.75, y < 0.85). The output is a list of zones with per-division counts, which is what the LLM sees when it calls getCsiSpatialMap(pageNumber).

The spatial map is how the LLM answers questions like "what's in the top-right of this sheet?" or "where are the MEP systems concentrated?" without having to scan every word box on the page. The 3×3 demo below is a simplified view of a single page; the real default grid is 9×9.

At the project level, buildCsiGraph() converts the per-page CSI tags into a graph: nodes are CSI divisions, edges are co-occurrence relationships between divisions (with three types: co-occurrence, cross-reference, and containment), and clusters are pre-defined groupings: MEP (22, 23, 26, 27, 28), Architectural (08, 09, 12), Structural (03, 05), and Site (31, 32, 33). The graph carries a fingerprint that BP uses as a cache key so it can avoid re-computing the graph when nothing on the project has changed.

The graph is what makes LLM-driven navigation tractable. Tools like getCrossReferences return hub pages ranked by incoming reference count; lookupPagesByIndex({ index: "csi" }) answers "which pages have Division 22?" in O(1). When the LLM wants to find plumbing plans, it doesn't scan 200 pages — it queries the graph once and gets page numbers back.

All three layers surface to the LLM as tool calls. Section 09 walks through the full tool set, but the CSI-specific story is:

• getProjectOverview() returns the project-level CSI divisions and cluster membership — the coarse first look.
• getCsiSpatialMap(pageNumber)returns the per-page heatmap — the "zoom in" query.
• getCrossReferences(pageNumber?) returns the cross-reference edges and hub pages — navigation.
• lookupPagesByIndex({ index: "csi", key: "22" })is the O(1) "give me every page tagged with Division 22" query.
• detectCsiFromText(text)lets the LLM run the 3-tier matcher on arbitrary input strings (e.g. a user's question).

The run path is: admin opens Admin → AI Models, a tab rendered by src/app/admin/tabs/AiModelsTab.tsx, picks a model from the models table and a project, confirms the cost warning, and clicks Run. That fires POST /api/yolo/run, which writes a new processingJobs row and calls startYoloJob() in src/lib/yolo.ts. That function creates an AWS SageMaker Processing job pointing at the YOLO ECR container, mounts the project's pages/ prefix in S3 as input, and sets yolo-output/ as the output destination.

While the job runs (usually a few minutes per project on an ml.g4dn.xlarge), the admin UI polls GET /api/yolo/status every ~5 seconds and shows live status, execution ID, and CloudWatch logs. When the container finishes, it writes per-page detection JSONs to S3; a webhook hits POST /api/yolo/load, which reads the JSONs, normalizes them into the annotations table (with source = "yolo"), and triggers a refresh of the CSI spatial heatmap + heuristic engine in YOLO-augmented mode.

Because a SageMaker Processing job can cost real money if mis-triggered, BP has several layers of safety:

• Company-level sagemakerEnabled toggle. Flipped off by default. Flipping it on requires the admin password stored in app_settings. When off, the entire YOLO run path returns an error immediately without touching AWS.
• Quota enforcement. Per-company concurrent-job caps check against the processingJobs table before starting a new job. Toggleable in the same admin panel.
• Per-user canRunModels flag. Regular members can view YOLO results but cannot initiate a run. Admins get the flag by default; root admins can grant it selectively.
• Root-admin-only model sharing. A YOLO model uploaded by one company is not automatically visible to others. The root admin has to grant model access per-company via the modelAccess table.

Once a YOLO run completes and results are loaded, they show up in the viewer's Detection Panel (DetectionPanel.tsx, ~780 lines of React). The panel has three sub-tabs:

Each YOLO model in BP carries a confidence threshold (default 0.25). The threshold applies both to storage (low-confidence detections can be filtered at ingest by the admin config) and to display — the toolbar's per-model slider in the YOLO dropdown filters the overlay live without mutating the underlying data.

On top of confidence, the toolbar exposes a trade filter (dropdown populated from the distinct trades inferred from CSI codes) and a CSI code filter (searchable dropdown). Both apply to the canvas overlay independently of confidence; they let estimators zero in on a single scope without fighting with confidence sliders.

BP ships reference models trained on construction drawings. The specific classes available depend on which models are registered in the models table for your company. Some commonly-useful classes:

The tables, title_block, and drawings classes are special: Auto-QTO (Section 07) strictly requires them. The drawings class marks the content region of a sheet, and tables + title_block mark regions to exclude from counts (so you don't double-count tags that appear inside a schedule).

The heuristic engine (src/lib/heuristic-engine.ts) runs in two modes. Text-only mode fires during the initial processing pass; YOLO-augmented mode re-runs after YOLO data loads. Each rule has optional yoloRequired and yoloBoostersfields. A rule like "if the page contains the word 'concrete' AND a tables class was detected, infer schedule_presentwith CSI division 03" will skip silently during text-only mode and fire when YOLO runs later.

{
  id: "concrete-schedule",
  outputLabel: "schedule_present",
  outputCsiCode: "03",
  minConfidence: 0.6,
  textKeywords: ["concrete", "mix design"],
  yoloRequired: ["tables"],          // will skip until YOLO runs
  yoloBoosters: ["title_block"],     // adds confidence if present
  spatialConditions: [
    { type: "contains", region: "tables", textRegion: "header" },
  ],
}

This is the stacking story the rest of the docs will refer back to: YOLO models are not a replacement for heuristics, they're an additional signal that heuristics can chain on top of. A new YOLO class becomes a new input for existing rules, a new input for Auto-QTO, and a new input for the CSI spatial heatmap — without anyone having to touch the rules. Tools stack.

Open the Schedules/Tables panel from the toolbar (the pink-accented button). The panel renders TableParsePanel.tsx, which orchestrates five tabs: All Tables, Auto Parse, Guided, Manual, and Compare/Edit Cells. Each tab points at the same saved region data (pageIntelligence.parsedRegions[]) but exposes a different parsing strategy.

Auto Parse is the default path. You draw a bounding box around the table region on the canvas and click Process Regions. The backend runs a multi-method parse pipeline and merges the results into a single grid. Each method is a fallback for the others — a grid-line-heavy schedule is easy for Camelot; a crowded CAD-printed one is easier for TATR or img2table.

Results are merged via src/lib/grid-merger.ts and persisted through POST /api/table-parse. The saved parse becomes a parsedRegion on the page with { type: "schedule", data: { headers, rows, tagColumn, colBoundaries, rowBoundaries, csiTags } }.

Sometimes auto parse gets the structure nearly right but misplaces a row boundary or merges two columns. Guided Parse is the answer. You still draw a region, but the panel exposes three tuning sliders whose defaults live in GuidedParseTab.tsx:44:

• Row tolerance— maximum vertical drift between OCR tokens considered part of the same row. Too tight and valid rows split; too loose and adjacent rows merge.
• Min column gap— smallest horizontal gap that separates two columns. If the panel is merging columns, widen this.
• Min hits ratio— fraction of rows a column must appear in to count as a real column. Filters out one-off blobs.

As you drag a slider, the panel re-posts to POST /api/table-parse/propose with debounce and redraws the grid line overlay on the canvas. When the grid looks right, you save the parse through the same /api/table-parse endpoint as Auto Parse.

For hostile tables that defeat both Auto and Guided — scanned PDFs with faint grid lines, handwritten schedules, tables that mix multiple scales — Manual Parse lets an estimator define every header and every row by hand. The Compare/Edit Cells tab is an after-the-fact review surface: pick two parse attempts (e.g. Auto with default config and Auto after tweaking), diff the cells, and keep the better one. Both live in the same panel for workflow continuity.

Parsing a schedule into a grid is useful, but the real leverage comes from Map Tags: binding each row's tag value to every YOLO shape instance that contains that tag text somewhere on the drawings. Once a schedule has been parsed with a tag column identified, the Map Tags section appears. The user picks which YOLO class the tag is drawn inside (usually circle or hexagon) — or selects "no shape" to let BP find free-floating text matches — and clicks Map Tags. The binding is processed by POST /api/projects/[id]/map-tags-batch, which calls into src/lib/yolo-tag-engine.ts to do bbox-intersection + OCR text matching across every page.

The result is a set of YoloTags written to the yolo_tagstable and surfaced in the Detection Panel's Tags sub-tab. Each YoloTag is the anchor data structure that Auto-QTO reads from to compute final quantities — one row per tag value, one instance per YOLO match, pages tracked per instance. The connection from schedule to drawings to counts is literally this step.

Under the hood, findOccurrences() in src/lib/tag-mapping/find-occurrences.ts dispatches on item.itemType to one of five matchers. Auto-QTO almost always picks yolo-object-with-tag-shape (type 4); the other four support composite-classifier, manual QTO, and future workflows. Each matcher produces raw hits; a shared scorer composes pattern match, region weight, scope, and fuzziness into a single confidence score and tier.

The persisted data after a successful parse + map is:

• pages.page_intelligence.parsedRegions[]— one entry per parsed table, with { type: "schedule", bbox, data: { headers, rows, tagColumn, ... } }.
• annotations— no new rows (parse alone doesn't add annotations).
• yolo_tags— one row per unique tag value discovered during Map Tags, with a list of matched annotation IDs and page numbers.
• projectIntelligence.schedules[] catalog is updated by the post-processing summarizer on the next run of computeProjectSummaries(), so Auto-QTO can find the schedule without loading every page.

Given a material type (doors, finishes, equipment, plumbing, or electrical), Auto-QTO:

1. Finds or asks you to parse the relevant schedule page.
2. Reads the parsed schedule's tag column.
3. Asks you which YOLO tag-shape class the tags are drawn inside.
4. Runs Map Tags (Section 06) over the entire project, binding every unique tag value to its YOLO shape instances, while excluding the schedule region itself and the title block so it doesn't double-count.
5. Produces a line-item list with counts, pages, and an editable review surface. Estimators can hand-adjust before exporting.
6. Exports to CSV / Excel for the bid package.

The thing to understand: Auto-QTO does not invent quantities. It simply counts tag occurrences identified by YOLO + OCR, and the fidelity of the count is a function of the fidelity of the YOLO model and the schedule parse. If the model missed a door, Auto-QTO will miss that count; that's why the review step matters and why the user always has an override.

Auto-QTO hard-blocks the material picker unless the project's YOLO run includes three specific classes: tables, title_block, and drawings. These are exclusion / inclusion markers for the counting logic — without them, Auto-QTO can't cleanly differentiate "tags inside the schedule" from "tags out on the drawings."

const QTO_STRICT_EXCLUSION_CLASSES = ["tables", "title_block", "drawings"] as const;
const QTO_RECOMMENDED_CLASSES = ["grid", "vertical_area", "horizontal_area"] as const;

If a strict class is missing, Auto-QTO shows a blocker callout with a link to Admin → AI Models: you need to run a YOLO model that has those classes before you can proceed. The recommended classes (grid, vertical_area, horizontal_area) are soft — missing them just produces a warning banner, not a block.

Auto-QTO starts with the material picker. Each option binds to a schedule category that the table classifier uses when suggesting pages for the schedule step. Custom material types are supported via a free text input — BP singularizes by stripping a trailing s as a rough stem.

const MATERIALS = [
  { type: "doors",      label: "Doors",       scheduleCategory: "door-schedule",       icon: "D" },
  { type: "finishes",   label: "Finishes",    scheduleCategory: "finish-schedule",     icon: "F" },
  { type: "equipment",  label: "Equipment",   scheduleCategory: "material-schedule",   icon: "E" },
  { type: "plumbing",   label: "Plumbing",    scheduleCategory: "plumbing-schedule",   icon: "P" },
  { type: "electrical", label: "Electrical",  scheduleCategory: "electrical-schedule", icon: "Z" },
];

Once a material is picked, Auto-QTO drops you into a step machine whose state lives in the qto_workflows table. The canonical step IDs come from AutoQtoTab.tsx:11:

const STEP_SEQUENCE = ["select-schedule", "confirm-tags", "map-tags", "review", "done"] as const;

Under the hood, the counting engine supports five item-type strategies via findItemOccurrences() in src/lib/yolo-tag-engine.ts. Auto-QTO almost always defaults to type 4 (yolo-object-with-tag-shape), but the other four are available to composite-classifier and manual QTO workflows:

• yolo-only — count instances of a class with no text.
• text-only — count occurrences of a literal OCR string, no YOLO.
• yolo-with-inner-text — YOLO shape containing specific text.
• yolo-object-with-tag-shape — primary object + tag-shape combo (the default for Auto-QTO).
• text-pattern — detect a repeating tag series (T-01, T-02, T-03, ...).

In demo mode (isDemo === true), Auto-QTO workflows persist in the Zustand store only — they disappear when the tab is closed. The same is true for annotations, markups, takeoff items, and parse results. This is a deliberate design choice so that the public /demo/project/* route can let anonymous users drive a full Auto-QTO workflow without polluting the shared demo project.

Bucket Fill is the top strip of the Area tab inside the QTO panel (src/components/viewer/AreaTab.tsx). It appears as a tri-state button: disabled (no active area item), idle, active, or barrier mode. The state is controlled by two Zustand flags: bucketFillActive and bucketFillBarrierMode. A third store field, bucketFillResolution, drives the dominant tuning knob (see below).

The client-side Web Worker at src/workers/bucket-fill.worker.tsdoes all the heavy lifting. It's one pass: no retry, no speculative seeding. The single-pass design is the reason the tool feels instant even on a 4096-pixel image: no round-trip to the server, no Python subprocess, just an OffscreenCanvas and a tight TypeScript loop.

The four knobs are not equal. If your fill leaks, or stops short, or over-bleeds through text, the order in which to reach for them is:

Post-2026-04-22, the worker does not pre-erase text blocks. Letter boundaries simply act as dark pixels and the flood stops at them like it stops at walls. The reasoning: pre-erasing OCR'd text was error-prone (it enlarged bboxes and erased parts of adjacent walls) and the user rarely wants a fill to cross text anyway — text in a room almost always labels the room or notes something inside it, which stays inside the polygon.

1. Open QTO → Area.
2. Calibrate the page scale (Set Scale→ click two points → enter distance + unit). Without calibration the areas still render but the quantity column will say "page units" instead of a real measurement.
3. Create an area item (name, color) or click an existing one to make it the active target.
4. Click the Bucket Fill button to arm. Pick a resolution on the slider (1k / 2k / 3k / 4k).
5. Click inside the room you want to measure. The worker runs; you see the preview overlay appear.
6. If the fill leaked through an open doorway, toggle Barrier mode. Click two points to draw a virtual wall. Click inside the room again. Repeat until the fill is sealed.

For U-shaped rooms and hallways enclosing a courtyard, the worker runs findHoleBorders()after the outer contour trace. Each hole is simplified separately with Douglas– Peucker. The preview overlay uses fill-rule="evenodd" so the courtyard renders as a true hole rather than a filled island, and computeRealArea() subtracts the hole areas from the outer polygon.

When the client worker fails (very old browser, extremely large images, corrupted ImageBitmap), the viewer falls back to the server path: POST /api/bucket-fill → src/lib/bucket-fill.ts → scripts/bucket_fill.py(Python OpenCV). The server path predates the Web Worker and uses an adaptive-threshold algorithm rather than Otsu, so its results can differ on low-contrast images. It's a safety net, not the preferred path.

{
  "type": "result",
  "polygon": [{ "x": 0.142, "y": 0.388 }, ...],
  "holes": [[{ "x": 0.32, "y": 0.51 }, ...]],   // evenodd-compatible
  "vertexCount": 24,
  "areaFraction": 0.017,     // decorative — use computeRealArea()
  "retryHistory": [...]      // present only when worker retried
}

Bucket Fill returns a polygon in normalized 0–1 coordinates. To turn that into square feet, BP needs two pieces of information: the scale calibration for the current page, and the page's pixel dimensions. src/lib/areaCalc.ts runs computeRealArea(vertices, pageWidth, pageHeight, calibration)— shoelace formula in pixel space, divided by the calibration's pixels-per-unit, returning a real area in the calibrated unit. Holes are subtracted. Supported units:

Bucket Fill is not a feature on its own — it's an input method into the Area tab of the takeoff panel. Once the polygon is created, it behaves like any other area takeoff entry: the underlying annotations row has source = "takeoff", the group rollup appears in the QTO panel, the item can be edited, re-colored, moved between groups, or exported to CSV with the rest of the project. This is the design pattern the whole tool lives on: new capabilities stack on top of existing ones. Bucket Fill adds a fast path to create area polygons; everything downstream (aggregation, grouping, export) is unchanged.

A blueprint LLM has a fundamental problem: it can't read the PDF. Even if you chunk a 200-page drawing set into text, the raw OCR is too noisy (page numbers, dimensions, plot stamps, revision blocks) and too long to fit into a context window while leaving room for reasoning. BP solves this by inverting the flow:

1. The LLM does not see the blueprint directly. What it sees is a compact structured summary built by the context builder.
2. The LLM gets tools.Twenty of them — the full BP_TOOLS set. They query the pre-computed structured data, run BP engines on arbitrary inputs, and (for a small subset) drive the viewer. Tools are what give the model leverage.
3. Tools compose inside an agentic loop. The model can call multiple tools in parallel per round, feed results back, and iterate up to ten rounds per turn before being forced to answer.
4. Context budgets are per-model. A Sonnet call gets a very different slice of data than a Groq call. Admins can override priorities per-company via a preset system.

The "LLM tool making" story the user gets isn't a feature in the UI — it's the shape of the tool registry in src/lib/llm/tools.ts and the pattern you use when adding a new tool: write a tool definition with a JSON Schema input, write an executor, flip a switch in executeToolCall(), and the model can call it on the next request. The next subsection enumerates all twenty.

Every tool in BP_TOOLS gets a card below, pulled live from src/lib/llm/tools.ts. Filter by group. Action tools (the ones that mutate data or drive the viewer) are marked amber so the distinction between "read" and "write" is visually obvious.

The set is small by design. Each tool corresponds to one of the structured surfaces BP already maintains, rather than being a low-level primitive the model has to compose. An LLM given 20 purpose-built tools will pick the right one faster than one given 80 composable primitives.

• Navigation tools — getProjectOverview, getPageDetails, lookupPagesByIndex, getCrossReferences— answer "where is" questions without paging through pages.
• Structured reads — getAnnotations, getParsedSchedule, getCsiSpatialMap, getSpatialContext— pull a single structured chunk at a time, so the model can ask for exactly what it needs.
• Text fallback — searchPages and getPageOcrText let the model hit raw OCR only when structured data is insufficient. Raw OCR sits at priority 10 in the context builder for the same reason: last resort.
• Engine invocation — detectCsiFromTextlets the model run the 3-tier CSI matcher on a user's phrase. detectTagPatterns runs the tag-pattern detector. Tools can wrap BP engines so the model can do analysis on the fly.
• YOLO tag tools — scanYoloClassTexts, mapTagsToPages, getOcrTextInRegion— bridge between OCR text and YOLO regions. These are what let the model answer "how many doors have a 90-minute fire rating on the second floor" by joining schedule rows to shape detections.
• Action tools (amber) — navigateToPage, highlightRegion, createMarkup, addNoteToAnnotation, batchAddNotes. The viewer interprets these as side effects. The model can say "show me page 42" and the viewer actually scrolls there.

BP's chat endpoint (POST /api/ai/chat) invokes streamChatWithTools() on the configured adapter. All three SDK adapters (anthropic.ts, openai.ts, groq.ts) implement the same interface:

async *streamChatWithTools(options: LLMToolUseOptions): AsyncIterable<ToolStreamEvent> {
  const maxRounds = options.maxToolRounds ?? 10;
  const tools: Tool[] = options.tools.map(toAnthropicShape);
  const msgHistory = prepareMessages(options.messages);

  for (let round = 0; round < maxRounds; round++) {
    const stream = await client.messages.stream({ model, system, messages: msgHistory, tools, ... });

    for await (const event of stream) {
      if (event.type === "content_block_delta" && event.delta.type === "text_delta")
        yield { type: "text_delta", text: event.delta.text };
      else if (event.type === "content_block_start" && event.content_block.type === "tool_use")
        yield { type: "tool_call_start", name: event.content_block.name, id: event.content_block.id };
    }

    const finalMsg = await stream.finalMessage();
    const toolUseBlocks = finalMsg.content.filter(b => b.type === "tool_use");

    if (toolUseBlocks.length === 0 || finalMsg.stop_reason !== "tool_use") {
      yield { type: "done" };
      return;
    }

    const toolResults = [];
    for (const block of toolUseBlocks) {
      const result = await options.executeToolCall(block.name, block.input);
      yield { type: "tool_call_result", name: block.name, id: block.id, result: JSON.stringify(result) };
      toolResults.push({ type: "tool_result", tool_use_id: block.id, content: JSON.stringify(result) });
    }

    msgHistory.push({ role: "assistant", content: finalMsg.content });
    msgHistory.push({ role: "user", content: toolResults });
  }

  yield { type: "text_delta", text: "\n\n(Reached maximum tool call rounds)" };
  yield { type: "done" };
}

The key behaviors to notice: text deltas stream live (the user sees the response materialize a word at a time); tool calls don't block the stream (the model's reasoning text shows up before the tools execute); each round's tool calls run in parallel before the next LLM turn starts; the loop terminates as soon as stop_reason !== "tool_use" or after 10 rounds, whichever comes first. On Opus-sized models, 3 rounds is typical; 10 is a safety cap, not an expected value.

Before the loop even starts, the server calls assembleContextWithConfig() in src/lib/context-builder.ts. The function takes a list of candidate sections (CSI codes, classification, annotations, parsed tables, etc.), sorts them by priority, and packs them into a character budget chosen for the current model. Bigger-window models get more context; smaller free-tier models stay lean to leave room for tool rounds.

The fallback default is DEFAULT_CONTEXT_BUDGET = 24000 characters — ~6000 tokens — which is what unknown providers and unknown models get.

SECTION_REGISTRYenumerates the 20 sections the context builder can assemble into a page- or project-scope prompt. Each has a default priority (lower = earlier, higher priority) and a description. At run time the builder sorts by priority, computes per- section budgets from the admin's preset or per-company overrides, fills each section to its budget, and truncates anything that overflows. Unused allocations flow into an overflow pool so the next section can use the slack.

There are three presets in SECTION_PRESETS:

The adapter is chosen by src/lib/llm/resolve.ts based on the llm_configs table and, optionally, per-user overrides from user_api_keys. The fallback chain is:

1. The user's API key (from user_api_keys, encrypted at rest).
2. Company-wide config from llm_configs (set by company admin).
3. Environment variable (ANTHROPIC_API_KEY, OPENAI_API_KEY, GROQ_API_KEY).

The adapter interface is identical across providers — LLMClient in src/lib/llm/types.ts defines streamChat() and streamChatWithTools(). Adding a new provider is a matter of writing a new file in src/lib/llm/ that implements the interface and wiring it into resolve.ts. For OpenAI-compatible endpoints (Ollama, self-hosted vLLM, llama.cpp servers), the existing openai.tsadapter works directly — you set provider = "custom" and a baseUrl.

The user-facing surface is Admin → LLM Context (src/app/admin/tabs/LlmContextTab.tsx). It exposes:

• Enable / disable each of the 20 sections per company.
• Override any section's default priority.
• Pick a preset or set custom percent allocations.
• Inspect post-assembly section metadata (included, truncated, char count) — so admins can see exactly what made it into the prompt.
• Edit the system prompt (overrides DEFAULT_SYSTEM_PROMPT).
• Attach company-specific domain knowledge (free-text).

The LLM provider / model picker lives next door in Admin → AI Models → LLM Config. Both pages write to the same set of tables and both updates take effect on the next chat turn (no deployment required).

BP is multi-tenant at the row level. Every user-visible table carries a company_id, and every /api/admin/* route runs a row-scope check in src/lib/audit.tsbefore reading or writing. A company admin sees their own company's projects, users, CSI config, heuristics, and LLM configs — nothing cross-company.

A root admin (a user with isRootAdmin = true) bypasses company scoping. Root admins can create new companies, assign root admins, edit any company's pipelineConfig, reveal global app settings, and flip the sagemakerEnabled toggle on any company. There is intentionally no UI for "become root admin" — the bit is set directly in the database by a system operator.

A browser hits CloudFront at assets.*for page images and thumbnails, and the ALB at the primary domain for everything else. The ALB routes HTTPS to two ECS services: the main app (a Next.js container) and Label Studio (a separate labeling UI for training data work). Both run in Fargate — no EC2 to manage. Secrets come from Secrets Manager; the DB is RDS PostgreSQL; durable storage is S3 behind CloudFront.

blueprintparser-app is the Next.js 16 container defined in infrastructure/terraform/ecs.tf. Task definition: 2 vCPU / 4 GB. It serves the entire React app, handles every API route, runs Drizzle queries against RDS, pushes processing jobs to Step Functions, and proxies LLM calls. Auto scaling is CPU-based (target 70%) with memory as a guardrail (80%). A circuit breaker is enabled on deployments so a broken image rolls back automatically.

name            = "blueprintparser-app"
cpu             = 2048          // vCPU × 1024
memory          = 4096          // MiB
container_image = "{{ecr}}/beaver_app:latest"
container_port  = 3000
health_check    = { path = "/api/health", interval = 30, timeout = 5 }
execution_role  = "beaver_ecs_execution_role"   // ECR pull, logs, secrets read
task_role       = "beaver_ecs_task_role"         // S3, Textract, SageMaker, SFN
desired_count   = var.ecs_desired_count          // auto-scaled
deployment_controller = "ECS"
circuit_breaker = { enable = true, rollback = true }

The app task needs direct access to Secrets Manager (to pull DATABASE_URL, NEXTAUTH_SECRET, LLM keys), S3 (to write uploads and read page images), Textract (OCR), SageMaker (start/stop jobs), and Step Functions (start executions). Those are all attached to the task role in iam.tf.

Long-running processing is offloaded to a second ECS task named blueprintparser-cpu-pipeline. It's the same container image as the main app, just started with a different command (node scripts/process-worker.js) and a much bigger footprint (8 vCPU, 16 GB memory). The task runs the full preprocessing pipeline for a single project and then exits. This keeps the web task responsive during heavy PDF ingest.

The state machine in stepfunctions.tf(blueprintparser-process-blueprint) is what starts cpu-pipeline tasks. It's a straight line: ValidateInput → CPUProcessing → ProcessingComplete with a failure branch. Retries happen on TaskFailed with a 30-second interval and 2.0× backoff, up to 2 attempts. The state machine logs to a CloudWatch log group (/aws/states/blueprintparser-process-blueprint) for debuggability.

YOLO inference runs out-of-band on SageMaker Processing jobs. BP calls sagemaker:CreateProcessingJobfrom the app task with inputs pointing to the project's pages/ prefix in S3 and outputs pointing to yolo-output/. The container image comes from a second ECR repo (beaver_yolo_pipeline) built separately from the app image. The default instance type is ml.g4dn.xlarge, billed per run, which is the exact reason the sagemakerEnabled toggle exists.

S3 is the durability layer. The bucket is blueprintparser-data-{account_id} and the layout is stable:

{dataUrl}/                             // {company_id}/{project_public_id}
├── original.pdf                       // raw upload
├── thumbnail.png                      // 72 DPI cover image
├── pages/
│   ├── page_0001.png                  // 300 DPI display image
│   └── page_0002.png
├── thumbnails/
│   ├── page_0001.png                  // 72 DPI thumbnail
│   └── page_0002.png
├── yolo-output/                       // written by SageMaker
│   ├── page_0001_detections.json
│   └── page_0002_detections.json
└── exports/
    ├── takeoff.csv                    // user-exported CSVs
    └── labels.zip                     // Label Studio exports

Every file under pages/ and thumbnails/ is cached as public, max-age=31536000, immutableso CloudFront holds them forever. The cache-warming pass at the end of preprocessing primes edge locations so the first viewer open is fast. Filenames include the page number so they're effectively content-addressed.

Database storage uses PostgreSQL 16 on a db.t4g.mediumwith 50 GB gp3 that can auto-grow to 200 GB. Backups retained 7 days. Multi-AZ in production. All writes go through Drizzle; the schema lives in src/lib/db/schema.ts.

The full stack is in infrastructure/terraform/. Each file has a single responsibility:

Label Studio runs as a separate ECS task with an EFS volume mounted at /label-studio/data. The ALB routes labelstudio.* to the task; the main app integrates via /api/labeling/* routes. It reads from the same S3 bucket the main app writes to, so round-tripping a project from ingest to labeling and back works without cross-service copying.

Everything in this section is the deployed tier. You can ignore most of it and still run BP: the repo ships a docker-compose.yml that brings up a local PostgreSQL on port 5433 and lets you run npm run devagainst it. Textract, S3, and SageMaker are all gated by env vars — when they're missing, BP falls back to Tesseract for OCR, the local filesystem for images (or a dev-mode S3 emulator if you prefer), and nothing for YOLO. LLM chat still works if you have a Groq free-tier key. The table parsers (img2table, Camelot, TATR) all run locally from scripts/. Bucket Fill runs locally. Auto-QTO runs locally given a parsed schedule. The only hard dependency on AWS is YOLO inference.

[LLM-NAV:mental-model]

BlueprintParser is a graph builder for construction PDFs. It turns a multi-page drawing set into two data axes:

• Horizontal (per page)— each page carries OCR, CSI codes, text annotations, detected tables, a classification, and (optionally) YOLO detections. Stored on the pages row with a large pageIntelligence JSONB.
• Vertical (project-wide) — annotations (YOLO + user markups + takeoff), takeoff_items (count / area / linear), yolo_tags(schedule tag ↔ shape instance), and projectIntelligence (the project-level summary: disciplines, CSI network graph, hub pages).

Everything downstream — the LLM chat, the viewer, takeoff, the CSI spatial heatmap, Auto-QTO — reads from those two shapes. The fastest way to get oriented in the code is: read src/lib/db/schema.ts, src/types/index.ts, and src/lib/processing.ts#processProject().

[LLM-NAV:glossary]

BP's code uses construction-industry terms unapologetically. A model that doesn't know these will miscalibrate what the code is doing. Each card below gives the plain-English definition and the BP surface it appears in.

[LLM-NAV:landmarks]

[LLM-NAV:store-slices]

Every panel, every toolbar button, every canvas overlay reads from one of these slices. Subscribing at the slice level — rather than with a raw useViewerStore(s => s.field) — is how the 1,986-line store doesn't cause cascading re-renders. If you're adding UI that needs state from the store, check this map for the existing slice before creating a new one.

[LLM-NAV:tool-selection]

Section 9 has the full tool grid. This subsection is the selection heuristic: given a user question, which tool should an LLM call first?

[LLM-NAV:signal-valves]

// :131 — windowMatch hardcoded to true (multi-word text coherence not evaluated)
const windowMatch = true;

// :141-142 — two boosts hardcoded to zero
shapeContainBoost: 0,     // not yet produced by matchers; future refinement
objectAdjacencyBoost: 0,  // not yet produced by matchers; future refinement

Translation for a model reasoning about BP's capabilities: tag mapping scores are conservative. Every returned match has passed a pattern + region-weight + scope check, but the adjacency and shape-containment refinements that would let BP surface subtle discrepancies (e.g. "schedule says 12 doors of type D-01 but only 11 appear on plans") are not yet implemented. Don't claim that capability in responses. Point the user at Section 6 and Section 9 if they ask.

[LLM-NAV:flows]

Every feature in BP follows the same shape: user action → API route → DB/S3 write → Zustand store update → re-render. If you're reasoning about how a change would propagate, trace that path. The diagram below shows four of the most common flows side-by-side.

[LLM-NAV:hazards]

[LLM-NAV:extend]

[LLM-NAV:model-behavior]

1. Check the context budget. On Opus you have room to include raw OCR; on Groq or Haiku, rely on structured tools and skip raw OCR unless the question demands it.
2. Reach for the right tool first.The selection table above is the heuristic: for "where are the plumbing fixtures," lookupPagesByIndex is always cheaper than searchPages.
3. Ground every quantitative claim in a tool call. Do not invent counts. Auto-QTO is the source of truth for takeoffs; mapTagsToPages is the source of truth for tag instance counts.
4. Prefer actions over prose. If the user wants to see page 42, call navigateToPage rather than describing it.
5. Respect the signal-valve state.BP does not currently do adjacency-based cross-schedule discrepancy detection (see the warning above). If the user asks "does the door schedule match the plans," answer with what mapTagsToPages returns and note that you cannot detect subtler mismatches yet.

A few routes need extra context beyond the short description:

• POST /api/ai/chat is Server-Sent Events, not request/response. The response stream yields data: lines encoding a sequence of { type: "text_delta" | "tool_call_start" | "tool_call_result" | "done" } events. The client reads them as they arrive. DELETE on the same path clears the scoped conversation history.
• POST /api/yolo/run is the only way to trigger YOLO inference. The request body is { projectId, modelId } and the response returns the SageMaker execution ID. Watch it via GET /api/yolo/status. Results ingest via the webhook.
• POST /api/processing/webhook is the callback surface for Step Functions and SageMaker. Requests are HMAC-SHA256 signed with the PROCESSING_WEBHOOK_SECRET from Secrets Manager. Unsigned or mis-signed requests are rejected.
• POST /api/projects/[id]/map-tags-batchis the heavy-lifter for Auto-QTO. It takes a parsed schedule's tag column, a target YOLO class (or a free-floating-text marker), and runs the mapping across the entire project at once. Expect it to take several seconds for large projects.
• POST /api/bucket-fillreturns a polygon in normalized 0–1 coordinates, not the image space. The viewer converts to canvas coordinates at render time; areaCalc.ts converts to real-world units using the page's scale calibration.
• POST /api/csi/detect is the public entry point to the 3-tier CSI matcher (see Section 04). Accepts a text string in the body, returns an array of matches with codes, descriptions, divisions, trades, and confidence scores.
• /api/demo/* is a parallel, read-only mirror of the project and search routes that does not require auth. These are what power the /demoroute and the docs page's live component demos.

Every endpoint in the catalog above maps to a file under src/app/api/**/route.ts. The Next.js App Router uses directory-based routing, so /api/csi/detect is src/app/api/csi/detect/route.ts and /api/projects/[id]/map-tags-batch is src/app/api/projects/[id]/map-tags-batch/route.ts. Each file exports HTTP method handlers (GET, POST, etc.) and most hand off immediately to helper functions in src/lib/. Handlers are thin by design — the real logic lives in lib/ and is unit-tested.