drawing-agent

Workspace scaffolding for fine-tuning Stable Diffusion XL model using QLoRA to generate patent-ready drawings from patent description and claims text. UPDATE: Deprecated. Contact for questions/access.

View the Project on GitHub

Drawing Agent

SDXL QLoRA Fine-Tuning for Patent Drawings

Generate patent-style technical drawings from text using fine-tuned Stable Diffusion XL with QLoRA. This project provides a complete pipeline for processing USPTO patent data and training a model to generate drawings that comply with USPTO requirements.

Table of Contents

1. Features

2. Quick Start

# 0. Configure Neon connection string (once per shell)
export DATABASE_URL="postgresql://USER:PASS@HOST/dbname"

# 1. Setup directories and database
make dirs
make init-db

# 2. Add USPTO patent data to data/raw/
# Download from: https://bulkdata.uspto.gov/
# Place .zip files containing individual patents in data/raw/

# 3. Index patent files into database
make index

# 4. Build training datasets
make dataset

# 5. Configure accelerate for training
accelerate config
# Save to: configs/accelerate/default_config.yaml

# 6. Train the model
bash scripts/run_finetune.sh

3. Repository Layout

drawing-agent/
├── configs/                      # Configuration files
│   ├── dataset.figures.yaml     # Figures dataset config
│   ├── dataset.yaml             # Claims dataset config
│   ├── training.sdxl-qlora.yaml # Training hyperparameters
│   └── accelerate/              # Accelerate configs
├── data/
│   ├── raw/                     # USPTO bulk downloads (zipped patents)
│   ├── work/                    # Working directory
│   │   ├── extracted/           # Extracted XML and TIFF files
│   │   ├── images/              # Processed figure crops
│   │   └── control/             # Canny edge control maps
│   └── ds/                      # Output JSONL datasets
│       ├── train.jsonl
│       ├── val.jsonl
│       └── test.jsonl
├── scripts/
│   ├── setup_environment.sh     # Environment setup
│   ├── process_uspto.sh         # Process USPTO data
│   ├── run_finetune.sh          # Launch training
│   └── test_setup.sh            # Validate setup
├── src/training/                # Core Python modules
│   ├── index_docs.py            # ZIP indexing and extraction
│   ├── parse_xml.py             # USPTO XML parsing
│   ├── img_norm.py              # Image processing & OCR-based figure detection
│   ├── control.py               # Canny edge control map generation
│   ├── prompt.py                # Structured prompt building
│   ├── build_dataset.py         # Dataset construction from indexed patents
│   ├── load_fts.py              # Full-text search index loader
│   └── schema.sql               # Postgres database schema
├── outputs/                     # Model checkpoints and LoRA adapters
├── Makefile                     # Build automation
├── pyproject.toml               # Python package configuration
└── README.md                    # This file

Note: Patent index stored in DB (Neon.tech). Set DATABASE_URL env var.

4. Makefile Commands

# Database setup
make dirs            # Create required directories
make init-db         # Initialize Neon/Postgres schema (DATABASE_URL must be set)
make index           # Index ZIP bundles into database
make reindex         # Re-extract and re-index with --force

# Dataset building
make train           # Build train.jsonl
make val             # Build val.jsonl
make test            # Build test.jsonl
make dataset         # Build all splits

# Optional utilities
make fts             # Load full-text search index
make check           # Show database statistics
make counts          # Count rows per table
make vacuum          # Run VACUUM (ANALYZE)
make clean-outputs   # Remove generated images and JSONL files
make clean-all       # Clear database tables and remove extracted files

5. Dataset Structure

Each training sample in the JSONL output includes:

{
  "id": "US1234567:page123_FIG_3A.png",
  "image_path": "data/work/images/US1234567_page123_FIG_3A.png",
  "control_path": "data/work/control/US1234567_page123_FIG_3A_canny.png",
  "prompt": "Style: USPTO patent line art...",
  "doc_id": "US1234567",
  "fig_label": "FIG. 3A",
  "bbox": [x1, y1, x2, y2],
  "original_size": [width, height],
  "resize_meta": {
    "orig_width": 1500,
    "orig_height": 1200,
    "resized_width": 2048,
    "resized_height": 1638,
    "offset_x": 0,
    "offset_y": 205,
    "scale": 1.365,
    "canvas_size": [2048, 2048]
  }
}

Key Fields:

6. Structured Prompts

Prompts are hierarchically constructed by prompt.py:

Style: USPTO patent line art, monochrome, 300dpi, white background.
Type: block_diagram

Primary Context:
FIG. 3A illustrates a system architecture showing processor-memory connections.

Secondary Context:  
The system comprises a processor 100, memory 110, and bus 120 for data transfer.

Spatial Relations:
- processor 100 connected to memory 110
- bus 120 between processor 100 and memory 110

Prohibitions: no shading, no color, no text outside reference numerals

Diagram Type Detection: Automatically classified based on text analysis: flowchart, block diagram, mechanical, electrical schematic, graph, perspective, orthographic

7. Configuration

7.1 Dataset (configs/dataset.figures.yaml)

storage:
  processed_dir: data/figures/processed/hf_dataset

schema:
  prompt: string
  figure_no: string
  diagram_type: string
  patent_id: string
  image: Image
  control: Image

splits:
  train: train
  validation: validation
  test: test

image_processing:
  resolution: 2048
  center_crop: true
  random_flip: false

7.2 Training (configs/training.sdxl-qlora.yaml)

model:
  pretrained_model_name_or_path: stabilityai/stable-diffusion-xl-base-1.0
  revision: fp16
  variant: fp16

qlora:
  r: 32                          # LoRA rank
  alpha: 16
  dropout: 0.05
  target_modules: [to_k, to_q, to_v, to_out.0]
  quantization_bits: 4
  compute_dtype: bf16

training:
  resolution: 2048
  train_batch_size: 1
  gradient_accumulation_steps: 8
  num_train_epochs: 3
  checkpointing_steps: 500
  learning_rate: 1.0e-4
  lr_scheduler: cosine_with_restarts

8. Pipeline Overview

8.1 Indexing (index_docs.py)

8.2 XML Parsing (parse_xml.py)

8.3 Image Processing (img_norm.py)

8.4 Control Maps (control.py)

8.5 Prompt Generation (prompt.py)

8.6 Dataset Building (build_dataset.py)

9. Hardware Requirements

Minimum:

Recommended:

Processing Performance:

10. Third-Party Services

10.1 Core Dependencies

10.2 Image Processing

10.3 Data Management

11. Model Inference

After training, generate patent-style drawings using the fine-tuned model:

from diffusers import StableDiffusionXLPipeline
import torch

# Load base model with LoRA adapters
pipeline = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
)
pipeline.load_lora_weights("outputs/sdxl-qlora-run-3/checkpoint-3500/unet_lora")
pipeline.to("cuda")

# Generate patent drawing
prompt = """
Style: USPTO patent line art, monochrome, 300dpi, white background.
Type: block_diagram

Primary Context:
System architecture showing processor-memory connections.

Spatial Relations:
- processor 100 connected to memory 110
- bus 120 between processor 100 and memory 110

Prohibitions: no shading, no color, no text outside reference numerals
"""

image = pipeline(
    prompt,
    num_inference_steps=50,
    guidance_scale=7.5
).images[0]
image.save("patent_drawing.png")

12. Database Schema

The Neon/Postgres database tracks all processed patents:

CREATE TABLE docs (
  doc_id TEXT PRIMARY KEY,
  xml    TEXT NOT NULL,
  tiffs  JSONB NOT NULL
);

CREATE TABLE figures (
  id        TEXT PRIMARY KEY,
  doc_id    TEXT NOT NULL REFERENCES docs(doc_id) ON DELETE CASCADE,
  tiff_path TEXT NOT NULL,
  fig_label TEXT,
  bbox_x1   INTEGER,
  bbox_y1   INTEGER,
  bbox_x2   INTEGER,
  bbox_y2   INTEGER
);

CREATE TABLE text_fts (
  doc_id TEXT NOT NULL REFERENCES docs(doc_id) ON DELETE CASCADE,
  section TEXT NOT NULL,
  content TEXT NOT NULL,
  content_tsv tsvector GENERATED ALWAYS AS (to_tsvector('english', coalesce(content, ''))) STORED,
  PRIMARY KEY (doc_id, section)
);

CREATE INDEX idx_text_fts_content_tsv
  ON text_fts USING GIN (content_tsv);

CREATE OR REPLACE VIEW doc_overview AS
SELECT
  doc_id,
  xml,
  jsonb_array_length(tiffs) AS tiff_count
FROM docs;

13. Troubleshooting

Index fails with “multiple XML files” error:

OCR doesn’t detect figures:

CUDA out of memory during training:

No figures split from multi-figure sheets:

Database connection errors:

14. Dataset Inspection

Query the database to inspect indexed patents:

# Count documents, figures, and text entries
python src/training/db_admin.py counts

# Show connection and session details
python src/training/db_admin.py info

# Run an ad-hoc SQL query (DATABASE_URL must be exported)
python - <<'PY'
import os
import psycopg

dsn = os.environ["DATABASE_URL"]
with psycopg.connect(dsn) as conn:
    with conn.cursor() as cur:
        cur.execute("""
            SELECT doc_id, jsonb_array_length(tiffs) AS pages
            FROM docs
            ORDER BY doc_id
            LIMIT 10
        """)
        for doc_id, pages in cur.fetchall():
            print(doc_id, pages)
PY

View generated JSONL samples:

# Inspect first training sample
head -n 1 data/ds/train.jsonl | jq '.'

# Count samples per split
wc -l data/ds/*.jsonl

15. Project Status

DEPRECATED: This project is now inactive. Further development has been folded into a commercial project. Last checkpoints available via GCP Storage.

16. License

This repository is publicly viewable for portfolio purposes only. The code is proprietary. Copyright © 2026 Phaethon Order LLC. All rights reserved. See LICENSE for terms.

17. Resources