Skip to content

Generator Contract Specification

Status: Canonical Reference
Scope: All generator files in generators/
Last Updated: January 2, 2026 15:35:14
Created: December 12, 2025 15:38:10

This document defines the contract for test case generator files. Generators enable stress testing, edge case discovery, and reproducible test generation.

Table of Contents

File Structure

Naming Convention

generators/{problem_id}_{slug}.py
Component Format Example
problem_id 4-digit zero-padded LeetCode ID 0001, 0004, 0051
slug snake_case problem name two_sum, median_of_two_sorted_arrays

Examples: - generators/0001_two_sum.py - generators/0004_median_of_two_sorted_arrays.py - generators/0051_n_queens.py

Required Elements

Every generator file MUST contain:

Element Required Description
generate() function βœ… Main entry point for test generation
Docstring with constraints βœ… LeetCode constraints documentation
Edge cases βœ… Known edge cases yielded first

Optional Elements

Element Optional Description
generate_for_complexity() β­• For time complexity estimation
Helper functions β­• Internal _generate_case() etc.
Custom generators β­• generate_all_sizes() etc.

generate() Function

Function Signature

def generate(count: int = 10, seed: Optional[int] = None) -> Iterator[str]:
    """
    Generate random test case inputs.

    Args:
        count: Number of test cases to generate
        seed: Random seed for reproducibility (optional)

    Yields:
        str: Test input in the same format as .in files
    """

Contract Rules

Rule Requirement Rationale
Yield format Must match .in file format Runner passes to solve() via stdin
Reproducibility Same seed β†’ same output Enables failure reproduction
Edge cases first Yield known edge cases before random Catch corner-case bugs early
Constraint compliance Respect LeetCode constraints Ensure valid test cases
JUDGE_FUNC required Solution must have JUDGE_FUNC No .out file for generated cases

πŸ“– See JUDGE_FUNC Specification for validation details.

Minimal Example

# generators/0001_two_sum.py
import json
import random
from typing import Iterator, Optional

def generate(count: int = 10, seed: Optional[int] = None) -> Iterator[str]:
    """Generate test cases for Two Sum."""
    if seed is not None:
        random.seed(seed)

    # Edge cases first (using canonical JSON format)
    edge_cases = [
        ([2, 7, 11, 15], 9),   # Classic example
        ([3, 2, 4], 6),        # Answer not first element
        ([3, 3], 6),           # Duplicate values
    ]

    for nums, target in edge_cases:
        yield f"{json.dumps(nums, separators=(',', ':'))}\n{target}"
        count -= 1
        if count <= 0:
            return

    # Random cases
    for _ in range(count):
        yield _generate_case()

def _generate_case() -> str:
    size = random.randint(2, 5000)
    nums = [random.randint(-10**6, 10**6) for _ in range(size)]
    i, j = random.sample(range(size), 2)
    target = nums[i] + nums[j]
    return f"{json.dumps(nums, separators=(',', ':'))}\n{target}"

Generator Design Patterns

Standard Template

# generators/{problem_id}_{slug}.py
"""
Test Case Generator for Problem {ID} - {Title}

LeetCode Constraints:
- {constraint_1}
- {constraint_2}
- ...

Time Complexity: O(?)
"""
import random
from typing import Iterator, Optional


# ============================================
# Random Test Generation (for functional testing)
# ============================================

def generate(count: int = 10, seed: Optional[int] = None) -> Iterator[str]:
    """
    Generate random test case inputs.

    Args:
        count: Number of test cases to generate
        seed: Random seed for reproducibility (optional)

    Yields:
        str: Test input in the same format as .in files
    """
    if seed is not None:
        random.seed(seed)

    # Edge cases first
    edge_cases = [
        # Known important test cases
    ]

    for edge in edge_cases:
        yield edge
        count -= 1
        if count <= 0:
            return

    # Random cases
    for _ in range(count):
        yield _generate_case()


def _generate_case() -> str:
    """Generate a single random test case."""
    # Implementation here
    pass


# ============================================
# Complexity Estimation (controlled size)
# ============================================

def generate_for_complexity(n: int) -> str:
    """
    Generate test case with specific input size for complexity estimation.

    Args:
        n: Target input size

    Returns:
        str: Test input with size approximately n
    """
    pass

Edge Case Design

Edge cases should cover:

Category Examples
Boundary values Min/max constraints, empty inputs
Special cases Single element, all same values
Negative cases Negative numbers, zero
Classic examples LeetCode example inputs

Example (Median of Two Sorted Arrays):

import json

# Store as data structures, not strings
edge_cases = [
    ([], [1]),                    # nums1 is empty
    ([1], []),                    # nums2 is empty
    ([1, 3], [2]),                # Classic odd total length
    ([1, 2], [3, 4]),             # Classic even total length
    ([-5, -3, -1], [2, 4, 6]),    # Negative and positive
    ([1], [1]),                   # Same single element
]

for nums1, nums2 in edge_cases:
    yield f"{json.dumps(nums1, separators=(',', ':'))}\n{json.dumps(nums2, separators=(',', ':'))}"

Guaranteed Valid Input

For problems requiring valid solutions exist, ensure generated inputs are solvable:

def _generate_case(size: int) -> str:
    """
    Generate a Two Sum case with guaranteed solution.

    Strategy:
    1. Generate random array
    2. Pick two random indices
    3. Set target = nums[i] + nums[j]
    """
    nums = [random.randint(-10**6, 10**6) for _ in range(size)]
    i, j = random.sample(range(size), 2)
    target = nums[i] + nums[j]  # Guaranteed to have solution

    return f"{','.join(map(str, nums))}\n{target}"

Weighted Random Distribution

For more thorough testing, weight towards challenging cases:

def generate(count: int = 10, seed: Optional[int] = None) -> Iterator[str]:
    # ...

    for _ in range(count):
        # Weight towards larger n (more thorough testing)
        n = random.choices(
            population=range(1, 10),
            weights=[1, 1, 2, 3, 4, 5, 6, 7, 8],  # Higher weight for larger
            k=1
        )[0]
        yield str(n)

Complexity Estimation Generator

Function Signature

def generate_for_complexity(n: int) -> str:
    """
    Generate test case with specific input size for complexity estimation.

    Args:
        n: Target input size

    Returns:
        str: Test input with size approximately n
    """

Purpose

The --estimate flag uses this function to: 1. Generate test cases of increasing sizes 2. Measure execution time for each size 3. Fit curve to estimate Big-O complexity

πŸ“– See Test Runner Β§ Complexity Estimation for usage.

Example

def generate_for_complexity(n: int) -> str:
    """
    Generate test case with specific input size.

    For Two Sum:
    - n is the length of nums array
    - Expected complexity: O(n) with hash map
    """
    n = max(2, n)  # Ensure minimum valid size
    return _generate_case(n)

Size Semantics

Define what "n" means for your problem:

Problem Type n Meaning
Array problems Array length
String problems String length
Two-array problems Total elements (m + n)
Matrix problems Total cells (rows Γ— cols)
Graph problems Number of nodes or edges

Input Format Specifications

Format Rules (Canonical JSON)

Rule Requirement
1 line = 1 parameter Follow function signature order
JSON literal Each line is a valid JSON value
No spaces after , [1,2,3] not [1, 2, 3]
Double quotes only "abc" not 'abc'
Lowercase booleans true/false not True/False

πŸ“– See Test File Format for complete format specification.

Common Formats

Single array: 

# One parameter: nums
"[2,7,11,15]"

Array + target: 

# Two parameters: nums (line 1), target (line 2)
"[2,7,11,15]\n9"

Two arrays: 

# Two parameters: nums1 (line 1), nums2 (line 2)
"[1,3]\n[2]"

Matrix (grid): 

# One parameter: 2D array as single line
"[[1,2,3],[4,5,6],[7,8,9]]"

String parameter: 

# String must be JSON double-quoted
"\"hello\""

Single integer: 

# Plain number
"4"

Using json.dumps for Serialization

import json

def _generate_case() -> str:
    nums = [3, 2, 2, 3]
    val = 3
    # Use separators to avoid spaces
    return f"{json.dumps(nums, separators=(',', ':'))}\n{val}"
    # Output: "[3,2,2,3]\n3"

⚠️ Avoid manual string formatting: 

# Wrong: has spaces
f"{nums}"  # -> "[1, 2, 3]"

# Correct: use json.dumps
json.dumps(nums, separators=(',', ':'))  # -> "[1,2,3]"

JUDGE_FUNC Requirement

Why JUDGE_FUNC is Required

Generated test cases have no expected output (.out file). The solution MUST validate correctness using JUDGE_FUNC:

Generator β†’ Input only β†’ Solution β†’ Output β†’ JUDGE_FUNC validates

πŸ“– See JUDGE_FUNC Specification for complete documentation.

Generator-Specific Considerations

When JUDGE_FUNC is used with generators (judge-only mode):

Parameter Value Implication
actual Solution output Parsed via ast.literal_eval() if valid
expected None No .out file exists
input_data Raw input string Use to validate correctness

The JUDGE_FUNC MUST be able to validate using only actual and input_data when expected is None.

Example Pattern

def judge(actual, expected, input_data: str) -> bool:
    # Parse input to understand problem constraints
    n = int(input_data.strip())

    # Validate actual output against problem requirements
    if not _is_valid_output(actual, n):
        return False

    # For judge-only mode: use known answers or algorithmic validation
    if expected is None:
        return _validate_without_expected(actual, n)

    # For static tests: compare with expected
    return actual == expected

JUDGE_FUNC = judge

Running Generated Tests

Command Line Usage

# Static tests + N generated tests
python runner/test_runner.py {problem} --generate N

# Only generated tests (skip static tests)
python runner/test_runner.py {problem} --generate-only N

# Reproducible with seed
python runner/test_runner.py {problem} --generate N --seed 12345

# Save failing cases to tests/
python runner/test_runner.py {problem} --generate N --save-failed

# Complexity estimation
python runner/test_runner.py {problem} --estimate

πŸ“– See Test Runner Specification for full CLI reference.

Output Format

============================================================
πŸ§ͺ Test Results: 0001_two_sum
============================================================

πŸ“ Static Tests:
0001_two_sum_1: βœ… PASS [judge]          0.12ms
0001_two_sum_2: βœ… PASS [judge]          0.08ms

🎲 Generated Tests (seed=12345):
gen_1: βœ… PASS [judge-only]              0.45ms
gen_2: βœ… PASS [judge-only]              0.52ms
gen_3: ❌ FAIL [judge-only]              0.38ms
   β”Œβ”€ Input ─────────────────────────────────
   β”‚ 5,3,8,1,2
   β”‚ 11
   └─────────────────────────────────────────

πŸ’‘ To reproduce: python runner/test_runner.py 0001 --generate 3 --seed 12345
============================================================

Failure Reproduction

When a generated test fails:

  1. Re-run with same seed: 

    python runner/test_runner.py 0001 --generate 10 --seed 12345

  2. Save failed case: 

    python runner/test_runner.py 0001 --generate 10 --save-failed
# Creates: tests/0001_failed_1.in

  3. Debug specific case: 

    python runner/case_runner.py 0001 failed_1

Best Practices

Generator Checklist

  • Docstring with LeetCode constraints
  • seed parameter for reproducibility
  • Edge cases yielded first
  • Random cases respect constraints
  • Input format matches .in files
  • Solution has JUDGE_FUNC defined
  • generate_for_complexity() if using --estimate

Performance Considerations

Consideration Recommendation
Generation speed Keep generators fast (< 1ms per case)
Constraint limits Use LeetCode max constraints for stress tests
Practical limits Don't exceed O(N!) or exponential complexity bounds

Testing Your Generator

# Manual test
from generators.{problem} import generate

for i, test_input in enumerate(generate(count=5, seed=42)):
    print(f"--- Case {i+1} ---")
    print(test_input)
    print()

Quick Reference

Generator Template

# generators/{problem_id}_{slug}.py
"""
Test Case Generator for Problem {ID} - {Title}

LeetCode Constraints:
- {constraints}
"""
import json
import random
from typing import Iterator, Optional


def generate(count: int = 10, seed: Optional[int] = None) -> Iterator[str]:
    if seed is not None:
        random.seed(seed)

    # Edge cases (store as data structures)
    edge_cases = [
        ([1, 2, 3], 4),  # example: (nums, target)
    ]
    for nums, target in edge_cases:
        yield f"{json.dumps(nums, separators=(',', ':'))}\n{target}"
        count -= 1
        if count <= 0:
            return

    # Random cases
    for _ in range(count):
        yield _generate_case()


def _generate_case() -> str:
    # Generate valid input using json.dumps
    nums = [random.randint(1, 100) for _ in range(10)]
    target = random.randint(1, 200)
    return f"{json.dumps(nums, separators=(',', ':'))}\n{target}"


def generate_for_complexity(n: int) -> str:
    return _generate_case_with_size(n)

CLI Reference

# Run with generation
python runner/test_runner.py {problem} --generate N
python runner/test_runner.py {problem} --generate-only N
python runner/test_runner.py {problem} --generate N --seed S
python runner/test_runner.py {problem} --generate N --save-failed

# Complexity estimation
python runner/test_runner.py {problem} --estimate

πŸ“– See Test Runner Specification for full CLI reference.

Document Content
Test File Format Canonical .in/.out format specification
Solution Contract SOLUTIONS, JUDGE_FUNC, COMPARE_MODE, file structure
Test Runner Specification CLI options, output format, troubleshooting
Architecture Migration Polymorphic pattern migration guide