← Projects

Solidity Copilot

In progress
AISolidityEVMSecurityEvaluation

Small contract tasks show whether an AI can write Solidity that behaves as promised. The project checks the output independently and treats a compile as one small signal.

Technical An isolated evaluator runs pinned Foundry compilation, hidden behavioral tests, interface and invariant checks, and security triage against constrained contract tasks. B0 through B2 prompt and retrieval baselines precede any bounded LoRA or QLoRA experiment.

Source
GitHub →
Demos

Short and full walkthroughs play inline on the project page.

Short demo →Full demo →
Solidity Copilot demo

Case Study

Problem

I want a trustworthy way to measure whether a model can turn a precise contract specification into correct Solidity, without pretending that compiling equals useful or that fine-tuning is the first move.

What I designed and implemented

  • Constrained, machine-readable contract tasks with optional reference-contract context
  • Model outputs: Solidity sources, Foundry tests for claimed behavior, and assumptions where the spec is ambiguous
  • Isolated evaluator harness the model never sees: pinned toolchain compile, hidden tests, interface and invariant checks, static security triage
  • Lineage-aware train/val/test splits with immutable provenance, hashes, and license review

Architecture

  • B0: structured prompt only
  • B1: structured prompt plus fixed few-shot examples
  • B2: structured prompt plus train-only retrieval and few-shot examples
  • Bounded LoRA or QLoRA only if baselines show a repeatable, data-addressable Solidity/Foundry failure
  • Hidden tests and held-out reference implementations stay out of prompts, retrieval, and training data

Engineering decisions

  • Evaluation and provenance before fine-tuning
  • LoRA can teach recurring patterns and constraint-following; it does not fix ambiguous specs, missing API knowledge, bad retrieval, or weak evaluation
  • Split by contract-family lineage, not individual files, to keep forks and near-duplicates from leaking across sets
  • A model that merely compiles is not a passing grade

Verification and testing

  • Pinned Solidity/Foundry compile
  • Evaluator-owned hidden behavioral tests
  • Required interfaces, access controls, events, custom errors, and invariants
  • Static security triage for new serious findings
  • Reproducible media kit under public/ with smoke-report screenshots and walkthrough clips

Limitations and what is simulated

  • Educational and experimental only
  • No private keys, no mainnet deployment, no audited or production-ready claims
  • No unlicensed or provenance-free training corpus

Production differences

  • If the harness is honest, it becomes an AI evaluation and adaptation system for verifiable financial-infrastructure code.

Video demos

Short demo

Full demo