Formal Code Quality
Benchmark preview · September 2026
Introduction
We are building a benchmark to assess how well AI systems produce correct, reusable formal mathematical code in Lean. The benchmark examines the decisions that make a contribution useful within an existing library: how it builds on established definitions and results, structures its arguments, and exposes mathematical constructions for subsequent use.
Formal mathematics is a cumulative effort. A formalization can establish a theorem while leaving its underlying constructions difficult to understand or apply elsewhere. A useful library contribution connects those constructions to existing theory, provides appropriate interfaces, and avoids unnecessary duplication. These choices affect how much work is required to use, maintain, and extend the mathematics.
Our benchmark studies these differences through tasks developed with Mathlib contributors from their previous contributions. Models receive a mathematical goal and an existing library context, and choose how to implement the requested development. Evaluation combines correctness checks with task-specific criteria for proof structure, library reuse, interfaces, documentation, and scope. The aim is to assess concrete choices that support reuse and maintenance while allowing different valid mathematical approaches.
This preview presents one task and two model attempts. Both attempts pass the task-specific mathematical checks, but differ on selected quality criteria. We use their implementations to illustrate what those criteria detect and why the differences matter. The examples demonstrate the evaluation approach; they do not establish an overall ranking of the models.
What we evaluate
Our evaluation covers three areas:
- Mechanical checks: Does the contribution build, pass required linters, and avoid proof placeholders and disallowed axioms?
- Mathematical correctness: Does it faithfully formalize and establish all the requested mathematics, without weakening the claim or assuming the result?
- Code quality: Does it fit the library and support future use and maintenance?
Code quality is assessed along six dimensions:
- Library reuse: Does the solution build on existing definitions, lemmas, and constructions without unnecessary duplication?
- API design: Does it expose useful constructions and supporting lemmas at an appropriate level of generality, while keeping implementation details private?
- Naming and notation: Are names accurate, discoverable, and consistent with library conventions?
- Documentation: Does the documentation explain the mathematics accurately and credit relevant sources?
- Proof robustness and readability: Are proofs clearly organized and maintainable, without fragile reliance on implementation details?
- Scope and organization: Are changes focused on the task, with declarations in appropriate modules and relevant imports?
Each task translates these dimensions into concrete criteria, allowing different valid mathematical approaches.
| Area | Evaluation method | What is checked |
|---|---|---|
| Mechanical checks | Build, linters, and axiom audit | Compilation, repository conventions, and proof integrity. |
| Mathematical correctness | Task-specific Lean checks and semantic review | Required results, intended definitions, assumptions, and edge cases. |
| Code quality | LLM review against task-specific criteria, supported by mechanical checks where applicable | Reuse, interfaces, names, documentation, proofs, and organization. |
Each criterion is either a blocker, which must be satisfied for a solution to pass, or a non-blocker, which contributes to its quality score. Quality requirements can also be blockers when they are essential to the task.
Grounded in real contributions
Our first task is based on a merged Mathlib contribution. It pairs a mathematical goal and repository conventions with concrete criteria for the quality of the resulting library contribution. We are working with contributors to expand the collection.
The prompt leaves the model free to choose its approach. The rubric allows alternative valid constructions; each quality finding must identify the relevant code and explain why the issue matters.
One task: the Cantor set
This task is based on a contribution by Vasilii Nesterov. It asks the model to connect two descriptions of the same mathematical space: the ternary Cantor set and infinite sequences of Boolean values.
Full task prompt
Description
Prove that Real.ofDigits is continuous as a map from digit sequences ℕ → Fin b to ℝ for every base b.
Establish a homeomorphism between the ternary Cantor set and ℕ → Bool.
Repository guidelines
Follow Mathlib's naming, documentation, and style conventions. Use lake build Mathlib.Import.Path to build a modified module, and use lake test to run the full test suite.
The existing library already provides much of the machinery. Solving the task involves finding those pieces and composing them into a clear implementation.
Two attempts, viewed through the rubric
Both attempts pass the task-specific checks for continuity and the homeomorphism. Their treatment of the existing library reveals a difference that those checks alone do not capture.
| Criterion | Rubric | Luna | Sol |
|---|---|---|---|
Continuity of Real.ofDigits, for every base | Mathematical correctness | Pass | Pass |
| Cantor set–Boolean stream homeomorphism | Mathematical correctness | Pass | Pass |
| Focused source footprint | Scope and organization | Pass | Pass |
| Discoverable public names | Naming and notation | Pass | Pass |
| Reuse the existing Cantor-to-binary inverse | Library reuse | Fail | Pass |
| Construct the homeomorphism as data | Proof robustness and readability | Fail | Fail |
These are selected checks on two individual attempts. They illustrate the evaluation; they are not aggregate model results.
A closer look at library reuse
The module already defines cantorToBinary, which maps a Cantor-set point to a Boolean stream. The reuse criterion asks whether the solution uses that existing inverse when implementing the same operation.
GPT-5.6 Luna Reuse criterion: fail
Luna reconstructs the decoding operation locally: it obtains the ternary digits and tests whether each digit equals two.
let a : ℕ → Bool := fun n =>
decide ((cantorToTernary x.1).get n = 2)
This adds another implementation of a conversion the library already provides. Reusing the existing definition would make the connection explicit and avoid maintaining duplicate decoding logic.
Inspect Luna’s full patch
Original model output, including both modified modules. Download patch
diff --git a/Mathlib/Analysis/Real/OfDigits.lean b/Mathlib/Analysis/Real/OfDigits.lean
index f206d72c68..88342f00e0 100644
--- a/Mathlib/Analysis/Real/OfDigits.lean
+++ b/Mathlib/Analysis/Real/OfDigits.lean
@@ -101,6 +101,34 @@ theorem abs_ofDigits_sub_ofDigits_le {b : ℕ} {x y : ℕ → Fin b} {n : ℕ}
(ofDigits_nonneg _) (ofDigits_le_one _)
simp
+theorem continuous_ofDigits {b : ℕ} :
+ Continuous (ofDigits : (ℕ → Fin b) → ℝ) := by
+ rw [continuous_iff_continuousAt]
+ intro x
+ change Filter.Tendsto ofDigits (nhds x) (nhds (ofDigits x))
+ rw [Metric.tendsto_nhds]
+ intro ε hε
+ obtain rfl | hb := (Nat.one_le_of_lt (b_pos x)).eq_or_lt
+ · filter_upwards [] with y
+ have hy : y = x := Subsingleton.elim y x
+ subst y
+ simp [hε]
+ have hpow : Filter.Tendsto (fun n : ℕ => ((b : ℝ) ^ n)⁻¹) Filter.atTop (nhds 0) := by
+ simpa [inv_pow] using
+ (tendsto_pow_atTop_nhds_zero_of_abs_lt_one (r := (b : ℝ)⁻¹) (by
+ rw [abs_of_nonneg]
+ · have hb' : (1 : ℝ) < b := by exact_mod_cast hb
+ exact (inv_lt_one₀ (by positivity)).2 hb'
+ positivity))
+ obtain ⟨n, hn⟩ := (Metric.tendsto_atTop.1 hpow ε hε)
+ have hnhds : (Set.Iio n).pi (fun i => {x i}) ∈ nhds x :=
+ set_pi_mem_nhds (Set.finite_Iio n) fun i hi => by
+ exact mem_nhds_iff.2 ⟨{x i}, subset_rfl, isOpen_discrete _, rfl⟩
+ filter_upwards [hnhds] with y hy
+ exact lt_of_le_of_lt (abs_ofDigits_sub_ofDigits_le (fun i hi => by
+ rw [Set.mem_pi] at hy
+ exact hy i hi) (n := n)) (by simpa using hn n le_rfl)
+
/-- Converts a real number `x` from the interval `[0, 1)` into sequence of
its digits in base `b`. -/
noncomputable def digits (x : ℝ) (b : ℕ) [NeZero b] : ℕ → Fin b :=
diff --git a/Mathlib/Topology/Instances/CantorSet.lean b/Mathlib/Topology/Instances/CantorSet.lean
index a1a1324c55..4a53e15de8 100644
--- a/Mathlib/Topology/Instances/CantorSet.lean
+++ b/Mathlib/Topology/Instances/CantorSet.lean
@@ -312,4 +312,44 @@ theorem cantorSet_eq_zero_two_ofDigits :
· rw [← ha.right]
exact ofDigits_zero_two_sequence_mem_cantorSet ha.left
+/-- The ternary Cantor set is homeomorphic to the space of Boolean sequences. -/
+noncomputable def cantorSetHomeomorphNatBool : cantorSet ≃ₜ (ℕ → Bool) := by
+ let ternary (a : ℕ → Bool) : ℕ → Fin 3 := fun n => if a n then 2 else 0
+ let f : (ℕ → Bool) → cantorSet := fun a =>
+ ⟨ofDigits (ternary a), ofDigits_zero_two_sequence_mem_cantorSet (by
+ intro n
+ cases h : a n <;> simp [ternary, h])⟩
+ have hternary : Continuous ternary := by
+ apply continuous_pi
+ intro n
+ simpa [ternary, Function.comp_def] using
+ ((continuous_of_discreteTopology :
+ Continuous (fun z : Bool => if z then (2 : Fin 3) else 0)).comp (continuous_apply n))
+ have hf : Continuous f := by
+ apply Continuous.subtype_mk
+ exact (Real.continuous_ofDigits).comp hternary
+ have hinj : Function.Injective f := by
+ intro a b hab
+ apply funext
+ intro n
+ have h : ternary a = ternary b :=
+ ofDigits_zero_two_sequence_unique
+ (by intro n; cases h : a n <;> simp [ternary, h])
+ (by intro n; cases h : b n <;> simp [ternary, h])
+ (congrArg Subtype.val hab)
+ have := congrFun h n
+ cases ha : a n <;> cases hb : b n <;> simp [ternary, ha, hb] at this ⊢
+ have hsurj : Function.Surjective f := by
+ intro x
+ let a : ℕ → Bool := fun n => decide ((cantorToTernary x.1).get n = 2)
+ have ha : ternary a = (cantorToTernary x.1).get := by
+ funext n
+ have hn := cantorToTernary_ne_one (x := x.1) (n := n)
+ generalize hd : (cantorToTernary x.1).get n = d at hn ⊢
+ fin_cases d <;> simp [a, ternary, hd] at hn ⊢
+ refine ⟨a, ?_⟩
+ apply Subtype.ext
+ simpa [f, a, ha] using ofDigits_cantorToTernary x.2
+ exact ((hf.isClosedEmbedding hinj).toHomeomorphOfSurjective hsurj).symm
+
end ternary02
GPT-5.6 Sol Reuse criterion: pass
Sol defines the inverse using the existing conversion and its stream accessor.
let toBinary : cantorSet → ℕ → Bool :=
fun x ↦ (cantorToBinary x).get
The implementation directly expresses its relationship to the library’s established API. Subsequent proofs use this inverse without introducing a separate decoding definition.
Inspect Sol’s full patch
Original model output, including both modified modules. Download patch
diff --git a/Mathlib/Analysis/Real/OfDigits.lean b/Mathlib/Analysis/Real/OfDigits.lean
index f206d72c68..03d3f34dc5 100644
--- a/Mathlib/Analysis/Real/OfDigits.lean
+++ b/Mathlib/Analysis/Real/OfDigits.lean
@@ -4,6 +4,7 @@ Released under Apache 2.0 license as described in the file LICENSE.
Authors: Vasilii Nesterov
-/
import Mathlib.Algebra.Order.Floor.Semifield
+import Mathlib.Analysis.Normed.Group.FunctionSeries
import Mathlib.Analysis.SpecificLimits.Normed
import Mathlib.Tactic.Rify
@@ -101,6 +102,35 @@ theorem abs_ofDigits_sub_ofDigits_le {b : ℕ} {x y : ℕ → Fin b} {n : ℕ}
(ofDigits_nonneg _) (ofDigits_le_one _)
simp
+/-- Interpreting a sequence of digits as a real number is continuous. -/
+theorem continuous_ofDigits (b : ℕ) : Continuous (ofDigits : (ℕ → Fin b) → ℝ) := by
+ rcases b with _ | b
+ · letI : IsEmpty (ℕ → Fin 0) := ⟨fun x ↦ Fin.elim0 (x 0)⟩
+ rw [continuous_def]
+ intro s hs
+ convert isOpen_empty
+ exact Set.eq_empty_iff_forall_notMem.mpr fun x _ ↦ isEmptyElim x
+ · rcases b with _ | b
+ · convert (continuous_const : Continuous fun _ : ℕ → Fin 1 ↦ (0 : ℝ)) using 1
+ funext x
+ simp [ofDigits, ofDigitsTerm]
+ change Continuous fun x : ℕ → Fin (b + 2) ↦ ∑' n, ofDigitsTerm x n
+ apply continuous_tsum (u := fun n ↦ (b + 1 : ℝ) * ((b + 2 : ℝ) ^ (n + 1))⁻¹)
+ · intro n
+ unfold ofDigitsTerm
+ fun_prop
+ · simp_rw [pow_succ', mul_inv, ← inv_pow, ← mul_assoc]
+ exact Summable.mul_left _ <| summable_geometric_of_lt_one (by positivity) (by
+ apply (inv_lt_one₀ (by positivity)).mpr
+ have : 0 ≤ (b : ℝ) := by positivity
+ linarith)
+ · intro n x
+ rw [Real.norm_eq_abs, abs_of_nonneg ofDigitsTerm_nonneg]
+ have h := ofDigitsTerm_le (digits := x) (n := n)
+ norm_num [Nat.cast_add, Nat.cast_ofNat] at h ⊢
+ convert h using 1
+ ring
+
/-- Converts a real number `x` from the interval `[0, 1)` into sequence of
its digits in base `b`. -/
noncomputable def digits (x : ℝ) (b : ℕ) [NeZero b] : ℕ → Fin b :=
diff --git a/Mathlib/Topology/Instances/CantorSet.lean b/Mathlib/Topology/Instances/CantorSet.lean
index a1a1324c55..d5f35045aa 100644
--- a/Mathlib/Topology/Instances/CantorSet.lean
+++ b/Mathlib/Topology/Instances/CantorSet.lean
@@ -29,6 +29,8 @@ This file defines the Cantor ternary set and proves a few properties.
* `ofDigits_zero_two_sequence_mem_cantorSet`: any such sequence corresponds to a number
in the Cantor set.
* `ofDigits_zero_two_sequence_unique`: such a representation is unique.
+* `cantorSetHomeomorph`: the ternary Cantor set is homeomorphic to the space of infinite
+ Boolean sequences.
-/
/-- The order `n` pre-Cantor set, defined starting from `[0, 1]` and successively removing the
@@ -312,4 +314,41 @@ theorem cantorSet_eq_zero_two_ofDigits :
· rw [← ha.right]
exact ofDigits_zero_two_sequence_mem_cantorSet ha.left
+/-- The ternary Cantor set is homeomorphic to the space of infinite Boolean sequences. -/
+noncomputable def cantorSetHomeomorph : cantorSet ≃ₜ (ℕ → Bool) := by
+ let ternary : (ℕ → Bool) → ℕ → Fin 3 := fun a n ↦ if a n then 2 else 0
+ have ternary_ne_one (a : ℕ → Bool) (n : ℕ) : ternary a n ≠ 1 := by
+ simp only [ternary]
+ split <;> simp_all
+ let toCantor : (ℕ → Bool) → cantorSet := fun a ↦
+ ⟨ofDigits (ternary a), ofDigits_zero_two_sequence_mem_cantorSet (ternary_ne_one a)⟩
+ let toBinary : cantorSet → ℕ → Bool := fun x ↦ (cantorToBinary x).get
+ have toCantor_toBinary (x : cantorSet) : toCantor (toBinary x) = x := by
+ apply Subtype.ext
+ change ofDigits (ternary (toBinary x)) = x
+ rw [show ternary (toBinary x) = (cantorToTernary x).get by
+ funext n
+ simp [ternary, toBinary, cantorToTernary, Stream'.get_map]]
+ exact ofDigits_cantorToTernary x.property
+ have toBinary_toCantor (a : ℕ → Bool) : toBinary (toCantor a) = a := by
+ have hternary : (cantorToTernary (toCantor a)).get = ternary a :=
+ ofDigits_zero_two_sequence_unique (fun _ ↦ cantorToTernary_ne_one)
+ (ternary_ne_one a) (by
+ rw [ofDigits_cantorToTernary (toCantor a).property])
+ funext n
+ have hn := congrFun hternary n
+ simp only [cantorToTernary, Stream'.get_map, ternary] at hn
+ grind
+ let e : (ℕ → Bool) ≃ cantorSet :=
+ { toFun := toCantor
+ invFun := toBinary
+ left_inv := toBinary_toCantor
+ right_inv := toCantor_toBinary }
+ have he : Continuous e := by
+ apply Continuous.subtype_mk
+ exact (Real.continuous_ofDigits 3).comp <| continuous_pi fun n ↦ by
+ exact (continuous_of_discreteTopology : Continuous fun b : Bool ↦
+ if b then (2 : Fin 3) else 0).comp (continuous_apply n)
+ exact (e.toHomeomorphOfContinuousClosed he he.isClosedMap).symm
+
end ternary02
What this reuse criterion requires
The judge checks for unnecessary reconstruction of the existing inverse and must identify the relevant code and maintenance cost. A genuinely different valid homeomorphism may have a different inverse; the criterion does not require every solution to reference cantorToBinary.
A closer look at construction readability
A homeomorphism contains maps in both directions and proofs that they are inverse and continuous. This task’s construction criterion asks for those data fields to be explicit, with tactic proofs confined to proof fields. The aim is to make the maps and their supporting proofs easy to locate and inspect separately.
Both attempts assemble the final homeomorphism inside a top-level tactic block, interleaving local constructions and proofs before returning the result. Sol makes an intermediate equivalence explicit, but neither attempt uses the required form for the final definition. Both therefore fail this six-point criterion under the preview rubric.
This is a task-specific readability requirement. A top-level tactic block does not by itself establish that a definition is mathematically incorrect or fragile; both attempts pass the separate homeomorphism check.
Run settings
One attempt per model is shown, using the same task prompt and pinned Mathlib starting point. Luna used medium reasoning effort; Sol used high. Both ran through Codex CLI with network access disabled and a 60-minute task limit.
Recorded model runtimes were 14 minutes 51 seconds for Luna and 19 minutes 33 seconds for Sol. Prompt-based criteria used GPT-5.6 Luna at medium reasoning effort, with one judge vote per criterion.
Lean: v4.25.0-rc2
Mathlib base: c7f32eaa9fc2c319f13a75c630ef9cf3752dff89