PolyBench — Teaching and Evaluating LLMs to Reason About Polymer Design

The problem

Polymer design is multi-constraint by nature

A single design prompt jointly specifies property targets, available reactants, and functional constraints. A PolyBench-trained model has to decompose the prompt and emit a valid SMILES repeat unit that satisfies all of them at once.

Prompt

Design a polymer with a glass transition temperature (T_g) between 80–100 °C using CC(CCC(=O)O)C(=O)O and CC(C)(O)c1c(O)cccc1Br as monomers, while maintaining chain flexibility for processing. The polymer must retain bromine functionality for post-polymerization modification.

Property target Reactants Functional constraint

Model reasoning

Constraints: T_g 80–100 °C, flexibility, retain –Br.
Diacid → flexibility; aromatic diol → rigidity.
Balance segments to hit T_g while preserving mobility.
Position Br away from reactive sites.
Condensation via –OH groups forms ester linkages.

Answer. Alternating units of the brominated aromatic ring connected via ester linkages to flexible aliphatic segments. *Oc1cccc(Br)c1C(C)(C)OC(=O)CCC(C)C(*)=O

Figure 1. An example from PolyBench. The property target, reactants, and functional constraint must be satisfied jointly — this is what makes polymer design tasks compositional rather than single-fact lookups.

Construction

How PolyBench is built

Four stages, from raw polymer corpora to (instruction, question, CoT, solution) tuples. Splits are applied to polymers before task generation, so the test set contains unseen polymers, properties, and task variations.

Knowledge base

Aggregate open-source experimental and synthetic sources covering 13M+ polymers: SMILES, thermal and mechanical properties, monomer precursors, and reaction types.

Data preparation

Standardize notation, units, and structural descriptors with RDKit and subject-matter-expert input to build one polymer profile per entry.

QA generation

Generate tasks programmatically from templates (78%) or by prompting a teacher model for open-ended formats (22%). Splits are applied before generation.

CoT distillation

Condition teacher models on the full profile and gold answer so they explain provided evidence rather than recall it, then fact-check the resulting traces.

PolyBench overview. Left: the four-stage creation pipeline from knowledge base through data preparation, QA generation, and knowledge-augmented CoT distillation. Right: the six task categories ordered by reasoning complexity. — **Figure 2. PolyBench overview.** *Left:* the creation pipeline, yielding non-overlapping splits of 125.3K train, 1.83K dev, and 2.31K test instances. *Right:* task organization — six categories arranged by reasoning complexity, each annotated with its query formats and generation source (D = distilled, P = programmatic).

Results

7B–32B models trained on PolyBench compete with frontier LLMs

We fine-tune Qwen-2.5-7B/14B/32B, Phi-4-14B, and ChemLLM-7B with LoRA for 3 epochs on the PolyBench train and dev sets, and compare against off-the-shelf, chemistry-aligned, and closed-source frontier models. Full per-task numbers across all six categories are in the paper; the headline findings are below.

0.95Chem. validity

General chemistry alignment does not transfer to polymers

Off-the-shelf base models struggle across the board, especially wherever SMILES is required. Chemistry-aligned models — Llamole-8B, LlaSMol-8B, ether0-24B, ChemLLM-7B — underperform on almost every PolyBench task: their knowledge is mostly about small molecules and does not carry over. Training the same ChemLLM-7B on PolyBench consistently beats its own base version, lifting chemical validity of generated structures from 0.25 to 0.95.

16.8MAE, best overall

PolyBench training beats every similar-sized baseline

Across all six task categories, PolyBench-trained models outperform their off-the-shelf and domain-aligned counterparts, with the largest gains at 14B and 32B. Exact match on structural counting rises from 0.50 to 0.87 for Qwen-2.5-14B+CoT, and property-prediction error for Phi-4-14B drops from 46.2 to 16.8 MAE — the lowest of any model evaluated, frontier systems included.

0.82Tanimoto sim.

Design stays hard even for frontier LLMs

Frontier models do well on foundational tasks but struggle to generate chemically valid SMILES that satisfy property, structure, and synthesis requirements at once. PolyBench-trained models reach higher structural similarity to the target polymer (0.82 vs. 0.48 for the strongest frontier baseline) while holding lower synthetic accessibility scores — placing their generations at the edge of the readily synthesizable range.

14B+where CoT pays

CoT gains scale with model size — and with task compositionality

For 7B models, CoT training improves only 4–6 of 14 metrics; for the 14B–32B variants it improves 8–12. The clearest gains land on compositional tasks such as Advanced Property Reasoning, while tasks needing precise quantitative targets see limited or even negative gains — there the added reasoning mostly adds verbosity.

Constraint satisfaction on open-ended design

A subset of the design problems is open-ended: generate a molecule satisfying multiple competing constraints. Since several distinct molecules may satisfy the stated requirements, we score these reference-free — evaluating the generated molecule directly against the constraints stated in the question, using two independent verifiers.

68%

Verified deterministically by RDKit

Hard constraints, decidable from structure alone — no model judgment involved.

32%

Scored by GPT-5.6-Luna

Soft constraints that cannot be resolved structurally, handled by a separate LLM verifier.

0.68Constraint sat.

Phi-4-14B+CoT leads all models, including every frontier baseline

On the combined RDKit + LLM measure over the 200 open-ended design questions, our PolyBench-trained Phi-4-14B+CoT variant achieves the highest constraint satisfaction of any model evaluated — ahead of every frontier baseline.

Analysis

A compositionality gap, not a knowledge gap

Standard metrics do not explain why models fail. Using diagnostic sub-questions generated alongside the main tasks, we run three experiments on PolyBench-trained Phi-4-14B+CoT to separate missing knowledge from failure to combine it.

Exp 1 · Skill gap

Can the model answer atomic sub-questions in isolation? Mean precision is high across core skills — 89.4% for inference, 89.3% for definitions, 83.7% for comparison. The pieces are largely there.

Exp 2 · Skill selection

Does the model's own CoT cover the required sub-questions? Precision stays high (93.0% on Conceptual Knowledge, 84.8% on Advanced Reasoning) but recall collapses — 47.4%, 44.1%, and just 7.0% on Comparison & Ranking.

Exp 3 · Composition

Handed the sub-questions and their answers, scores rise across every category — but composing them into a consistently constraint-satisfying solution stays brittle.

Task category	Exp 2 — CoT coverage		Exp 3 — mean score with added context
Task category	Precision	Recall	No context	+ SubQs	+ SubQs & answers
Structural Understanding	79.5%	47.2%	5.08	4.68	6.02
Conceptual Knowledge	93.0%	47.4%	4.98	6.26	6.48
Property Prediction	44.0%	16.4%	3.14	2.96	3.46
Property Comparison & Ranking	11.3%	7.0%	4.72	4.49	5.92
Advanced Property Reasoning	84.8%	44.1%	5.10	5.94	6.46
Synthesis & Design	36.3%	16.6%	4.30	4.84	5.28

Table 1. Diagnostic probes on Phi-4-14B+CoT. Models often know the necessary pieces (high precision) but fail to select and propagate the right constraints and steps in their own reasoning (low recall) — evidence of a compositionality gap rather than missing knowledge.

Availability

Data, code, and models

All associated data and code will be released under CC BY-NC 4.0 for non-commercial research use in the project GitHub repository. Fine-tuned model weights will be released on Hugging Face under their respective base model licenses (MIT for Phi-4, Apache 2.0 for Qwen-2.5).

CoT traces were generated via the official Claude-3.5-Sonnet and GPT-4o APIs for research in a specialized domain. We will release distilled CoT traces consistent with each provider's terms of service; where terms restrict redistribution, we will release only our post-processed, verified reasoning chains rather than raw teacher outputs.

On potential risks. No generated structure in this work has been experimentally synthesized or characterized. Outputs should be treated as candidate hypotheses for expert review rather than design recommendations, and we caution against using them to direct laboratory work without independent verification.