A factory buyer types "find me 42 络钼 bolts, 16674, M10*35 but need 50 length." That garbled string contains a phonetic typo for a chrome-moly steel grade (42CrMo), a truncated national standard (GB/T 16674), and a geometric self-contradiction (length 35 mm and 50 mm). A general-purpose chatbot will happily answer it — and get someone killed, because in procurement a fluently wrong bolt is worse than no bolt. This paper is about building a language model that treats "I'm not sure" as a first-class answer.
The Alibaba team starts from an open MoE base model (35B total parameters, only ~3B active per token) and rebuilds its training data around diagnosed failures rather than raw volume: the same engineering fact gets rewritten across 10 genres and 8 writing styles so the model stops recognizing knowledge only when it's dressed in standards-committee prose; fluent-but-wrong paragraphs get surgical two-percent edits instead of regenerations; and QA pairs are synthesized precisely where the current checkpoint fails an 8-sample probe. Downstream, they formalize a hard rule the model itself cannot satisfy alone: product eligibility is decided by evidence under three-valued logic, where missing, stale, or conflicting catalog data evaluates to unknown — and unknown is never satisfied.
The result is an openly released checkpoint that, in randomized production A/B trials at ~100,000 users per arm per day, cut response latency from 6–7 s to 1.5 s while lifting gross merchandise value +4.25% and satisfied buyer inquiries +8.3% (all p ≤ 0.01), and raised attribute-standardization from 62.5% to 85.2%. Offline, the domain-adapted checkpoint beats its non-adapted twin by +2.97 pp exact match on procurement-query structuring (95% CI [2.11, 3.86]) — and outscores MoEs 10× its active size on an industrial benchmark.
A buyer needs a replacement pump for a corrosive process line. The catalog returns a pump that matches the category and nominal flow perfectly — and would fail on day one, because nobody recorded what seal it uses. Why is the most common "successful" answer in industrial search also the most dangerous one?
Formally, the paper defines a demand interpreter that maps a query $q$ and its conversational context $x$ — against a versioned registry of category schemas $S$ (grounded in national standards like GB/T), a frozen product-evidence snapshot $P_t$, risk tiers $T$, and a policy bundle $\Gamma$ — into a structured intent state:
Each resolved constraint inside $H$ is not a blob of text but a fully typed record:
Two design details matter more than they look. Polarity $\pi$ is decoupled from origin $r$, so "no cast iron" stays an explicit, excluded constraint rather than being flattened into a positive one. And anything decision-critical that the buyer never said — medium concentration, operating temperature, seal construction — goes into $U$, not into a confident guess. If the category or risk tier can't be uniquely resolved, the system commits to $\bot$ and admits no product pool at all.
The paper illustrates the whole pipeline with an authentic, de-identified fastener inquiry. Try the three failure modes yourself:
Click each pathology to see how a general LLM and IndustryLLM parse the same token. The interpretation updates live.
The query never names a certificate, a coating, or a torque spec — and a good interpreter must notice that, not paper over it. Every field the buyer failed to specify becomes a row in $U$, and per the paper's Table 1, the safe action for a decision-critical absence is a targeted clarification — never a hallucinated default industrial specification.
Boolean logic has two values, and in safety-critical procurement that is exactly one too few. The paper's central interface adds a third — unknown — and refuses to round it in either direction. Here is the machine that decides whether a bolt, a pump, or a flange may legally enter the eligible set.
Three formal pieces make the gate run. First, an intent state is ready only if the category and risk tier resolved, no critical field is unresolved, and every derived constraint still carries valid evidence:
Second, each candidate product $p$ supplies a typed attribute set $A_t(p)$, of which only the subset $V^\Gamma_t(p)$ has valid, non-conflicting evidence (within a conflict group, only the highest-precedence source survives):
Third — the heart of the paper — every hard requirement $z \in H$ evaluates under three-valued logic, where $S_\Gamma$ means "valid evidence supports $z$" and $C_\Gamma$ means "valid evidence contradicts $z$":
Four candidate pumps, five hard constraints. Click any evidence chip to cycle its state: verified → stale → missing → conflict → back. The eligibility table re-evaluates under Equation 5 on every click.
This is the paper's boundary line, stated as engineering: language models structure demand; current product records determine eligibility. The model may know everything ever written about corrosion — it still cannot turn an absent catalog field into a verified value. Flip the logic toggle above and watch how many unqualified pumps a two-valued system would happily ship.
Continued pre-training usually means "dump the domain corpus in." IndustryLLM's corpus is instead a balanced instrument: a fifth standards, a tenth real buyer noise, a quarter filtered web — and over half general replay, on purpose. Drag the sliders and watch the forgetting-risk meter respond.
Each source earns its slot. The filtered open web (≈25B retained tokens) passes a two-tier MacBERT pipeline: a seven-way classifier isolates six core industrial categories, then a regression head distilled from proprietary LLM annotations scores each document's pedagogy and information density on a 0–10 scale. The curated archive (≈5B tokens) contributes encyclopedias, white papers, monographs, and national standards (GB/T) that open-web crawls systematically under-represent. The platform commerce records (≈10B tokens) — transaction logs, catalog descriptions, raw buyer inquiries, strictly de-identified — are where the model meets real typos like "42 络钼."
One accounting subtlety the paper is unusually careful about: the ≈20B-token transformed subset is subsumed within the 40B domain subtotal, not added on top. And the 100B figure is a retained source pool, not cumulative exposure — both the main and annealing phases run multiple epochs, terminating on validation-loss convergence.
Sliders start at the paper's values (Table 3). The reconstruction slider allocates transformed tokens inside the domain subtotal. Live updates as you drag.
The corpus is not the model, but it decides what the model can know: standards define canonical terminology, commerce logs map dialect to catalog, and the 60B-token replay is what lets a domain specialist keep its MMLU-Pro score — the trade-off Chapter 4 measures directly.
A model trained only on standards-committee prose learns that 42CrMo lives in standards-committee prose. Ask the same question in a buyer's dialect and the knowledge vanishes. The fix is almost absurdly simple: say the same thing ten different ways — and it buys +6.7 points without losing a single MMLU-Pro point.
The pipeline has three stages: (1) Genre Allocation — the editor assigns up to four genres from ten candidates, ordered by fidelity, and abstains entirely if the content is ill-suited; (2) Style Conditioning — one of eight writing styles per retained genre, matched to operational context; (3) Controlled Generation — an independent rewrite per genre–style pair. Semantic equivalence is enforced by prompt guards rather than formal parsing — an honest limitation the paper states plainly.
The genre and style menus, with their observed allocation shares from the paper's Table 5:
Pick a genre and a style to preview how the same fact gets re-dressed. Chart updates live; the proxy experiment below shows what it buys.
Setting B is the cautionary tale: single-register adaptation buys domain gains and costs general capability (MMLU-Pro 63.51 → 62.56). Setting C gets more of the former and none of the latter. Linguistic diversity is not decoration — it is the difference between specialization and lobotomy.
The deadliest corpus defects are the ones classifiers can't see: a fluent paragraph citing an expired GB standard, an inverted tolerance sign, a wrong threshold that makes the verdict right by accident. Regenerating the text risks new hallucinations — so the paper edits two percent of the tokens and leaves the other ninety-eight alone.
The proxy experiment trains two Qwen3.5-2B-Base checkpoints on identical document sequences (2B tokens; edited spans ≈2% of total). Each transformation record becomes a pair of equally fluent statements differing only at the edited fact, and the models must choose. Chance is 50%. Results across seven engineering strata (Table 7):
Hover the bars for strata details. Note what the paper's Appendix A admits: option order wasn't randomized in historical logging, so positional bias can't be formally excluded — suggestive evidence, not proof.
The third transformation attacks a subtler pathology: knowledge latency, where facts are in the weights but unreachable. The pipeline asks the current checkpoint each candidate question eight times (T = 0.7, top-p = 0.8), has Qwen3.7-Max score each response 0–5, then computes a trimmed mean — dropping the single highest and lowest score before averaging:
Eight sliders = eight rollout scores from the checkpoint. The trim drops the extreme scores (shown hollow) and averages the rest. Below the threshold τ, the question gets a synthesized, source-grounded training answer.
Two honest results in one chapter. Surgical edits measurably steer which facts a model believes (+13.9 pp, with the caveat that option order wasn't randomized). And QA formatting activates dormant knowledge far more than clever question selection does — the boring transformation was the strong one. Failure-driven budget allocation survived contact with its own ablation for two of three mechanisms.
The model never changes: same architecture, same tokenizer, frozen vision encoder, from first gradient to last. Everything that moves is data and supervision — plus one unconventional optimizer choice that reached the finish line 2.2% lower in loss.
The inherited configuration (paper's Table 2), held strictly invariant so that every performance shift traces to data and supervision:
On a 35B testbed (500K samples, context 8,192, learning rate $2\times10^{-5}$), the team benchmarked the Muon optimizer (scale factor 0.5, split_qkv disabled) against AdamW. Muon reached AdamW's endpoint loss around iteration ≈185K, finished 0.030 lower (≈2.2% relative), and ran with 12.6% lower mean global gradient norms. The paper is careful: without a parallel full-scale AdamW run, this stands as design justification, not proven production superiority.
Left: the WSD learning-rate schedule (drag its shape). Right: stylized loss traces anchored to the paper's reported minima — main CPT 1.0157 @ step 11,660; annealing 0.6800 @ step 1,300; teacher-target SFT 0.4858 @ step 6,665 (Figure 4). Curves are illustrative reconstructions from those anchors, not raw logger exports — the paper says exactly this about its own figure.
By freezing architecture and tokenizer, the paper converts "did the model get better?" into a clean question about data. And the best-of-four teacher gate embodies a quiet thesis: alignment quality is a curation problem. One prompt, four candidates, ruthless filters — the SFT loss is vanilla cross-entropy; nothing changed except what counts as a good answer.
The paper refuses to pool its evidence — proxy ablations, benchmark diagnostics, offline components, and randomized production trials each get their own tier. Chapter by chapter, here is what each tier actually says: a 3B-active model outscoring 10×-bigger rivals, a +2.97 pp exact-match win with a confidence interval, and one honest asterisk on safety.
Every model from the paper's Table 12, placed by activated parameters (log scale) and IndustryBench score. Toggle raw vs. safety-adjusted (Final SV). Hover any dot for details.
The developmental lineage on IndustryBench (raw 0–3); general-capability deltas from swapping SFT targets to best-of-four teacher answers (13 benchmarks); and the isolated CPT contribution: procurement-query structuring, No-Think mode, Base+SFT vs CPT+SFT.
Look at the middle chart and the trade-off the team accepted: teacher targets buy +8.6 on HumanEval-Plus and +7.1 on GPQA-Diamond, and pay −1.8 on MBPP-Plus and −0.5 on IF-Eval — long deductive chains beat short idiomatic completions. Supervision shape is a dial, not an upgrade. And the safety adjustment (2.120 → 2.030) is the paper measuring its own verbosity risk: reasoning traces can smuggle in unverified technical claims.
Latency from 6–7 seconds to 1.5. GMV +4.25%. Attribute standardization from 62.5% to 85.2%. All in randomized 50:50 A/B trials with ~100,000 persistent users per arm per day, all p ≤ 0.01 — and all honestly labeled as a stack effect, not a weights effect.
Simulate day-by-day conversion under a sticky 50:50 split. Drag the true lift and watch the measured lift and z-score evolve — with 100,000 users per arm per day, even small true effects separate from noise fast. "New day" draws the next day of traffic (a real state change, not a re-render).
The full production ledger, beyond the 18-day inquiry trial: a two-week randomized normalization experiment (an attribute-value cleanup layer before ranking) lifted product CTR +2.4%, total inquiries +4.32%, and satisfied inquiries +8.3%. An observational pre–post search deployment (no control arm — the paper says so) saw demand-satisfaction +6.5%, transaction conversion +7.11%, and product-search PV-to-order rising 0.148% → 0.156%. And a blind multi-dimensional rubric over 1,000 real dialogs scored the No-Think stack 60.1 vs 53.7 for the incumbent — biggest wins in information-security compliance (+3.22) and parameter faithfulness (+1.39), with the model conceding conversational tone (−0.58), a trade the authors accepted deliberately.
Latency is not cosmetic. A 6-second reply is a 7-second reply with queueing, and industrial buyers simply leave — every latency cut in the trial converts directly into more questions asked, more constraints captured, and more qualified quotes. The fastest safe answer is also the most profitable one.
Everything in this paper funnels toward one boundary: a language model can learn what 42CrMo means, but only a live, timestamped, auditable record can say whether this bolt, today, is certified. Click through the architecture to see who owns which half of the truth.
Click any node to inspect its role. The demand track (top) is parametric and learned; the supply track (bottom) is evidence-gated and timestamped — and they meet only at the gate.
The limitations section is unusually specific about what remains unproven: proxy ablations ran at 2B–4B under single seeds; option order in the 2AFC probe wasn't randomized; teacher targets changed token exposure, so compute wasn't matched; production results bundle weights with prompts and serving; the empirical scope is Chinese B2B commerce only; the vision encoder stayed frozen; and generator, editor, and judge all come from one model family — intra-family blind spots are possible. Weights and inference configs are released openly on Hugging Face, while the corpora (GB/T archives, transaction logs) are not redistributable — the paper distinguishes checkpoint openness from lineage reproducibility as separate governance dimensions.
Industrial procurement is where AI's usual failure mode — fluent overconfidence — meets physical consequence. This paper's answer is a division of labor: let the model understand the question, let the evidence answer it, and never let "probably" stand in for "verified." Unknown is not satisfied. Not sometimes. Never.