SIM-CoT

ICLR 2026

SIM-CoT

Supervised Implicit Chain-of-Thought

Xilin Wei1,2, Xiaoran Liu1,4, Yuhang Zang2, Xiaoyi Dong2, Yuhang Cao2, Jiaqi Wang2,4, Xipeng Qiu1,4, Dahua Lin2,3

1Fudan University 2Shanghai AI Laboratory 3The Chinese University of Hong Kong 4Shanghai Innovation Institute

Corresponding authors

Paper arXiv Code Models BibTeX

Implicit reasoning, made visible

Think in latents.
Decode on demand.

SIM-CoT reasons through a few continuous latent tokens instead of a written chain of thought, then answers. The decoder that supervises each latent during training is removed at inference, and can be attached again to read every latent as an explicit step.

A watermelon word problem is embedded and passed to the LLM, which produces implicit latent tokens and the answer 21. The optional decoder reads the latents as 0.3*120=36, (120-36)*3/4=63 and 84-63=21.

+8.2

points over Coconut

GPT-2 · GSM8k-Aug

+3.0

points over CODI

LLaMA 3.1 8B · GSM8k-Aug

2.3×

fewer tokens than explicit CoT, with 2.1 points higher accuracy

GPT-2 · GSM8k-Aug

0

added inference overhead

The decoder is used only in training

Abstract

Stable, scalable and interpretable implicit reasoning.

Implicit Chain-of-Thought (CoT) methods offer a token-efficient alternative to explicit CoT reasoning in Large Language Models (LLMs), but a persistent performance gap has limited their adoption. We identify a core latent instability issue when scaling the computational budget of implicit CoT: as the number of reasoning tokens increases, training often becomes unstable and collapses. Our analysis shows that this instability arises from latent representations becoming homogeneous and losing semantic diversity, caused by insufficient step-level supervision in current implicit CoT methods. To address this, we propose SIM-CoT, a plug-and-play training module that introduces step-level supervision to stabilize and enrich the latent reasoning space. SIM-CoT employs an auxiliary decoder during training to align each implicit token with its corresponding explicit reasoning step, ensuring latent states capture distinct and meaningful information. The auxiliary decoder is removed at inference, preserving the efficiency of implicit CoT with no added overhead. It also provides interpretability by projecting each latent token onto an explicit reasoning vocabulary, enabling per-step visualization and diagnosis. SIM-CoT significantly improves both in-domain accuracy and out-of-domain stability of implicit CoT methods, boosting Coconut by +8.2% on GPT-2 and CODI by +3.0% on LLaMA-3.1 8B. It further surpasses the explicit CoT baseline on GPT-2 by 2.1% with 2.3× greater token efficiency, while closing the performance gap on larger models like LLaMA-3.1 8B.

The problem

More latents should help. Instead, training collapses.

Explicit CoT gets better with more reasoning tokens, so the natural move for implicit CoT is to add latents. With Coconut, accuracy rises from one to four latents, then collapses to 12.5% at five. The collapsed latents become nearly identical, drift away from the vocabulary embedding space and lose operator information.

Figure 1 of the paper. (a) Coconut training accuracy as latents are added one stage at a time, dropping to a worst point of 12.5% after the fifth latent. (b) Number, operator and answer accuracy for 1, 2, 4 and 5 latents and for the failed 5-latent model, where operator accuracy is 4.7. (c) Latent distance narrows while vocabulary distance grows in the failed model. (d) A normal model decodes numbers and operators; a failed model decodes mostly numbers in both latents.
Figure 1. Coconut trained on GSM8k-Aug with a growing number of latents.
a

Accuracy collapses

Accuracy improves up to four latents. With five it drops to 12.5% and keeps swinging.

b

Operators disappear

Operator accuracy of the decoded latents falls from 83.0 to 4.7 in the failed five-latent model.

c

Latents converge

Pairwise latent distance falls to 4.21, far below any healthy setting (20.30–28.34), and distance to the vocabulary center rises to 39.39.

d

Latents say the same thing

Both latents of a failed model decode mostly to numbers. A normal model keeps operators in its second latent.

Method

Supervise every latent step.

Coconut supervises only the answer, and CODI distills a whole trajectory. Neither tells the model which latent should encode which step. SIM-CoT adds an auxiliary decoder during training that must generate the k-th reasoning step from the k-th latent alone.

Figure 2 of the paper. Coconut applies only answer-level supervision with a cross-entropy loss on the answer. CODI adds trajectory-level supervision by distilling hidden states from an explicit-CoT teacher. SIM-CoT feeds each implicit CoT token into a decoder model that predicts the corresponding explicit reasoning step, with a cross-entropy loss on both the step prediction and the answer.
Figure 2. Coconut (top left), CODI (top right) and SIM-CoT (bottom).

Answer-level

Coconut

Cross-entropy on the final answer only. The intermediate latents are left weakly constrained.

Trajectory-level

CODI

Distills from an explicit-CoT teacher by aligning hidden states of the full reasoning trajectory.

Step-level

SIM-CoT

Aligns each latent with its own reasoning step through the auxiliary decoder, so every latent carries one step.

Latents

zk=Hθ(U(k−1)), U(k)=U(k−1)⊕zk

For K fixed steps, the last hidden state is fed back as the next input.

Step supervision

ℒstep=−∑k,t log pφ(yk,t | zk, yk,<t)

The decoder sees only zk and must generate step sk.

Objective

ℒ=λstepℒstep+λlmℒans-lm

Step gradients flow through z1:K into the LLM.

At inference the decoder is removed. The model runs its K latent steps and decodes the answer directly, so decoding takes T + K + La positions, the same as the implicit method it is added to.

Results

Consistent gains on every backbone.

Added to Coconut or CODI, SIM-CoT raises GSM8k-Aug accuracy on GPT-2 and on LLaMA 3 models from 1B to 8B without adding tokens. On GPT-2 it is the first training-based implicit method to surpass explicit CoT.

Trained on GSM8k-Aug. In-domain: GSM8k-Aug test set. Out-of-domain: GSM-Hard, MultiArith and SVAMP.

Table 1 · GPT-2. Accuracy (%) and number of generated tokens.
MethodIn-domainOut-of-domain
MethodGSM8k-AugTokensGSM-HardMultiArithSVAMPAverageTokens
SFT-CoT explicit42.727.69.085.041.645.224.7
No-CoT19.12.24.341.116.420.61.4
iCoT30.12.25.755.529.430.21.4
Coconut36.612.28.183.536.242.611.4
+ SIM-CoT 44.8+8.212.29.390.840.746.9+4.311.4
CODI42.012.29.493.041.748.012.6
+ SIM-CoT 42.6+0.612.29.492.842.648.3+0.312.6
Table 2 · LLaMA 3.2 1B. Accuracy (%) and number of generated tokens.
MethodIn-domainOut-of-domain
MethodGSM8k-AugTokensGSM-HardMultiArithSVAMPAverageTokens
SFT-CoT explicit58.425.313.996.765.758.823.1
No-CoT28.81.26.350.326.727.81.9
iCoT19.01.24.439.040.928.11.9
Coconut33.213.27.063.343.738.011.9
+ SIM-CoT42.2+9.013.29.387.743.947.0+9.011.9
CODI52.713.211.995.060.655.813.4
+ SIM-CoT 56.1+3.413.212.796.261.556.8+1.013.4
Table 3 · LLaMA 3.2 3B. Accuracy (%) and number of generated tokens.
MethodIn-domainOut-of-domain
MethodGSM8k-AugTokensGSM-HardMultiArithSVAMPAverageTokens
SFT-CoT explicit71.527.717.098.371.062.122.4
No-CoT38.31.29.588.752.950.41.4
CODI60.87.214.398.773.362.17.5
+ SIM-CoT 62.3+1.57.214.698.874.962.8+0.77.5
Table 3 · LLaMA 3.1 8B. Accuracy (%) and number of generated tokens.
MethodIn-domainOut-of-domain
MethodGSM8k-AugTokensGSM-HardMultiArithSVAMPAverageTokens
SFT-CoT explicit71.727.416.598.373.162.622.2
No-CoT39.51.29.888.055.351.01.6
CODI61.17.215.599.578.164.47.5
+ SIM-CoT 64.1+3.07.216.3100.079.465.2+0.87.5

More latents, no collapse.

Coconut and SIM-CoT on GPT-2 as the number of latents grows. Each latent spans two implicit tokens.

Decoder size LLaMA 3.2 1B with 1B, 3B and 8B decoders
Accuracy (%). The baseline is CODI on LLaMA 3.2 1B.
In-domainOut-of-domain
DecoderGSM8k-AugGSM-HardMultiArithSVAMP
Baseline52.711.995.060.6
+ 1B decoder56.112.796.261.5
+ 3B decoder50.411.695.659.8
+ 8B decoder50.011.794.256.8
Soft thinking Training-free soft thinking on top of Coconut and SIM-CoT
Accuracy (%).
MethodGSM8k-AugGSM-HardMultiArithSVAMP
Coconut36.68.183.536.2
+ soft thinking36.78.385.236.0
SIM-CoT44.89.390.840.7
+ soft thinking45.09.491.540.8
Training setup Learning rates, epochs and curriculum
ModelMethodLearning rateEpochs
GPT-2Coconut1 × 10−415 + curriculum
LLaMA 1BCoconut1 × 10−415 + curriculum
GPT-2CODI3 × 10−340
LLaMA 1BCODI8 × 10−410
LLaMA 3BCODI3 × 10−48
LLaMA 8BCODI1 × 10−46

Adam (β1 = 0.9, β2 = 0.999), weight decay 0.1, batch size 128 for GPT-2 and LLaMA 1B and 64 for LLaMA 3B and 8B, early stopping with a patience of 3 epochs. Coconut curriculum: every three epochs one explicit step is replaced by a latent until the maximum is reached, then 15 more epochs.

Interpretability

Read every latent.

Each latent is trained to carry one step, so the training decoder can translate latents back into text. On these GSM8k problems the decoded steps form a chain that ends at the correct answer.

Distinct, but still grounded.

SIM-CoT pushes latents further apart than any setting without step supervision, while keeping them near the vocabulary embedding space.

Latents vs. vocabulary embeddings PCA view of normal, failed and SIM-CoT latents
Appendix figure: PCA projections of two implicit tokens against the vocabulary embeddings. Normal implicit tokens are well separated near the vocabulary cloud; failed tokens collapse together and drift far away; after SIM-CoT the tokens are separated again and stay close to the vocabulary.
Figure 6. Rows: normal implicit tokens, failed implicit tokens, and implicit tokens after SIM-CoT.

Citation

BibTeX

@inproceedings{wei2025simcot,
  title={{SIM-COT}: Supervised Implicit Chain-of-Thought},
  author={Wei, Xilin and Liu, Xiaoran and Zang, Yuhang and Dong, Xiaoyi and Cao, Yuhang and Wang, Jiaqi and Qiu, Xipeng and Lin, Dahua},
  booktitle={International Conference on Learning Representations},
  year={2026}
}