Von der Wort-Maschine zur Muster-Maschine

[AribertDeckers]

13.9.2026
Von der Wort-Maschine zur Muster-Maschine


https://x.com/HowToPrompt__/status/2099186312033259653

[*quote*]
--------------------------------
How To Prompt @HowToPrompt__

China just published a paper that might end the current LLM era.

They open-sourced NCP-ArchPreview, an 8.9B model that thinks in concepts instead of predicting one word at a time.

Every AI you use right now is playing a massive game of autocomplete.

ChatGPT, Claude, Gemini, they all generate answers exactly the same way.

One. Word. At. A. Time.

It's called Next-Token Prediction. And it is deeply inefficient.

Because humans don't think in single words. We think in concepts.

Now, a team out of Shanghai just proved they can teach AI to do exactly the same thing.

They dropped a paper introducing NCP, Next Concept Prediction.

Instead of blindly guessing the next word, this new architecture builds a hidden "latent space".

It groups information together into larger, discrete concepts.

The model predicts the concept first.

Then it uses that underlying concept to guide the actual words it types out.

The results rewrite the economics of training AI.

They built an 8.9-billion parameter model using this new method.

It achieved the exact same baseline intelligence as a state-of-the-art open model using only 51.3% of the training data.

It reached full convergence using half the tokens.

When fully trained, it crushed standard models, scoring massive 6-point gains in complex math and reasoning.

And it did it while requiring 15% less standard compute.

We have spent the last three years brute-forcing intelligence by feeding models trillions of individual words.

But the future of AI isn't about predicting the next token.

It's about understanding the thought before it's even spoken.

Image


https://pbs.twimg.com/media/HSHOy_UbQAAT_aj?format=jpg&name=4096x4096

7:19 PM · Sep 13, 2026
50.5K Views


--------------------------------
Relevant people

    How To Prompt
    @HowToPrompt__
    Trustworthy AI education.
--------------------------------
[*/quote*]




[*quote*]
--------------------------------
NCP-ArchPreview Technical Report 2026-09-11

NCP-Arch Preview Technical Report:

Moving towards Latent Space Language Models through Next Concept Prediction

The Intern-NCP Team

Shanghai AI Lab; LUMIA Lab, Shanghai Jiao Tong University

https://huggingface.co/collections/ArchSpace-Collection/ncp-archpreview
https://github.com/InternLM/lmdeploy
https://github.com/LUMIA-Group/ncp_olmo_eval

Abstract

We introduce NCP-ArchPreview, a latent-space language model that pushes autoregressive pretraining beyond standard next-token prediction (NTP).

Alongside NTP, the model learns through Next Concept Prediction (NCP) to predict discrete concepts that span multiple tokens, introducing an explicit and more challenging concept-level objective while preserving standard token-level autoregressive generation.

NCP-ArchPreview builds a latent space by constructing a product-quantized concept vocabulary directly from its hidden states, and subsequently learns to predict future concepts via a dedicated Concept Module.

These predicted concepts are then fed back to the token level to guide subsequent generation, with NTP and NCP trained jointly end-to-end.

We scale this architecture to 8.9B parameters and train it on 5.73T tokens from the Dolma-3 dataset, marking the largest demonstration of a latentspace language model to date

Remarkably, by consuming only 51.3% of the total training tokens, NCP-ArchPreview achieves the final pretraining loss of OLMo-3-7B.

Following full pretraining, it outperforms OLMo-3-7B by 2.45 points on the downstream macro-average, including a notable 5.99point gain on GSM8K.

Controlled experiments isolate a clear progression of performance gains stemming from both the latent architecture and the NCP objective.

Furthermore, utilizing only 85% of the standard computation, NCP-ArchPreview approaches the training loss of a strictly parameter-aligned 8.9B baseline.

The learned latent space remains highly valuable after the pretraining stage: updating just the 17M-parameter VQ module yields a novel, lightweight interface for domain adaptation, while a simple injection of concept representations into a DFlash2 drafter improves the mean accepted length by 4.17% with negligible overhead.

By proving the viability of joint token and concept modeling at scale, these results position latent-space prediction not merely as an auxiliary objective, but a highly efficient and scalable architectural blueprint for next-generation foundation models.

We release the Stage-1 checkpoints obtained every 100,000 training steps, the corresponding drafter models, and the final Stage-1 / Stage-2 checkpoint.
--------------------------------
[*/quote*]