- Defeating nondeterminism in LLM inference https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/
- All text and data is converted into vectors through embeddings
- An LLM prediction is a massive sequence of floating-point operations such as matrix multiplies, reductions, and non-linearities on those vectors
- Floating-point operations are approximate, and changing the order of additions or multiplications can cause rounding to shift results slightly
- On modern hardware, operations are run in parallel and often batched together with other requests, which changes the shapes going into kernels
- A kernel is like the cashier that processes your math, and many kernels today are not batch-invariant, so their internal math order depends on batch size or layout, causing two identical inputs to take slightly different computational paths depending on who else is batched with you
- These tiny shifts usually do not matter, but when two candidate tokens are very close in probability, that minuscule difference can flip the winner
- The nondeterminism is not just random floats, but also system-level batching and kernels that respond differently to different batch sizes, making LLM inference nondeterministic in practice
- The fix is to design batch-invariant kernels that always give the same result for the same input, no matter the batch size, so server batching stays efficient and outputs stay deterministic
- AI will not make you rich https://joincolossus.com/article/ai-will-not-make-you-rich
- AI growth looks less like the internet and more like containerisation: a big cost-saving technology that standardizes infrastructure, consolidates quickly, and mostly benefits downstream industries and consumers
- The internet era made investors who bet on “the new thing” rich because experimentation was cheap and permissionless and anyone could launch a startup from a dorm room or garage
- That freedom to tinker created surprising new markets and gave rise to entirely new giants (Google, Amazon, Facebook), not just extensions of old ones
- By contrast, AI tinkering is constrained: building frontier models is too expensive, access is controlled by incumbents, and most app value can be captured by the model providers.
- So fortunes in AI won’t come from owning the new tech itself, but from companies that reimagine industries with AI-driven cost savings and pass benefits on to consumers.
- Why language models hallucinate https://openai.com/index/why-language-models-hallucinate/
- Hallucinations are plausible but false statements generated by language models
- Language models hallucinate because standard training and evaluations measure model performance in a way that encourages guessing rather than honesty about uncertainty
- So the fix is to penalize confident errors more than you penalize uncertainty, and give partial credit for appropriate expressions of uncertainty during evaluations
- Oracle https://www.fabricatedknowledge.com/p/oracle-and-animal-spirits
- Oracle is going all-in on building the biggest GPU-first cloud, mainly for OpenAI. Microsoft slowed spend, Oracle stepped in, and became OpenAI’s core infra partner.
- Their database still runs the world. For ERP, CRM, and warehousing it’s the default. Postgres and Mongo get the hype, but they’re not truly substitutes yet.
- Oracle’s bet is to bring vectors and inference right into the database. They already sit inside every enterprise, and now they’re also the GPU heavyweight.
- If they tie their database franchise to inference, they could become the default AI runtime for enterprises everywhere.
- Processes and threads https://planetscale.com/blog/processes-and-threads
What I’ve been reading (W41, 2025)
Nondeterminism in LLMs, AI wealth, Oracle’s rise, and nerding out on computers