Architecture AI Updates: October 28, 2026

1. Server-Side Source Control for an AI-Driven Merge Rate

InfoQ. InfoQ analyzed Anthropic’s move of its monorepo to Scion, a source control system built after daily merged changes rose from about 2,500 in May to more than 30,000 in September, with CI performing about 25 million checkouts a day against nearly 100 GB of main-branch history. Scion runs commits, merges, rebases, history and blame on the server and exposes them through APIs, so tools can query diffs without a checkout. Changesets are content-addressed and record only files changed against the first parent. Creating changesets is decoupled from updating references, so no cross-server coordination is needed. Clients keep bounded caches and virtualized checkouts that fetch immutable trees and blobs on demand through a node-local daemon and regional caches backed by Spanner and cloud storage, with a fixed tier order so a regional failure cannot overload the authoritative servers. Anthropic reports cache-hit rates above 99% per tier, a CPU and memory footprint about 200 times smaller than its Git mirrors, and checkouts two orders of magnitude faster. The trade-offs are the loss of offline independence, dependence on server availability, and bespoke integrations with code review, search and CI. Anthropic does not position Scion as a general Git replacement, and InfoQ contrasts it with Uber’s GitFarm, which centralizes Git operations without changing the data model. Source

2. KV Cache Capacity, Not Concurrency, Sets the Limit for LLM Serving

InfoQ. In a QCon AI talk published by InfoQ, Momento CEO Khawaja Shams (formerly DynamoDB lead at AWS) treats the KV cache as the central constraint in LLM serving. For a 4B model he estimates about 72 KB of KV cache per token, so a 20,000-token prompt needs about 1.6 GB. Because each cache entry depends on every earlier token, a changed token such as an injected timestamp invalidates the rest of the prefix. In his vLLM benchmark on a 24 GB GPU, p99 time to first token was about 2.6 s at 12 concurrent sessions and jumped to about 38.9 s at 14 once the cache stopped fitting. Adding 16 GB of host DRAM brought that back to about 3.5 s. Session-affinity routing lifted two-GPU throughput from 128 to 201 output tokens per second, compared with 110 on one GPU. He recommends tiered storage (VRAM, DRAM, NVMe, recompute), chunked parallel fetches from remote caches, disaggregated prefill and decode with prefix-aware routing, and admission control based on a per-token KV budget instead of a fixed session count. Source

3. Fitting a 3B-Parameter Model on a Phone: Palettization, Shared KV and Adapters

ByteByteGo. ByteByteGo examined how Apple runs its roughly 3-billion-parameter foundation model on iPhone, based on Apple’s published engineering material. Weights are compressed with palettization, where short references point into a small lookup table. An earlier version mixed 2-bit and 4-bit weights at an average of 3.7 bits (about 1.39 GB), and the newer model uses 2 bits for decoder weights and 4 bits for embeddings, trained with quantization-aware training. Cross-layer KV sharing splits the model into two blocks at a 5:3 depth ratio, with the second block reusing keys and values from the last layer of the first, which cuts KV-cache memory by 37.5%. Features share one base model through rank-16 LoRA adapters of tens of megabytes that are loaded and swapped as needed. Requests the device cannot handle go to Private Cloud Compute, where encryption limits decryption to validated nodes. The article also notes AFM 3 Core Advanced, a sparse 20B-parameter model that activates 1 to 4 billion parameters, stores the full model in flash, and loads only the selected expert weights into RAM. Source