AI Architecture: September 4, 2026

1. Compressing a System Prompt Into Learned Embeddings Instead of Words

Shopify. Gisting attacks system prompt cost at the embedding layer rather than by rewriting text. Training runs in two phases: a teacher pass executes the model with the original prompt to produce reference outputs, then a student pass runs it with learned gist tokens, minimizing divergence between the two until they align. The finished gist embeddings fold directly into the model’s embedding matrix, so runtime needs no custom infrastructure. On Shopify’s Sidekick GraphQL agent the system prompt went from roughly 6,000 tokens to 1,500 gist tokens, a 4:1 compression that cut median time-to-first-token from 438ms to 354ms, end-to-end latency from 6.8s to 4.2s, and raised throughput from 20.2 to 23.4 queries per second. The architectural point is that a long system prompt is a fixed, known input, which makes it a candidate for amortization at training time rather than a per-request cost. Source