How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications
In an essay grader we run in production, that vocabulary split the text the model writes most into many tokens. This article previews some results from the VocabX paper, which we will publish soon: how we specialized the vocabulary around the cost of each token rather than its frequency, and cut measured inference cos…