Thinking in Distributions
I love that a foundational work of statistics and Newton’s early work on calculus belong to the same decade. John Graunt published his analysis of London’s Bills of Mortality in 1662; Newton developed his early calculus in the mid-1660s. Leibniz’s independent development followed in the 1670s. Neither field has a tidy single birthday, but the proximity is striking.
Calculus is commonly seen as a relatively advanced course only for technically leaning students. Basic statistics now reaches students across the social sciences, business, and medicine, including many who will never study calculus. If we grant that calculus is more complex than statistics, I’m tempted to infer that the historical coincidence is due to how foreign distributions and probabilities are to the human mind. In my own work, I repeatedly encounter requests for a single number where the decision actually depends on better handling the range of possible outcomes.
Using a distribution can be easier and more effective than developing a more sophisticated point estimate. The modeling and system architecture can be straightforward once the problem is stated that way. The harder move is recognizing that the point estimate was an incomplete answer.
Forecasting
Everyone wants to know the future. But if the business needs to buy inventory, “How many units will we sell next quarter?” can be a surprisingly incomplete question. The goal is not to perfectly match a theoretical expected demand. Actual demand could be dramatically higher or lower than that single point. The practical question is how much inventory to buy, weighing the probability of running out against the cost of holding whatever remains unsold.
That difference can change what we build, how we evaluate it, and where we spend the next month of work. Yet the request often arrives as a demand for a better point estimate.
Suppose our best forecast for each of two products is 100 units next month. For one, actual demand is equally likely to be 95 or 105. For the other, it is equally likely to be 50 or 150. Both forecasts correctly describe expected demand, but they imply very different purchasing decisions. Buying 110 units covers either outcome for the first product and leaves a substantial chance of a shortage for the second.
If the goal is to have enough stock in 95 percent of cycles, we need an estimate of the relevant upper percentile of demand over the replenishment period. If the goal is to balance the costs of excess stock and shortages, those costs determine which part of the distribution matters. Standard inventory models formalize this relationship between purchasing decisions and demand quantiles.
A rough distribution that gets the relevant region approximately right can therefore support a better purchase than an excellent estimate of expected demand alone. “Rough” still has to mean useful at the decision boundary. Getting the middle right while badly understating the chance of a surge would defeat the purpose.
Demand forecasting can supply exactly this information. The mistake is treating a point forecast as the finished product, then leaving the buyer to invent the uncertainty around it. Attaching a confidence interval for the estimated mean does not fix the problem: uncertainty about the mean and variation in actual future demand are different quantities. A prediction interval addresses future outcomes; a confidence interval for the mean can become very narrow while actual demand remains highly variable.
This is a particularly expensive kind of false precision. We can spend months refining the center of the forecast while leaving the purchasing decision almost untouched. Before commissioning a more sophisticated point forecast, I would test whether a simple baseline with a credible estimate of demand variability produces better inventory decisions. Compare the resulting shortages and excess stock, rather than assuming that lower forecast error will translate into a better purchase.
The purchase order will eventually contain a single number. Thinking in distributions is how we decide which number belongs there.
Accuracy
Mean squared error already gives greater weight to large errors. The objection is not that it ignores extremes. It is that its particular weighting may have little to do with the consequences of using the model. Are overestimates and underestimates equally costly? Does twice the error really cause four times the damage? Does an error matter at all until it crosses a threshold that changes an action?
Even when the metric fits, a single score can conceal the operating choices available to us. The same issue arises when we evaluate classification accuracy: reliability can vary substantially across the cases that contribute to the overall score.
Consider a hypothetical model that gets 90 percent of cases right. Suppose a screening rule identifies half the cases on which its accuracy is 99 percent. Accuracy on the other half would be 81 percent. The overall score describes both halves together, but it does not tell us whether the model should handle every case.
We might use it automatically on the first half and send the rest to an existing process. The model outputs we accept are now 99 percent accurate without any change to the model. That improvement comes with 50 percent coverage. The existing system might handle the remaining cases better, or a separate model could perform better with a narrower mandate. The accuracy and cost of handling that remaining half still count toward the performance of the whole system. But looking beyond the overall score reveals options about where to employ the model that can lead to a more effective total system.
In the inventory example, we examined possible outcomes for a given product. Here, we are examining differences in reliability across cases. Recognizing that variation lets us decide which cases to automate.
Building the decision system from input to outcome – rather than just training the model in the middle – gives us opportunities to make improvements at multiple points. Before inference, we can route cases using what we know about their source, format, or similarity to previously successful cases. After inference, we can validate the output and decide whether to accept it, retry, or use a fallback.
Finding a signal that separates reliable outputs from unreliable ones can be easier than improving a mature model directly. A known document format, a successful reconciliation against a recorded total, or an independently checkable constraint may provide evidence that another round of model training cannot cheaply reproduce.
We still have to demonstrate that the signal works. We cannot identify the “best half” by looking at answers we will only learn later. A confident answer is not necessarily correct, and a retry may repeat the same mistake. Routing and validation need to demonstrate their value on cases whose outcomes were unavailable when the rules were chosen. The claim is testable: how much can we reduce the error among accepted outputs, at what coverage and cost?
Once we stop requiring a single model to treat every case, we have more options than “improve the model” or “wait until accuracy is high enough.” We can improve selection, validation, or fallback. Which improvement matters depends on where the consequential errors occur. The opportunity is to find the cheapest way to improve the decisions that reach production, even when the model’s overall benchmark score stays exactly where it was.
Speed
Speed is an area where thinking in distributions is already familiar. Teams routinely discuss the slowest one percent of requests alongside average response times. The less obvious question is what happens when we assemble those components into a workflow.
A component can look reliable on its own and contribute to a system that regularly misses its deadline. Average durations do add for sequential stages, even when those durations are dependent. But their 99th-percentile durations generally do not add up to the pipeline's 99th percentile. For parallel work that must all finish, completion time depends on the slowest required response.
In a simplified example, a request needs responses from 100 services, each with an independent one percent chance of missing a shared deadline. The chance that at least one misses is about 63 percent: one minus 0.99 raised to the hundredth power. A rare delay in each component becomes a common delay in the assembled workflow. Dependence between services changes that calculation; it is another reason to examine the whole system. The broader problem of latency tails becoming consequential at scale is well established.
Improving average speed might barely help the requests that miss the deadline. We might instead need to reduce the slow tail, return a partial result, or stop waiting for an optional component.
This also constrains the model strategy. Validation, retries, and fallbacks may improve correctness while making an answer arrive too late to use. Quality and speed have to be evaluated together under the policy we intend to deploy.
Across these examples, the useful question is what happens when outcomes vary. Which outcomes change the decision? What does each mistake cost? Can we recognize the cases that need different treatment? What happens when the components are assembled?
We may eventually reduce the answer to an inventory quantity, an acceptance threshold, or a deadline. But we should do that after examining the distribution and the available actions. Otherwise we risk spending months making a number more accurate when the decision needs information that single number cannot contain.