Take a sentence like She was building a gazebo. Has she built one? Not necessarily. The work may be underway, but the sentence does not promise a finished structure. That small gap between an event happening and its endpoint being reached has become a test of whether language models can reason about meaning. But what if a model’s wrong answer is partly a clue that the test itself is asking the wrong question?
That is the puzzle Kaiqiao Han and Yizhou Sun investigate in a recent study of a language benchmark built around the so-called imperfective paradox. The name sounds grand; the everyday issue is familiar. We can describe an action in progress without knowing whether it reached its goal. A carpenter was building a gazebo; a storm interrupted the work. Did the carpenter build a gazebo? The sentence leaves the eventual outcome open. It does not prove the gazebo was finished, but it does not prove the carpenter never finished it later, either.
To score a model on such questions, a benchmark typically asks whether one sentence supports another. The answer might be entailed, contradicted, or unknown. The trap is that these labels demand more than a guess about what probably happened. A storm can interrupt construction without ruling out a later return to the job. If a test marks that possibility as flatly false, a model that chooses “unknown” may be reasoning carefully—and still be marked wrong.
The researchers found three places where the test could blur the issue. First, it treated a verb as if it carried the same event structure wherever it appeared. Yet the whole phrase matters. Run in the park describes an activity without a built-in finish line; run a mile supplies one. Even the verb pick shifts: picking apples can continue without a set endpoint, while picking an apple points toward a particular result. A model asked to reason about completion first has to work out what kind of event the full phrase describes.
Second, the benchmark treated completion as a neat, context-free boundary. Ordinary language can be less tidy. What counts as having cleaned a garage or washed a car may depend on what the speaker means by “cleaned” or “washed.” That does not mean any small fragment of a task counts as completion: building only a house’s foundation is not ordinarily building the house. The point is narrower: a single rule may not fit every predicate and context.
The third clue involved the difference between not established and shown to be false. If a sentence says someone was cutting fabric for a dress but realized the cut was wrong, the benchmark labeled the claim that she cut the fabric for the dress as false. Han and Sun argue that interruption or trouble does not always settle whether the relevant event occurred or whether its endpoint was reached. Depending on the details, a claim may be unresolved rather than contradicted. A test that forces the wrong label can turn a dispute about annotation into an apparent failure of reasoning.
How often were the labels genuinely open to question? The original benchmark contained 400 examples, split evenly among four kinds of cases. Three native English-speaking annotators, given background on the linguistic distinctions, judged whether other labels could also make sense. For 70 examples—17.5% of the set—at least two annotators accepted an alternative to the benchmark’s official answer. The uncertainty clustered in the accomplishment cases, which involve an endpoint: 38% in one group and 29% in another. In the two activity groups, the figures were just 2% and 1%. Even when asked to pick one preferred label, annotators matched the original answer less often for those endpoint-focused cases.
That is a substantial clue, but not a verdict that the benchmark is useless. Most original labels still won majority preference, and the annotation protocol itself matters: the annotators had been taught the relevant linguistic distinctions first. The study therefore does not establish what every everyday reader would choose with no guidance. It does show that some supposed errors are not as clear-cut as a single answer key suggests.
The team then built 200 new examples in carefully matched pairs. The idea was to hold much of the wording and scenario steady while changing the detail that should alter the inference—such as whether an event had a defined path or whether its outcome was stated. They also broke the task into intermediate questions: What kind of event is this? Did it occur? Does the sentence establish that it reached its endpoint? Only then should the model choose a final label. This lets researchers ask not just whether the answer was wrong, but where the reasoning went off course.
One diagnostic probe produced a revealing pattern. Researchers gradually strengthened the information about an event, moving from intention to occurrence, interruption, explicit non-completion, and finally completion. On the controlled probe, four tested models reached 100% accuracy when completion was explicitly stated. But when the event was known to have occurred without evidence that it was completed, accuracy ranged from 0% to 20%. The models’ mistakes often appeared at the moment they treated “it happened” as enough to conclude “it reached its endpoint.” The authors call this Sufficiency Bias. It is different from simply hallucinating a completed outcome: in other checks, models did not claim the endpoint had been reached, yet still accepted the completed-event statement.
Prompts did not provide a simple escape hatch. Asking models to explain themselves or consider a counterfactual could change which label they chose, but a shift in answers did not always mean a better grasp of the sentence. Some instructions used terms such as “activity” and “goal-oriented action” without explaining how to distinguish them. In other tests, changing meaningful labels to arbitrary symbols altered performance. The researchers call this Surface-form Attraction: a model’s choice can be pulled by the wording of the answer labels, even when the underlying reasoning task stays the same.
Nor do the findings show that models have mastered event meaning. The researchers also found that identifying a phrase’s event type is itself difficult, and that models can stumble when translating intermediate judgments into a final label. Their larger-model experiments were promising in some respects, but performance varied by case; the authors describe their evidence as initial. The newly constructed examples are controlled by design, which makes them easier to diagnose but less like the full variety of natural conversation.
So the mystery is not solved by declaring either the models or the test innocent. A language model can confuse an event’s beginning with its completion; a benchmark can also mistake uncertainty for contradiction, or assign one rigid interpretation to a phrase people read in more than one way. The useful next question is more precise: when an answer goes wrong, did the model miss the event’s endpoint, misunderstand the phrase, or follow an answer key that leaves no room for reasonable doubt?
