A faint line crosses a star field. It might be a fast-moving asteroid, a satellite, a meteor, or a flaw in the camera. For astronomers trying to sift through huge numbers of images, the first question is simpler: is that streak really there, and where does it begin and end? A detector that answers well on one telescope’s images may stumble when the camera, sky background, or image noise changes. A new benchmark puts that problem under a brighter light.
Jiayou He and Jessica Yao assembled NEO-Bench, a collection of 8,376 images drawn from five different sources. The images include Hubble observations, pictures from a Stellina smart telescope, meteor-monitoring images, and other telescope datasets. The point was not merely to gather more examples. It was to preserve the differences among them and test whether a detection method trained on some sources could work on a source it had never seen.
That distinction matters because an astronomical streak is not a diagnosis. A real object moving across the field during an exposure leaves a trail, but so can a cosmic ray or a detector defect. Satellite tracks, meteors, and debris can also resemble the targets a search is meant to find. NEO-Bench therefore treats an elongated trace accepted by its source dataset as a positive streak, rather than claiming every positive is a near-Earth object. The researchers note that the available data do not consistently identify which trails belong to orbit-confirmed near-Earth objects.
The benchmark brings together two related tasks: decide whether an image contains a streak, then draw a box around it so a person or later software can inspect it. The collection contains 5,969 streak-positive images and 2,407 negative ones. To check the labels, both authors independently reviewed 100 positive images from each source. They agreed on 492 of the 500 sampled images, or 98.4 percent. That audit supports the quality of the sample, while leaving room for ambiguous images and differences in how the original datasets were assembled.
Next came the test of portability. The researchers compared four approaches: two classical line-finding techniques, Hough and Radon; a method that models a streak as a blurred line; and YOLO26L, a learned object detector. In a within-source test, a method was trained or tuned using one dataset and evaluated on its own source. In the more demanding leave-one-source-out test, the method used the other sources for training or calibration, then faced the held-out source. It was a way to ask whether performance survived a change of telescope, not just whether a detector could master familiar examples.
On familiar images, the learned detector looked formidable. YOLO26L had an average F1 score of 0.987 for deciding whether an image contained a streak, and 0.763 for locating streaks with boxes. The F1 score combines precision—the share of alarms that are right—with recall—the share of actual streaks found. A high score on a familiar dataset is useful, but it does not answer the question a distributed observing network cares about: what happens when the next image comes from somewhere else?
That is where the tidy picture broke. Across all evaluated methods, image-level F1 fell in 14 of 20 held-out-source comparisons. For localization, the score fell in 13 of 20. The strongest model still had image-level declines on four of the five unseen sources. More strikingly, its localization score dropped on all five, by an average of 0.263. A detector might still notice that something crossed the frame, in other words, while becoming less dependable at pointing to the trace itself.
The result exposes a subtle trap in testing. A model can appear nearly flawless when the training and test images share a source, yet carry over that success imperfectly to unfamiliar data. The cameras differ in resolution, exposure, background, noise, and the shape or brightness of streaks. Even the labeling history can vary. Those differences are not minor decoration around the science; they shape the evidence a detector learns to recognize. NEO-Bench makes that shift measurable by setting sources against one another rather than blending them into a single undifferentiated pile.
The benchmark also has limits. Its images are not a uniform, orbit-confirmed census of near-Earth objects, and some source test sets contain few or no negative images, making certain comparisons easier to overread. The paper evaluates a set of methods and datasets; it does not report a newly discovered asteroid or show that a particular object would have been missed in an operational survey. Its warning is narrower, and more useful for building tools: strong results within one source are not proof of reliable performance across observatories.
That makes NEO-Bench a new kind of instrument: not a telescope, but a way to stress-test the software that may eventually scan telescope images for moving objects. For planetary defense, broad coverage could depend on many instruments contributing observations. A detector that only works after being specially tuned for each camera would be harder to deploy widely. By showing where transfer breaks down, the benchmark gives future researchers a clearer target: build streak finders that keep their bearings when the sky is viewed through different hardware. The first step toward a more dependable search is discovering just how much the view can change.
