Cap the Spark benchmark chase at one or two wins - the deliverable is bits in RLC-AI on Portal, not a leaderboard listing
Situation
Across four channels Peter set the scope of the NVIDIA Spark benchmark work ahead of the Tuesday announcement. To Nathan Blackham and Brett Mastbergen he said not to spend too much time there, that all CIQ really needs is one test to point to, two is awesome, more is not necessarily better, and that after one or two what is actually needed is the right bits baked into RLC-AI and available on Portal for download, at which point CIQ can declare victory. He also directed benchmark selection by relative rather than absolute result - pick whichever benchmark showcases Rocky best against the other distros on the leaderboard and can produce results quickly, because this is not about the raw number. In the group DM with Brian Dawson, Bjorn Hovland and Lindsay Aamodt he corrected the plan: the leaderboard is third party, not controlled by NVIDIA, CIQ cannot guarantee any posting there by any date, so make plans that involve self-reported benchmarks. With Lindsay and Bjorn he put the third-party publication odds at about fifty percent while calling confidence in CIQ own benchmarks high.
Reasoning
The announcement needs a defensible proof point, not a sweep. One or two wins on benchmarks CIQ chose is enough for marketing to work with and cheap to produce, while chasing every front burns Linux Engineering time on something with no additional commercial return. The leaderboard is the part CIQ does not control, so Peter refused to let the launch plan depend on it and moved the dependency to self-reported numbers plus a downloadable artifact - both of which are inside CIQ own delivery chain.
Additional Context
Max Spevack produced the original tuned build and results but is hard to reach, which is precisely why Peter would not let the plan hinge on a third-party publication Max was midway through negotiating. The reproduction came back positive on Sept 10 - Brett reproduced Max results and CIQ wins the leaderboard race on the benchmark Max tuned for.
Observed Evidence
Four direct quotes across four separate conversations in the window, each setting a different edge of the same scope decision - how many tests, which test, who publishes, and what counts as done.
Matching Patterns
Confidence Breakdown
Reasoning Depth Analysis
People Involved
Source
reflection
AI Confidence
94%
Related Context
slack
All good. But genuinely, do not spend too much time there. All we really need is one test to point to. Two is awesome. More is... not necessarily better. After we have 1 or 2, what we really need is the right bits baked into RLC-AI, and available on Portal for download. At that point we can declare victory.
slack
Understood. One tweak. The leaderboard is third party, not controlled by Nvidia. And we cannot guarantee any posting there by any date. So make some plans that involve self-reported benchmarks.
slack
We want the one that showcases the best for rocky compared to other distros listed on the leaderboard... This is not about the raw number. It is about the relative numbers.
slack
Our own benchmarks - high. Third party published... I just do not know why they are not there NOW and max is so hit-miss on getting back to me on stuff like that. 50 percent?
Outcome
No outcome recorded yet.
Decision ID: e703341b-378f-4816-b61c-93254a3286e1