Cap the Spark benchmark chase at one or two wins - the deliverable is bits in RLC-AI on Portal, not a leaderboard listing

September 11, 2026 at 3:04 PMstrategymedium

Situation

Across four channels Peter set the scope of the NVIDIA Spark benchmark work ahead of the Tuesday announcement. To Nathan Blackham and Brett Mastbergen he said not to spend too much time there, that all CIQ really needs is one test to point to, two is awesome, more is not necessarily better, and that after one or two what is actually needed is the right bits baked into RLC-AI and available on Portal for download, at which point CIQ can declare victory. He also directed benchmark selection by relative rather than absolute result - pick whichever benchmark showcases Rocky best against the other distros on the leaderboard and can produce results quickly, because this is not about the raw number. In the group DM with Brian Dawson, Bjorn Hovland and Lindsay Aamodt he corrected the plan: the leaderboard is third party, not controlled by NVIDIA, CIQ cannot guarantee any posting there by any date, so make plans that involve self-reported benchmarks. With Lindsay and Bjorn he put the third-party publication odds at about fifty percent while calling confidence in CIQ own benchmarks high.

Reasoning

The announcement needs a defensible proof point, not a sweep. One or two wins on benchmarks CIQ chose is enough for marketing to work with and cheap to produce, while chasing every front burns Linux Engineering time on something with no additional commercial return. The leaderboard is the part CIQ does not control, so Peter refused to let the launch plan depend on it and moved the dependency to self-reported numbers plus a downloadable artifact - both of which are inside CIQ own delivery chain.

Additional Context

Max Spevack produced the original tuned build and results but is hard to reach, which is precisely why Peter would not let the plan hinge on a third-party publication Max was midway through negotiating. The reproduction came back positive on Sept 10 - Brett reproduced Max results and CIQ wins the leaderboard race on the benchmark Max tuned for.

Observed Evidence

Four direct quotes across four separate conversations in the window, each setting a different edge of the same scope decision - how many tests, which test, who publishes, and what counts as done.

Matching Patterns

40%
Protect Engineering Capacity(caps the work rather than letting marketing appetite set engineering scope)
35%
Purpose Is the Decision Procedure(restates what the benchmark is for - a proof point - rather than arguing individual test merits)
30%
Reclassify to Route - Decide Who Owns It, Then Give a Criterion Not a Verdict(gives a selection criterion - best relative showing, fastest to produce - rather than naming the benchmark)

Confidence Breakdown

35/35
Evidence
24/30
Pattern
20/20
Source
15/15
Corroboration

Reasoning Depth Analysis

Org Signal:Tells Linux Engineering that good enough and shipped beats comprehensive, and tells Marketing that engineering will supply a proof point but will not accept a dependency on a third party CIQ does not control.
Who Affected:Brett and Nathan get a bounded task instead of an open benchmark program. Lindsay gets a press plan that can survive the leaderboard never publishing. Dawson had to replan the announcement around self-reported numbers.
Precedent:Establishes that launch commitments get scoped to what is inside CIQ own delivery chain, and that a dependency on an outside publisher is a planning risk to be designed out rather than chased.
Consequences:Real: the announcement plan changed to self-reported benchmarks, and Peter told Brett explicitly to stop expanding the test matrix.
Timing:Now because the Tuesday announcement forced the question, and because Max being hard to reach made the third-party path an unbounded wait rather than a short one.

Related Context

💬
Group DM with Nathan Blackham and Brett Mastbergen

slack

All good. But genuinely, do not spend too much time there. All we really need is one test to point to. Two is awesome. More is... not necessarily better. After we have 1 or 2, what we really need is the right bits baked into RLC-AI, and available on Portal for download. At that point we can declare victory.

💬
Group DM with Brian Dawson, Bjorn Hovland, Lindsay Aamodt

slack

Understood. One tweak. The leaderboard is third party, not controlled by Nvidia. And we cannot guarantee any posting there by any date. So make some plans that involve self-reported benchmarks.

💬
Group DM with Nathan Blackham and Brett Mastbergen

slack

We want the one that showcases the best for rocky compared to other distros listed on the leaderboard... This is not about the raw number. It is about the relative numbers.

💬
Group DM with Lindsay Aamodt and Bjorn Hovland

slack

Our own benchmarks - high. Third party published... I just do not know why they are not there NOW and max is so hit-miss on getting back to me on stuff like that. 50 percent?

Outcome

No outcome recorded yet.

Decision ID: e703341b-378f-4816-b61c-93254a3286e1