Skip to content

Report per operation timings in CommandsPerformanceTest - #4244

Merged
vogella merged 1 commit into
eclipse-platform:masterfrom
vogella:commands-perf-timings
Aug 13, 2026
Merged

Report per operation timings in CommandsPerformanceTest#4244
vogella merged 1 commit into
eclipse-platform:masterfrom
vogella:commands-perf-timings

Conversation

@vogella

@vogella vogella commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

The four binding cache tests each measured one block and reported it through org.eclipse.test.performance. The performance database it reports to has not been configured for years, so commitMeasurements and assertPerformance were no-ops: the tests could neither fail nor report anything. This applies the same treatment OpenCloseEditorTest and EditorSwitchTest already received.

Each test now times its own work and prints min, median, 90th percentile and maximum via UIPerformanceTestUtil.reportTimings. A single cached look-up is far too fast to time individually, so the three hit tests time a batch and report the distribution over batches, warming up first and bounded by time instead of a fixed iteration count. The cache miss test rebuilds the binding set for every sample, since measuring a miss only works while the cache is cold and the previous version could therefore take a single sample and no more.

One real defect surfaced on the way. The reverse look-up asked for the bindings of the null command, which is loop invariant, so the JIT hoisted it out of the measured loop and it reported 0.8ns per look-up. It now cycles through commands that are actually bound and reports 3.1ns, in line with the forward look-up. The tests also assert that a look-up returned something, so they can fail.

Sample output, and note how tight the distributions are compared with the UI tests:

BindingCache hit hard [per 100000 look-ups]      n=200  min=0.45  p50=0.47  p90=0.50  max=0.51 (ms)
BindingCache hit hard reverse [per 100000 look-ups] n=200  min=0.31  p50=0.34  p90=0.41  max=1.09 (ms)
BindingCache hit soft [per 1000 look-ups]        n=200  min=7.56  p50=7.92  p90=8.19  max=12.16 (ms)
BindingCache miss large                          n=20   min=0.72  p50=1.01  p90=1.17  max=1.63 (ms)

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Test Results

   858 files     858 suites   45m 11s ⏱️
 8 160 tests  7 917 ✅ 243 💤 0 ❌
20 382 runs  19 728 ✅ 654 💤 0 ❌

Results for commit 05f42f8.

♻️ This comment has been updated with latest results.

The four binding cache tests measured one block each and reported it
through org.eclipse.test.performance. The performance database it reports
to has not been configured for years, which made commitMeasurements and
assertPerformance no-ops and left the tests unable to fail or to report
anything.

Each test now times its own work and prints min, median, 90th percentile
and maximum via UIPerformanceTestUtil.reportTimings. A single cached
look-up is far too fast to time individually, so the three hit tests time
a batch and report the distribution over batches, warming up first and
bounded by time rather than by a fixed iteration count.

The cache miss test rebuilds the binding set for every sample. Measuring
a miss only works while the cache is cold, so the previous version could
take a single sample and no more.

The reverse look-up asked for the bindings of the null command, which is
loop invariant and was hoisted out of the measured loop, reporting 0.8ns
per look-up. It now cycles through commands that are actually bound and
reports 3.1ns, in line with the forward look-up. The tests also assert
that a look-up returned something, so they can fail.
@vogella
vogella force-pushed the commands-perf-timings branch from 187bbe2 to 05f42f8 Compare August 13, 2026 13:45
@vogella
vogella merged commit df53a41 into eclipse-platform:master Aug 13, 2026
18 checks passed
@vogella
vogella deleted the commands-perf-timings branch August 13, 2026 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant