Report per operation timings in CommandsPerformanceTest - #4244
Merged
Conversation
Contributor
Test Results 858 files 858 suites 45m 11s ⏱️ Results for commit 05f42f8. ♻️ This comment has been updated with latest results. |
The four binding cache tests measured one block each and reported it through org.eclipse.test.performance. The performance database it reports to has not been configured for years, which made commitMeasurements and assertPerformance no-ops and left the tests unable to fail or to report anything. Each test now times its own work and prints min, median, 90th percentile and maximum via UIPerformanceTestUtil.reportTimings. A single cached look-up is far too fast to time individually, so the three hit tests time a batch and report the distribution over batches, warming up first and bounded by time rather than by a fixed iteration count. The cache miss test rebuilds the binding set for every sample. Measuring a miss only works while the cache is cold, so the previous version could take a single sample and no more. The reverse look-up asked for the bindings of the null command, which is loop invariant and was hoisted out of the measured loop, reporting 0.8ns per look-up. It now cycles through commands that are actually bound and reports 3.1ns, in line with the forward look-up. The tests also assert that a look-up returned something, so they can fail.
vogella
force-pushed
the
commands-perf-timings
branch
from
August 13, 2026 13:45
187bbe2 to
05f42f8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The four binding cache tests each measured one block and reported it through
org.eclipse.test.performance. The performance database it reports to has not been configured for years, socommitMeasurementsandassertPerformancewere no-ops: the tests could neither fail nor report anything. This applies the same treatmentOpenCloseEditorTestandEditorSwitchTestalready received.Each test now times its own work and prints min, median, 90th percentile and maximum via
UIPerformanceTestUtil.reportTimings. A single cached look-up is far too fast to time individually, so the three hit tests time a batch and report the distribution over batches, warming up first and bounded by time instead of a fixed iteration count. The cache miss test rebuilds the binding set for every sample, since measuring a miss only works while the cache is cold and the previous version could therefore take a single sample and no more.One real defect surfaced on the way. The reverse look-up asked for the bindings of the
nullcommand, which is loop invariant, so the JIT hoisted it out of the measured loop and it reported 0.8ns per look-up. It now cycles through commands that are actually bound and reports 3.1ns, in line with the forward look-up. The tests also assert that a look-up returned something, so they can fail.Sample output, and note how tight the distributions are compared with the UI tests: