intfiction.org

Hello all! I am pleased to announce the release of my extension Benchmarking. It will soon be on the Inform 7 website, for now you can download it from http://curiousdannii.github.com/if/exte ... arking.i7x

Benchmarking is a performance test framework for I7. It has two types of users in mind:

1. Story and extension authors can use Benchmarking to compare alternatives for some slow programming task.
2. Interpreter authors can use Benchmarking to compare their interpreter with others, as well as to compare interpreter updates to see whether they have a performance benefit or deficit.

It is easy to set up, as the example shows for how to test text matching. If the tests do not run in the built-in I7 interpreter, then either use output.ulx from the Build folder with another interpreter, or (if you're on Windows) download the updated 6G60 build.

Any comments on this extension will be appreciated!

-------------------

I am also looking for contributions for a performance test suite. Has your I7 story ever run too slowly? Can you identify what the problem is? If so, then I'd love that example for the test suite. Please post here so that we can coordinate and not end up with 10 different scope tests.

There is '-)' missing in your example. Looks pretty neat otherwise.

It's an Inform 7 bug: http://inform7.com/mantis/view.php?id=377

Very nice! Thank you for this!

FYI, searching a 100-member list for a given value takes roughly 28 times as long (assuming that it's toward the end of the list) as simply retrieving the value from a given slot of the same list:



Not earth-shattering, but it is interesting to me how much longer a search takes versus a simple access.


I do have a couple of UI suggestions:

1) The game closes down immediately a few moments after finishing the last test. In Gargoyle, this means that the game window closes too. Please offer the user the chance to either rerun the tests or quit at the end, and don't quit without the pressing of an explicit key. It takes a long time to run any of the simple tests I'm trying, and it's really annoying to have the window just close in your face when they finish!

2) Currently it is a bit hard to customize the conditions for a test. For example, if I need to set up a list full of random data for my test(s) to work on, I need to do something like this:



What's difficult? First, the user needs to remember to add "first" to the when play begins rule to ensure that his rule prints before the program is taken over by the benchmarker; changing the benchmarker control rule to "last when play begins" would eliminate this problem. Second, if the user wants to print text in this code block, he needs to stick something like "wait for any key" in there; this calls glk_select. Most users won't realize that this is necessary and will bang their heads on the wall trying to figure out why their introductory text is not printing. Maybe a "press any key to begin" in the benchmarker's when play begin rule would fix this?

_________________
Glimmr: Advanced Graphics for I7
blog | download | bug-tracker

Hm, that shouldn't happen. It's the interpreter's responsibility to keep any last-printed Glk output visible to the user after glk_exit().

Looking into it, I was wrong--the window does stay open, but only until the user presses any key. So twice today I accidentally pressed a random key while trying to do something else and the window shut down. I still think it's a good idea to offer the user an explicit option to end the execution. EDIT: Aha! Using the scroll wheel apparently counts as a keypress, at least in the Mac OSX Gargoyle, so it was attempting to use the mouse to view the scrollback buffer that causes the window to close.

Another UI idea: Allow the user to request a transcript so that the data can be dumped to a file.

_________________
Glimmr: Advanced Graphics for I7
blog | download | bug-tracker

Hi Erik, I've added your three suggestions. Will post again when I upload the update.

In the future I plan to make it write out to a data file, and give you the option to compare with other interpreters. I'll remove the transcript option then.

Great, thanks. I've got what appear to be some intermittent issues with memory on some simple examples I've been playing with. I'll post again if I come up with something coherent to report.

_________________
Glimmr: Advanced Graphics for I7
blog | download | bug-tracker

Yes something is going wrong with the sampling - 93902 samples seems far too high to me, though it could be possible if your timer resolution is very good, ie. 50µs. I'm only getting 1ms in Windows Gargoyle . Can you check what "the minimum timer resolution" is, with an After running the benchmark framework rule? It stores the samples in a list, so with that many samples it's sure to cause problems.

The framework takes at least 5 samples, and runs each test for at least 5 seconds as well (mainly because that's what benchmark.js does.) But if too many samples is a problem I can limit that too. I'm sure 100 would be more than fine.

Would you also be interested in seeing the raw number of times a function was run?

If you compile Gargoyle with -DGARGLK_PROFILE_TIME, it uses a high resolution timer for glk_current_time calls. Resolution varies by platform but 50µs is in the ballpark.

Erik has access to a build like that, so possibly he's using it here.