intfiction.org

The Interactive Fiction Community Forum
It is currently Thu Sep 18, 2014 4:42 pm

All times are UTC - 6 hours [ DST ]




Post new topic Reply to topic  [ 25 posts ]  Go to page 1, 2, 3  Next
Author Message
PostPosted: Mon Jan 16, 2012 11:18 pm 
Offline
User avatar

Joined: Wed Oct 14, 2009 4:02 am
Posts: 1549
Hello all! I am pleased to announce the release of my extension Benchmarking. It will soon be on the Inform 7 website, for now you can download it from http://curiousdannii.github.com/if/exte ... arking.i7x

Benchmarking is a performance test framework for I7. It has two types of users in mind:

  1. Story and extension authors can use Benchmarking to compare alternatives for some slow programming task.
  2. Interpreter authors can use Benchmarking to compare their interpreter with others, as well as to compare interpreter updates to see whether they have a performance benefit or deficit.

It is easy to set up, as the example shows for how to test text matching. If the tests do not run in the built-in I7 interpreter, then either use output.ulx from the Build folder with another interpreter, or (if you're on Windows) download the updated 6G60 build.

Any comments on this extension will be appreciated!

-------------------

I am also looking for contributions for a performance test suite. Has your I7 story ever run too slowly? Can you identify what the problem is? If so, then I'd love that example for the test suite. Please post here so that we can coordinate and not end up with 10 different scope tests.


Last edited by Dannii on Fri Feb 17, 2012 11:12 pm, edited 3 times in total.

Top
 Profile Send private message  
 
PostPosted: Tue Jan 17, 2012 12:16 pm 
Offline
User avatar

Joined: Sat Dec 17, 2011 10:39 am
Posts: 76
There is '-)' missing in your example. Looks pretty neat otherwise.


Top
 Profile Send private message  
 
PostPosted: Tue Jan 17, 2012 12:49 pm 
Offline
User avatar

Joined: Wed Oct 14, 2009 4:02 am
Posts: 1549
It's an Inform 7 bug: http://inform7.com/mantis/view.php?id=377


Top
 Profile Send private message  
 
PostPosted: Tue Jan 17, 2012 2:01 pm 
Offline

Joined: Thu Oct 22, 2009 4:31 pm
Posts: 1227
Very nice! Thank you for this!

FYI, searching a 100-member list for a given value takes roughly 28 times as long (assuming that it's toward the end of the list) as simply retrieving the value from a given slot of the same list:

Quote:
Test results
Searching a simple list:
202.82µs ±0.14% (24652 samples)
Selecting from a simple list:
7.05µs ±0.58% (93902 samples)

Total running time: 10000080µs


Not earth-shattering, but it is interesting to me how much longer a search takes versus a simple access.


I do have a couple of UI suggestions:

1) The game closes down immediately a few moments after finishing the last test. In Gargoyle, this means that the game window closes too. Please offer the user the chance to either rerun the tests or quit at the end, and don't quit without the pressing of an explicit key. It takes a long time to run any of the simple tests I'm trying, and it's really annoying to have the window just close in your face when they finish!

2) Currently it is a bit hard to customize the conditions for a test. For example, if I need to set up a list full of random data for my test(s) to work on, I need to do something like this:

Code:
First when play begins:
	repeat with x running from 1 to 99:
		add a random number between 1 and 99 to search list;
	add 100 at entry 90 in search list;
	say "The search list is [number of entries in search list] items long: ";
	say search list in brace notation;
	wait for any key.


What's difficult? First, the user needs to remember to add "first" to the when play begins rule to ensure that his rule prints before the program is taken over by the benchmarker; changing the benchmarker control rule to "last when play begins" would eliminate this problem. Second, if the user wants to print text in this code block, he needs to stick something like "wait for any key" in there; this calls glk_select. Most users won't realize that this is necessary and will bang their heads on the wall trying to figure out why their introductory text is not printing. Maybe a "press any key to begin" in the benchmarker's when play begin rule would fix this?

_________________
Glimmr: Advanced Graphics for I7 (Build 6G60 only, at least until somebody decides they want to update them!)
blog | download | bug-tracker


Last edited by Erik Temple on Tue Jan 17, 2012 5:08 pm, edited 1 time in total.

Top
 Profile Send private message  
 
PostPosted: Tue Jan 17, 2012 3:46 pm 
Offline

Joined: Sat Jan 23, 2010 4:56 pm
Posts: 3381
Quote:
In Gargoyle, this means that the game window closes too.


Hm, that shouldn't happen. It's the interpreter's responsibility to keep any last-printed Glk output visible to the user after glk_exit().


Top
 Profile Send private message  
 
PostPosted: Tue Jan 17, 2012 4:02 pm 
Offline

Joined: Thu Oct 22, 2009 4:31 pm
Posts: 1227
zarf wrote:
Quote:
In Gargoyle, this means that the game window closes too.


Hm, that shouldn't happen. It's the interpreter's responsibility to keep any last-printed Glk output visible to the user after glk_exit().


Looking into it, I was wrong--the window does stay open, but only until the user presses any key. So twice today I accidentally pressed a random key while trying to do something else and the window shut down. I still think it's a good idea to offer the user an explicit option to end the execution. EDIT: Aha! Using the scroll wheel apparently counts as a keypress, at least in the Mac OSX Gargoyle, so it was attempting to use the mouse to view the scrollback buffer that causes the window to close.

Another UI idea: Allow the user to request a transcript so that the data can be dumped to a file.

_________________
Glimmr: Advanced Graphics for I7 (Build 6G60 only, at least until somebody decides they want to update them!)
blog | download | bug-tracker


Last edited by Erik Temple on Tue Jan 17, 2012 5:57 pm, edited 1 time in total.

Top
 Profile Send private message  
 
PostPosted: Tue Jan 17, 2012 5:55 pm 
Offline
User avatar

Joined: Wed Oct 14, 2009 4:02 am
Posts: 1549
Hi Erik, I've added your three suggestions. Will post again when I upload the update.

In the future I plan to make it write out to a data file, and give you the option to compare with other interpreters. I'll remove the transcript option then.


Top
 Profile Send private message  
 
PostPosted: Tue Jan 17, 2012 5:59 pm 
Offline

Joined: Thu Oct 22, 2009 4:31 pm
Posts: 1227
Great, thanks. I've got what appear to be some intermittent issues with memory on some simple examples I've been playing with. I'll post again if I come up with something coherent to report.

_________________
Glimmr: Advanced Graphics for I7 (Build 6G60 only, at least until somebody decides they want to update them!)
blog | download | bug-tracker


Top
 Profile Send private message  
 
PostPosted: Tue Jan 17, 2012 6:48 pm 
Offline
User avatar

Joined: Wed Oct 14, 2009 4:02 am
Posts: 1549
Yes something is going wrong with the sampling - 93902 samples seems far too high to me, though it could be possible if your timer resolution is very good, ie. 50µs. I'm only getting 1ms in Windows Gargoyle :(. Can you check what "the minimum timer resolution" is, with an After running the benchmark framework rule? It stores the samples in a list, so with that many samples it's sure to cause problems.

The framework takes at least 5 samples, and runs each test for at least 5 seconds as well (mainly because that's what benchmark.js does.) But if too many samples is a problem I can limit that too. I'm sure 100 would be more than fine.

Would you also be interested in seeing the raw number of times a function was run?


Top
 Profile Send private message  
 
PostPosted: Tue Jan 17, 2012 7:22 pm 
Offline

Joined: Tue Apr 27, 2010 1:02 pm
Posts: 801
Dannii wrote:
Yes something is going wrong with the sampling - 93902 samples seems far too high to me, though it could be possible if your timer resolution is very good, ie. 50µs.


If you compile Gargoyle with -DGARGLK_PROFILE_TIME, it uses a high resolution timer for glk_current_time calls. Resolution varies by platform but 50µs is in the ballpark.

Erik has access to a build like that, so possibly he's using it here.


Top
 Profile Send private message  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 25 posts ]  Go to page 1, 2, 3  Next

All times are UTC - 6 hours [ DST ]


Who is online

Users browsing this forum: Google [Bot] and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group