Automated testing with cheapglk

I’m trying to set up some automated tests for Kerkerkruip with the cheapglk build of git that’s included in the Inform 7 Linux CLI package. I’m running it with this node.js script, but am having quite a bit of trouble. I’m hoping some of you might have some ideas.

  1. When I run the interpreter normally it prints stuff to the screen as it goes. When run through node.js, I only get a single stream event with all of the text, even though it takes several minutes to run through the test case. node.js’ streams are meant to be very low overhead, they’re not meant to buffer things for you, they’re meant to just give you everything that comes through stdout. Any ideas why it waits until the next input?
  2. When run through the script it will not accept the command to start the game at Kerkerkruip’s menu. It will accept other commands, to go to the menu or play a specific seed, but the ‘n’ command will not work. This seems very bizarre. I guess this must be a problem in our Inform 7 code.
  3. When writing to stdin like this, do I need to wait for input to be requested, or can I just write it all at once? Or should I even put a delay in till after the output data is printed before doing so? (This is what the sendCommand function is for, though it’s not in use currently. It didn’t help with the ‘n’ not working btw.)
  4. Does anyone know of any other interpreters which could be used similarly (ie read/write through the standard streams?) Or is there one that I could give a list of commands non-interactively to run through? That would, I presume, solve the ‘n’ problem.

CheapGlk accepts character input and line input using the same call (fgets). This means that you need to add a newline after the “n” for keystroke input. It looks like you’re doing that, though.

The output is somewhat buffered by the stdio library. I don’t know what the parameters of that are – I’d expect every newline to flush the output stream. So maybe newline translation is the issue?

You’re using a build of git. I’ve only tested glulxe+cheapglk. I see no reason to believe that git would be buggy, but it might have done something to the cheapglk library that I’m not familiar with.

No delays. Cheapglk won’t notice them and it doesn’t use any itself.

Writing a non-interactive test with cheapglk is something of a nuisance. There’s no reliable way to know that cheapglk has finished output. When writing such tests, I continue reading from stdout until “\n>” shows up, and I assume that’s a new prompt. If the game prints a different prompt, or prints “>” at the beginning of a line for other reasons, the test goes off the rails.

An alternative is to use remglk. With this library, every command cycle is exactly one JSON stanza in and one JSON stanza out. This makes the test more reliable. Plus you can reliably distinguish character from line input, distinguish output from different windows, and so on. But obviously you have to do extra work to parse the JSON.

My test framework is at eblong.com/zarf/plotex/regtest.py . Note the functions GameStateCheap and GameStateRemGlk, which manage a subprocess in the two modes I’ve described.

Thanks for your thoughts Zarf. I have tried it with glulxe too, with no differences at all.

I’ve just had the thought that we could make it entirely non interactive, including quitting at the end. That would make things considerably simpler.

Here’s a regtest.py script that launches Kerkerkruip and then quits. Obviously this assumes that there are no existing save, data, or transcript files.

(“glulxer” is just my local copy of glulxe+remglk. You’ll have to compile that to use this.)

** game: Kerkerkruip.gblorb
** interpreter: glulxer
** remformat: yes

* startup

Kerkerkruip has a mode optimised for those using a screen reader.
>{char} n

>{char} 32
Malygris, the Wizard of Kerkerkruip!

>{fileref_prompt} test-transcript
Kerkerkruip 10
An IF roguelike by Victor Gijsbers
The Kerkerkruip team 
Free software 
Afterword by Victor Gijsbers 
Press a key to go back.

>{char} 32
Kerkerkruip 10
An IF roguelike by Victor Gijsbers
Entrance Hall

> l
Entrance Hall

> quit
Are you sure you want to quit?

> y

I ended up making Kerkerkruip run them non-interactively. I was able to solve the buffering problem with the unbuffer program from expect-dev.

If anyone’s interested, you can see the tests run at travis-ci.org/i7/kerkerkruip/