Help wanted writing I6 library tests

While thinking about I6 support, I realized that we don’t actually have a set of unit tests for the I6 library. This is a definite hole; David G’s 6/12 library work has run into some regressions which are only noticed ad-hoc. (E.g. inform7.com/mantis/view.php?id=1885 .)

I figure we have a great set of tests in the DM4 exercise list. But they’re not complete: there’s no compilable sample game for most of them, and there’s no list of responses to test.

I put together a framework which can run them:

github.com/erkyrath/inform6-test

I’ve added sample games for the first 15 DM4 exercises, which leaves 123 to go. Anybody want to help fill them in? There’s also space for general library tests (not related to a specific exercise), and also tests that demonstrate known bugs.

These are specifically tests for the (classic) 6/11 library. Once that’s all filled in, David can make use of it for work on the (ongoing) 6/12 and future library releases.

See the README info on that page for details.

I’ll help with this. I have compilable versions of many of the exercises from a recent read through of the DM4 and some other test programs that I’ve written while evaluating the current library beta.

Ok, I have inform6-test up and running. I’m seeing a couple of test failures in the bugs subdir:

Compiling bugs/l-61102.inf...
Inform 6.33N for Unix (30th August 2015)
* general
<LiteralCheck "You get into the rowboat.">: not found
Compiling bugs/l-61116.inf...
Inform 6.33N for Unix (30th August 2015)
* general
<LiteralCheck "You look ridiculous waving yours...">: not found

Are these failures expected?

Edit: nm, I see that the README says that they’ll fail.

Great. And, yes, the bugs folder is for tests that fail under 6/11.

I’m pushing my work to the dm4 branch of my fork of your inform6-test repo. Not sure if you want a pull request per exercise, which seems like it could get annoying, or in larger chunks. Let me know what you prefer.

Do one pull a day, I guess. That way my repo is reasonably up to date in case somebody else jumps in.

I may adjust formatting or add comments on your files, so you’ll want to merge upstream occasionally.

I’d be happy to help set it up for continuous testing on Travis, but I wonder if it would be better to have the tests run from the lib repo? It would be more efficient to not have to track commits with submodules…

You mean continuous testing for 6/12 library development?

Once this set of tests is complete, it can be rearranged for 6/12 work. Some tests will probably have to change anyhow. Not worrying about that yet.

So your goal currently is just to get all the tests there against the stable 6/11 library, for which they should all pass? In that case, yeah it would make more sense to wait until it’s complete before doing continuous testing.

I agree that it would be best to first target 6/11. That way we would get the first complete picture of how the library actually behaves rather than how we want it to behave. It would be best to first do that with 6/11 rather than 6/12, which is something of a moving target at the moment.

Yep.

I added a counting test:

{count=3} Clap

…checks that the word “Clap” occurs at least three times in the output. (On different lines or on the same line, both okay.)

Nice, thanks. I pulled down those changes and rebased my in-progress stuff on top of it.

I’m working on ex32, and I’ve encountered some weirdness. The test program + tests that I wrote produce incorrect output on glulx, but not zcode, and hang glulxe.

This is my test program (with everything after the #END directive elided):

Constant Story "Example 32";
Constant Headline "^";

Include "Parser";
Include "VerbLib";

Object Field "Field"
  with description "This is the field.",
  has  light;

! Modified to remove Dan's dyslexia (already tested in ex31) and retain just
! his personalized 'examine' grammar.

Object -> Dan "Dyslexic Dan"
  with name 'dan' 'dyslexic',
       grammar [;
           if (verb_word == 'examine' or 'x//') {
               verb_wordnum++; return -'danx,';
           }
       ],
       orders [;
           Examine: "~What,~ says Dan, ~you want me to examine ",
                        (the) noun, "?~";
           Inv: "~That I can do,~ says Dan. ~I'm empty-handed.~";
           default: "~Don't know how,~ says Dan.";
       ],
       initial "Dyslexic Dan is here.",
  has  animate proper;

Object -> "cow pie"
  with name 'cow' 'pie',
       description "That's one healthy cow.",
       before [;
           Examine: ;
           default: "Nope.";
       ];

[ Initialise;
  location = Field;
];

Include "Grammar";

Verb 'danx,' * 'conscience' -> Inv;

#END; ! test

This is the output when compiled for zcode and run with unix frotz:

Example 32                                                                      
Release 1 / Serial number 160402 / Inform v6.33 Library 6/11 S                  
                                                                                
Field                                                                           
This is the field.                                                              
                                                                                
Dyslexic Dan is here.                                                           
                                                                                
You can also see a cow pie here.                                                
                                                                                
>dan, x pie                                                                     
"What," says Dan, "you want me to examine the cow pie?"                         
                                                                                
>dan, x conscience                                                              
"That I can do," says Dan. "I'm empty-handed."                                  
                                                                                
>dan, x                                                                         
(the cow pie)                                                                   
"What," says Dan, "you want me to examine the cow pie?"                         

This the output when compiled for glulx and run with glulxe, either standalone with glkterm or with remglk as part of inform6-test:

Example 32
Release 1 / Serial number 160402 / Inform v6.33 Library 6/11 S

Field
This is the field.

Dyslexic Dan is here.

You can also see a cow pie here.

>dan, x pie
"Don't know how," says Dan.

>dan, x conscience
"That I can do," says Dan. "I'm empty-handed."

>dan, x
[glulxe hangs indefinitely here.]

Are there any known issues that would account for this?

Nope, that’s a new bug. See, we’re already making progress…

Filed inform7.com/mantis/view.php?id=1890 .

Yep. In light of the interpreter hang, I’m thinking that we should update dotest.py to add a timeout arg to the select call in accept_output(). Let me know if you want me to put together a patch for this. For now, I’ve commented out the “dan, x” commands in my ex32.inf to avoid hanging the entire test script.

Ideally we’d be testing both zcode and glulx.

Agree. Seems like we could link the glk port of frotz against remglk and be good to go.

Is that the one in the garglk source tree? Gargoyle defaults to bocfel now, so I’d assume there are advantages to using it instead.

Also, are binaries for gluxer available anywhere? Or could anyone compile it on debian/ubuntu for me? My C skills are sadly still basically nonexistent.

You’re probably right. I did a search for “glk zcode interpreter” and found a reference to the glk version of frotz on this Gargoyle page.

Try this:

mkdir ifstuff
cd ifstuff
git clone https://github.com/erkyrath/glulxe
git clone https://github.com/erkyrath/remglk
cd remglk
make
cd ../glulxe
Open Makefile in the text editor of your choice.
You'll see five triplets of variables at the top, four of which are commented out with # marks. These tell glulxe which glk library to use.
Comment out the cheapglk ones and uncomment the remglk ones. They look like this:
    GLKINCLUDEDIR = ../remglk
    GLKLIBDIR = ../remglk
    GLKMAKEFILE = Make.remglk
Save the file and exit your text editor.
make
mv glulxe glulxer

Go for it.

I’ve moved ex23.inf to the bugs directory, since it’s a bug.

As for the idea of Zcode testing… yeah, I suppose so. I have fizmo compiled with remglk lying around.