It seems to me that you have only a vague idea of what you're trying to accomplish. Perhaps you would even agree with that statement. You're gathering data to help you figure out what to do.
But I think you're already jumping to a big conclusion that data
is what you need here.
For one thing, the data you're looking for will be extremely hard to come by. It's hard enough for us to attract people to write games in ChoiceScript, and we're paying
our writers. Even restricting the question to Python users, why should people write games in your system instead of writing them in Ren'Py?
Even worse, the data you're trying to gather has little or nothing to do with the goals you've vaguely identified.
2) I really don't understand how the data you're gathering is relevant to the problem you're trying to solve.
Being that this is a qualitative study rather than a quantitative study, the data is mainly going to consist of decomposing stories into their component parts from a qualitative perspective. Hopefully, as I get a better feel for this genre, I'll be able to formulate questions that will allow me to design tests that have a better statistical foundation.
"Decomposing stories into their component parts from a qualitative perspective" is literary analysis. How familiar are you with modern literary theory?
Good literary analysis is hard; doing it with statistics is like trying to do it with both hands and feet tied behind your back. The history of 20th century literary theory has been the systematic disposal of the idea that literary analysis can be done in any systematic way.
If you want to do literary analysis, you'd be better off just reading a bunch of published multiple-choice interactive fiction and writing about it. There's plenty of it out there, and it won't require you to get people to re-write their games in your system.
That does not sound to me like a CYOA system on crack. That sounds to me like artificial intelligence. In particular, if you can "write realistic-sounding dialog on the fly" in response to speech recognition, you can pass the Turing Test.
I've seen this sort of straw man argument used against developing new technology before; however, I completely understand your skepticism. You are correct, it sounds like Artificial Intelligence and with modern neural networks and ever expanding knowledge in genetic algorithms, we're not as far as you think we are when it comes to developing machines that pass the Turing Test. I'm not proposing to develop a machine that can pass the Turing Test (although, that would be pretty bad ass), I'm proposing a much simpler project. That project being to develop a tool(s) for game writers to bring to the game development table in a very real way... Something on par with a game maker toolkit for designers or a Maya for artists.
I was making a very specific claim with which I think you can't really argue: the Turing Test is a test in which a machine is called upon to "write realistic-sounding dialog on the fly" in response to the tester's conversation. If you can do that, in response to speech recognition
, (which makes it even harder than the standard Turning Test which is performed purely via text), you've passed the Turing Test, not figuratively, but literally.
But let's say you just want to build a better toolkit for game designers. Traditionally, you'd do that not by writing your own system from scratch and then trying to attract writers, but by writing games in the existing systems and talking to writers who are already working in them. Ask them what they wish they had, then code it.
As it stands, you don't know where you're going, but you're going the wrong way.