Updates during the competition, aka the elephant in the room

Something that a lot of authors, particularly new authors, may not be aware of is that a non-trivial amount of people have mentioned, in public, that they either plan to or are tempted to automatically vote 1 for every game that updated during the comp, regardless of why they updated. They’re buried in page 11 of old forum threads here as well as external forums, but they exist. (The posts are written very much in “oh me, oh my, should I do this terrible thing? Of course I won’t, but oh, the temptation! However can one resist?” tone, one that strikes me as rather disingenuous.) Given that several authors have already pushed updates, I thought it best that those who have not should have full knowledge of this, if they would like to avoid 1 votes.

It’s instructive to think about just what this would entail. Fixed one typo? 1. Added a tester to the credits who wasn’t there before? 1. Got caught by that Quixe interpreter bug (emphasis on interpreter; it has nothing to do with authors’ code or effort) that most people hadn’t heard of until September 30 and that is only really noticeable once you’d already uploaded a game? 1. All of these are real examples from real updates in this year’s comp.

For those of you who weren’t following the aforementioned 11-page forum threads, this is a significant piece of unsettled discussion and unfinished community business that is now returning to roost. If the comp provides an option to update one’s games during the comp, it is fair to also ask that authors not be penalized for taking that option. I don’t think this is an unreasonable stance. The rules for judges don’t address this explicitly, but it would seem to fall under the purview of rule 7, “Every rating asserts that the judge who submitted it made a good-faith effort to actually play that game as intended. The competition organizers reserve the right to disqualify any ratings that appear to have been submitted under any other circumstances.”"

Wait, what? People are saying they’d do that? That’s… wow, no.

The only reason I’d vote a 1 is if the game is busted to the point it doesn’t work, it’s not IF (like… it’s an FPS or something), or it’s grossly offensive. I’ll even try it again and change my rating if it was busted, then the author updates it so it runs.

Now, I don’t think it’s unfair to play and rate a work based on an older un-updated version if that’s what was available when you got around to playing it, but I also don’t understand why an update would be considered a bad thing in any way.

This is all news to me. Confusing news. :neutral_face:

Has Rule 7 ever been invoked?

My impression is that Rule 7 is being reserved for only obvious and egregious cases of inappropriate judging. I think the organizers will only use it if the evidence is as clear as a smoking gun, and the number of votes have to be significant as well. I believe Rule 7 won’t be used to moderate a minority of community members with prejudices, even if they publicly blog that they hate (and 1-vote) mid-comp updates, or Twine, or non-downloadable games, or whatever.

Here’s what I propose:

  1. Authors who are opposed to the practice of 1-voting updated games should submit an update today, even if it changes nothing, to stand in solidarity with other authors.

  2. I’m a bit surprised/disappointed to see that there’s still no way to submit text comments with IFComp votes, which I think could help to move this conversation forward. (Maybe next year.) If authors could point to results and say, “It says here I got eight 1-votes for my mid-comp update,” then there’d at least be evidence to discuss, as opposed to speculation.

If anything, an updated game seems to be an indicator of quality; it looks like all the ones I’ve played and really enjoyed so far have updated, while two games released with serious implementation problems haven’t been touched. I would guess that updating the game mid-comp is correlated with paying attention to reviews and player feedback and striving to fix errors at all stages of the process, which is why those games were already in a pretty solid state when they were released.

Other people here are correct, 1 is reserved for broken and unplayable games.

The repercussions of an author entering an unpolished game are already ingrained in the system. Judges are not required to replay/re-vote on games that were updated. Updating the game is only for the benefit of judges who have not played it yet, to enhance their experience, and it also allows an author to increase their potential score. This is a win-win for everyone, while still creating an incentive for authors to enter a fully polished game from the start. It is a very reasonable system, and it disgusts me that people would go around giving 1s without proper justification.

  1. I would agree that it’s not really reasonable to auto-vote 1 on a game just because the author chose to update it, especially when they are following the rules of the competition.

  2. I think I agree with Dan here. This kind of “punishment voting” would probably run afoul of the existing Rule 7. So there’s really no reason anyone should be doing this at all, whether or not they think it’s a good idea (which again it isn’t).

  3. Perhaps it’d be worthwhile to add some clarifying wording to the Judge Guidelines, just like we did when the public discussion rule was changed. Something like “You should not be penalizing authors for doing things which are explicitly permitted by the rules (such as updating their games over the course of the competition).” Except written better by somebody smarter at a time other than 1am.

No consideration should be given to these scoundrels, Rule 7 needs to be enforced without mercy.

I think it should be fairly obvious who is doing it based on who is consistently giving 1s to updated games that the majority of judges are ranking as average or above.

While I still stand firm in my beliefs on what I wrote here, I think you are on to something in that they should make an additional rule next year that states a 1 is reserved for broken and unplayable games. While they could still give a 2, at least it would be a step in the right direction to correcting this problem. They could also give this new rule a nasty name that ridicules the culprits.

Additionally, and I imagine this would be very difficult to coordinate, but if all of the authors and people who run the IFComp were to agree to discard outliers in the judging, I believe this would also help to resolve the problem.

And since I am very tired as well, I feel it is pertinent to wonder why “outliers” is considered a misspelled word here. It is a very reasonable and common word.

Well, you say a 1 is reserved for broken and unplayable games. I generally agree, except I absolutely would award a 1 for a grossly objectionable game. And I mean grossly objectionable. Something wildly racist, for example. I think that’s a perfectly legitimate use of a 1.

You do bring up a valid argument against my proposed 1 rule. I still believe removing outliers in the judging average would satisfy your concern, and the concerns of the authors who got cheated. If something is that grossly objectionable, then your 1 isn’t going to be that far off of the average, and would be counted.

That’s fair. I was responding more to your suggestion of a new rule stating that 1s be reserved for “broken and unplayable games.” I’d prefer that read “broken, unplayable, and objectionable games.” I’d rather not have the comp rules disallow punishing a bigoted game in the harshest possible way.

I would actually take it a step further and completely eliminate my “broken and unplayable” stipulation, and your “objectionable” stipulation. And go entirely with remove “outliers” for the solution to the problem we are discussing.

Your concern is a valid one, the problem I see is that what is interpreted as objectionable is both on a societal and individual level. Defining what is grossly objectionable on a societal level would turn into a convoluted nightmare for the officials once the obvious ones are written down, and defining what is objectionable on an individual level is whatever that persons deems grossly objectionable. The result of this latter criteria is that it would accomplish nothing more than to give punishment voters carte blanche authority to give 1s on a whim without repercussion since they could just claim they found “x” grossly objectionable by whatever standards they adhere to, made up or not. A 1 will appear as a standard score in something wildly offensive, and an outlier in something that is not.

Eliminating outliers at the low end means you’d need to eliminate them at the high end too. And allowing statistics to potentially overwrite legitimate votes on either end isn’t something I’m comfortable with.

This discussion is also hypothetical. Nobody cast blanket 1-votes for updated games last year, and honestly I don’t think it’ll happen this year. If it does happen, then it won’t be a movement or anything like that. It’ll be one or two people, which is technically a non-trivial number, but a single vote is non-trivial in IFComp so that’s kinda moot. In my view, it’s not productive to rewrite the rules around such a small number of dissenting votes, especially not when they remain hypothetical. The guideline instructing judges to vote in good faith already covers this area.

Isn’t there already some sort of automated judging system that eliminates some outliers or ‘strange’ votes, anyway?

Perhaps the easiest thing to do is simply keep quiet on whether something’s been updated or not.

I know Inform games have a date stamp for when they’re compiled, but I think that can be overridden for the purposes of the comp. Can it? If so, the author just needs to be advised and informed on how to do it.

I don’t really want to open the can of worms of “what games deserve a 1?” but it’s safe to say that it’s not ideal to have games receive 1’s based not on content or implementation but on exercising a feature given to authors on the backend. It would be like giving a game a 1 solely for containing cover art, or for the author taking advantage of the lifted muzzle rule.

Unfortunately, this is no longer a hypothetical discussion, exactly. It was a hypothetical discussion when it took place on pages 3-10 of a thread ostensibly discussing Spring Thing rules that became almost entirely about the upcoming comp. It ceased to be a hypothetical discussion when the comp started and the possibility resurfaced. The key word here is “possibility”; since I posted the thread there have been a number of updates swearing to the high heavens and the italics and the bold that they aren’t ACTUALLY going to do it. I’m replying to here because the posters involved are unwilling to “explicity [sic] reject [intfiction.org]'s totalitarian so-called ‘Terms of Use’” and do it themselves. (This forum, according to that post, has also “lost the right to be taken seriously by reasonable adults,” and I in particular am peddling “the usual sort of hysterical nonsense from people who use that forum.” These discussions always reveal themselves as proxy wars, don’t they?)

Which is fine, in the way that actually following the rules of a contest is “fine.” But why is authors updating their entries something that needs to be vented about, or “got off one’s chest,” to begin with? Where, exactly, is the problem with authors doing something that has been explicitly allowed them by the comp rules for several years – and, as Steph pointed out, tends to be correlated with good entries, at least this year? Clearly this is not a lone opinion. I don’t actually expect an answer to this to be forthcoming, nor do I expect to change anyone’s minds. I just want to make authors aware of it. When I mentioned elsewhere to some authors that some voters might think less of them for updating your entry, the very next reply, within minutes, was someone surprised that this was even possible.

This is what I mean by “unfinished business.” The rules get changed, people resent that the rules got changed, and instead of being dealt with the resentment is left to stew until it bubbles over into judgments of authors based on the rules. Meanwhile, new authors come to the comp unaware that this is even happening.

Yes, it could, but the page displaying all the games will still show “last updated at (date):” ifcomp.org/ballot/

So if someone’s really vindictive, they can check their game shuffle, which has the “last time updated” listed. There is no way to stop them or to police a small amount of bias, but then again, there’s no way to stop people sending votes that bash parser or Twine games specifically.

Also I think the “I might give a 1” is very hypothetical. Someone said “I had the urge to do that, but that’s really mean.” And we should take that at face value.

I’ve had the urge to give a 1 to games before, but I took a step back and realized I was having a bad moment. That’s something people can’t police, nor should they. Unfair voting-down is going to happen, but all the same I suspect there’s a way to eliminate the most egregious of it, and the contest organizer has ways to chuck suspicious votes.

When I used the word hypothetical before, I was talking about 1-votes actually being cast for updated games. That hasn’t happened yet to my knowledge, and I think it’s unlikely to happen this year. Of course the discussion itself isn’t hypothetical; the discussion is still happening. I simply don’t expect it to materialize into any negative votes that authors need to worry about, and if by some chance it does, there will only be a few dissenting voters, and IFComp already has guidelines in place to address this.

I agree with what aschultz said too.

I have my doubts that many games this year will receive a 1 for the contents of the work themselves, so perhaps voters should be required to give a reason when giving a 1? I understand that this would introduce a lot of extra work for some folks on the backend of the comp, but voting a game a 1 because the author updated for minor spelling errors or the voter has an axe to grind with Twine games or whatever seems disruptive to the spirit of this competition. I say this as a first-time entrant, so take that as you will.

+1 CMG.

Perhaps in the dropdown with score they should add options for statistical purposes that aren’t scores:

Could Not Run - Broken
Could Not Run - Format
Category Fail - Not IF
Violates Entry Rules
No Play - Personal Preference

They could be tracked for data, but wouldn’t contribute or detract from a game’s actual score, leaving the 1 open for true expression of quality.