IF Archive improvements

I am taking some vacation time to do a bit of renovation work on the server.

The big change you will see is that the index page URLs now mirror the directory structure; they no longer have “X” or “.html”. So you can visit

ifarchive.org/indexes/if-archive/
ifarchive.org/indexes/if-archive/games/
ifarchive.org/indexes/if-archive … ition2018/

instead of

ifarchive.org/indexes/if-archive.html
ifarchive.org/indexes/if-archiveXgames.html
ifarchive.org/indexes/if-archive … n2018.html

(The old-style X URLs will still work, because URLs shouldn’t break.)

I’ve also added sha512 checksums to the Master-Index.xml file. I doubt this matters to anybody – I don’t think the md5 checksums were being used for anything – but md5 is old and broken, so the Ghost of Crypto-Nerds Past says “use something newer.”

(Really sha512 is also old and slightly broken. Maybe next Christmas I’ll install the latest Python and stick sha3 in there.)

The next big change will be that, sometime this week, I will put the site under the Cloudflare CDN service. This should handle load way more smoothly, and with way more bandwidth, than the volunteer mirrors which have existed for the past many years.

Volunteer mirror maintainers: thank you! You may continue to host your mirrors and rsync the IF Archive if you want to; but after Cloudflare goes up, your servers won’t be a critical resource. ifarchive.org/.

Cloudflare costs some money (at the level we intend to use it). This is cool because we have a nonprofit with money in the bank. Want to support the IF Archive? Your donation is appreciated: iftechfoundation.org/give/

2 Likes

The amazon smile link doesn’t seem to be working.

The Paypal button works.

Thanks for all you do for the IF community.

fos1

Looks like Amazon have simplified their links and it should now be: smile.amazon.com/ch/81-1920375

You can also sign in to smile.amazon.com, mouse over the “Supporting” thing below the main search bar, and click “Change” on the popup that appears showing the charity you’re currently supporting to search for the Interactive Fiction Technology Foundation.

Also Smile is only 0.5% and not all purchases are eligible, so it seems like it would be pretty easy to do more good by donating directly…I’ve apparently generated all of $3.16 over the last year for the organization I’ve been “supporting”.

But hey, every little bit helps.

1 Like

Thanks, we’ll update the link.

I’m kicking DNS over to CloudFlare. You may see HTTPS authentication errors (SSL cert errors) for the next day or so, as the CDN figures everything out.

mirror.ifarchive.org/ – they’re all the same server.

Everything should work now. If you find any problems, please let us know.

Note that the CloudFlare service does not include FTP. We have a long and fond history with FTP, but it’s not clear that anybody uses it any more. If you miss it, talk to us.

A few changes for this year’s holiday!

The index files have been slightly reformatted. This is mostly a back-end change – making it easier for the admins to edit the Index files without worrying about a fussy indentation system. (Which I think Volker Blasius invented back in the FTP days…)

The only nontrivial change you’ll notice is that some of the index pages now use HTML lists. For example, look at http://www.ifarchive.org/indexes/if-archive/games/spectrum/

However, we have some more interesting changes lined up for the next week or so. I’m taking this one step at a time, but watch this thread for further developments.

If you find any problems with the new index pages, please let me know.

Going in through FTP, I see this “new” format for the Index files, but when my gopher mirror gets them, they’re in the “old” format. What’s going on?

Does the mirror use HTTP? The Cloudflare cache may take some time to clear out. Although http://www.ifarchive.org/if-archive/utilities/Index shows the new format, so maybe that’s not it.

Oh, I also preserved the modification date on the files. (Since the mod date usually shows the last substantive addition, but these changes are purely about formatting.) That might be confusing your mirror.

I’m going through rsync. The crontab lines look like this:

10 11 * * * rsync -arvz --port=873 --bwlimit=100 xxx.xxx.xxx.xxx::indexes /var/gopher/if-archive/indexes
15 11 * * * rsync -arvz --port=873 --bwlimit=100 xxx.xxx.xxx.xxx::misc --exclude 'rsync-exclude.txt' /var/gopher/if-archive/misc
20 11 * * * rsync -arvz --port=873 --exclude-from=/var/gopher/if-archive/misc/rsync-exclude.txt --bwlimit=100 xxx.xxx.xxx.xxx::ifarchive.org /var/gopher/if-archive
25 11 * * * rsync -arvz --delete --port=873 --exclude '.*' --bwlimit=100 xxx.xxx.xx.xxx::ifarchive.org/unprocessed /var/gopher/if-archive

Okay, here’s the drop of the other shoe! If you look at the index pages (e.g. http://ifarchive.org/indexes/if-archive/games/glulx/ ), you’ll see that many of the entries have a new “IFDB entry” link.

This is exactly what it looks like. If we can associate an IF Archive file to an IFDB entry, we now display a link for it. It’s not just for game files, either. Many of the walkthrough files in http://ifarchive.org/indexes/if-archive/solutions/ have these links too.

Now, the caveats. I extracted these associations from the most recent IFDB database dump. This means:

  • The IFDB dump is from early November. Any games added in the past month won’t have links.
  • Any errors or omissions in IFDB are replicated without cleanup.
  • There’s no automatic mechanism to import IFDB updates. In a couple of months, I will go through another IFDB dump and see what’s changed. We are thinking about ways to automate the synchronization of data between the sites, but this is still very much in progress.
  • I relied on the gamelinks table in IFDB – this is what’s displayed as “Downloads” on IFDB pages. This usually includes all game-related files on the IF Archive, but not always.
  • Some Archive files refer to many IFDB entries. (For example, http://ifarchive.org/indexes/if-archive/games/c64/ contains several “combo packs” of games.) In these cases, you’ll see a list of apparently identical IFDB links. You have to click through to see which links go to which IFDB pages.
  • We only have IFDB links for files, not directories. This leaves IFComp entries somewhat out in the cold; those are stored as unpacked directories. We have a separate plan in progress to make this more consistent…
4 Likes

To answer the obvious question: what should you do if you see an error or omission? Please update IFDB. Add a download link to the appropriate IFDB page, referring to an ifarchive.org URL.

As I said, we’re still thinking about how this crosslink will be maintained and improved. My original idea was “add metadata for every IF Archive file”. Obvious, right? That’s what exists right now. (You can see that, e.g., https://ifarchive.org/if-archive/games/glulx/Index has tuid: lines for many files.)

But, of course, IF Archive files do not have a one-to-one relation with games. Some files include several games. Some games exist in several files. Some IF Archive files have nothing to do with games at all. (There are compilers, interpreters, articles, database dumps, etc.)

To do this “right”, we’d have to rely constantly on IFDB’s database, or else keep a synced copy of it. Neither of these seems like a good idea. Thus the current very minimal step.

More news as it develops.

1 Like

Does this mean the X-style URLs will continue to be generated for new games and content, or is it purely preserving old URLs for archival purposes, and every address will be slash-formatted going forward?

They will continue to be generated.

(That was last year’s change. You can observe that https://ifarchive.org/indexes/if-archiveXgamesXcompetition2019.html and https://ifarchive.org/indexes/if-archive/games/competition2019/ both exist.)

1 Like

Somehow my gopher mirror is now getting the new format of Index files. I guess it’s time for me to write a script to parse Index files into nice gophermap files.

1 Like

I updated the modification-time of all the Index files (by a couple of hours, so they still have their historical apparent dates). I guess that allowed your rsync to update.

Rsync is supposed to look at both the size and mod-time, so this shouldn’t have been necessary. (All the sizes changed.) But, whatever, as long as it works.

1 Like