silm.pw runs some custom software that allows players to upload areas by themselves, with no admin, builder or DM intervention neccessary. That software is called FWN (Frankenwinter Nights, haha). The premise is to take the load off devs and allow people to build maps in a collaborative fashion that was not possible before.
So, uh, what is FWN, again?
FWN is a Rails (web) application, that is a wrapper around a git repository, held together by angular for the frontend, and nwn-lib on the backend. It accepts uploads of gff data (.are, .git, ..) and containers (.erf, .mod) by any authorized account, checks the uploaded files for validity, and then commits them. While this may sound fancy, it is actually quite simple for the end user.
FWN supports browsing commits, locking and unlocking of resrefs for modification, downloading current and past data, viewing minimaps and managing the supermap (more on that later).
The backing (bare) git repository is handled by rugged, a low-level ruby git library. It has a flat file/directory structure, closely mirroring how files are packed in a .mod; but files are stored as deserialised yaml instead of binary gff data (I chose yaml over json because it is just a tiny bit prettier). This repository isn't exactly intended to be touched out-of-band (meaning outside of FWN), but it could be done if one would want to do mass-updates, as long as no concurrent FWN access happens and the lockbox would be clear.
(Also, for a somewhat-succinct history, see FrankenWinter Nights).
Index Imagination
Each authorized account has a "staging area", which is called an index in git terminology. This is basically a serverside temporary box where data can be dropped into and then committed in a single update. If you are familiar on how git works, this is modelled after that - but reimplemented to support more fine-grained control over it that couldn't have been done with native git indices.
Lock me baby
Because no automated merging mechanism for gff trees exists yet(not actually trivial), there are safeguards in place to prevent concurrent modification of data. These safeguards are the reason why builders have to jump through some hoops, the first of which being the lockbox mechanism.
That mechanism is, at it's simplest, just a flag that tells other people "I am working on this resref!", but FWN also disallows other people to lock the same files (or, indeed, make changes to them).
Stale data is even worse than stale beer
However, that isn't enough to prevent accidental overwrites with outdated, or even completely unrelated data.
Verifying that data is not stale, thus, is important. You don't want people to be able to (accidentally) upload outdated resrefs, overwriting edits by other players. This is a problem that used to feature frequently in the past, so getting it right this time was a priority.
Each gff file downloaded from FWN is embedded with metadata describing the commit the downloaded file is based on. You can see this in the object variables for areas, or comment field for resrefs that have no VarTable (like .git). At upload time, this data is extracted again and compared to the latest commit where the resref was touched. This could obviously be fudged or simply copy/pasted if one would want to.
Only changes that are direct descendants of current data are allowed to be imported (otherwise it would overwrite changes from other players with stale data). To make this process easier, FWN has a per-account lockbox where resrefs can (and must) be put into - a simple mechanism to stop players from editing the same files concurrently.
Since authors usually don't bother updating their local metadata between commits (by fetching it from FWN again), changes to previously-new files, or files based on the same parent by the same author, are still allowed.
Rewriting your bugs on index upload
Apart from that, data is also rewritten to account for convenience fixes so builders to not have to keep track of them, or things they miss regularily. Here's a very short excerpt of things we check for, just to give you an idea on what we try to catch serverside (sources if you're interested):
- Resref naming schemes to fit our style guide.
- Rudimentary access controls (i.e. most people can upload areas, but creature templates are restricted to encounter maintainers)
- Strip out all FWN metadata (after verification)
- Doors without a transition to be locked & flagged as Plot
- CExoLocs not having multiple languages, confusing clients (we only do English)
- Transition correctness
- .. and lots more
Uploaders see warnings and errors live as files are dragged in; this allows for a very smooth workflow (that is, until a native app comes along to make it even smoother-er). The rule of thumb here is to allow as much as possible, and only fail on errors that cannot be fixed automatically or would otherwise yield results unintended by the author. The point is to not catch mischief, but to help people avoid common pitfalls.
Committing
Once the index looks about right to the author, he provides a commit message (which is just what you think it is - some human-readable details on what his changeset does), and clicks commit. Data is converted to yaml, written to the git repository with proper attribution, automatically pushed to Stash, and then deployed to the server.
Sidenote: Because not all checks can be done on import this step provides another hook for verfication – for example checking that merchants never sell resrefs that don't exist.
Deploying
Deploying files takes a bit of effort. Each commit (actually, each resref change) triggers a special asynchronous Deploy job run in the background. That job pulls YAML data out of the repository, and then runs even more script hooks.
Those hooks are there to do further mangling - especially that kind of mangling that is required for live operations, but shouldn't ever end up in the source repository; for example:
- Resrefs are once again annotated with commit data; seeing those are just local variables, any script can query them serverside to see by whom, when and why that particular object was made.
- Ensuring placeables are flagged static or non-static where required (think AnimationState, readable descriptions; load & render performance).
- .. and so on ..
Additionally, metadata like extended tile information is being extracted, collated with what is available in game resources not otherwise accessible from scripting (like model names, tile IDs), and put into Redis as well, so that it can be accessed at very little cost. We also make extra minimap pngs and cache them for the web frontend.
The actual resulting gff data is serialised and put into Redis too, which nwserver now is free to access (nwnx_redis supports ResMan after all). nwserver is notified through Redis pubsub support, and a very simple script handler (see include) reloads the area with nwnx_areas.
Changes to non-area resrefs are handled a bit different. Those obviously only take effect when a new instance is spawned. Existing creatures, items, and so on stay where they are (but merchant objects are re-instanced on the fly as well by comparing commit sha on each access).
The net effect is that changes are live pretty much immediately after clicking the "Commit" button, with no further effort required on my part.
Upcoming on this blog: automatic, clickable area transitions, courtesy of FWN and a crufty google docs sheet.