NetHack+ Part 1 of 2 Part 2: results

A configurable NetHack environment

When I first came across NetHack, I had a ton of ideas:

  • Monte Carlo search (requires checkpointing)
  • Imitation learning or RL with massive amounts of data (requires game parallelization)
  • Language models (requires sandboxing and skills)
  • Curriculum learning with easier versions of the game (requires overriding game configs)
  • Just beating the game (so my agent and I would collectively have 1 win)

However, I soon realized none of these could work. NetHack is packaged within the common RL library Gym through the NLE environment, which wraps the original game binary. As a result, anything not possible in the original game remains impossible in NLE. As a result, none of my ideas were actually possible. So, I set out to fix this.

NLE and NetHack

NetHack (1987, a descendant of Hack and Rogue) is one of the original roguelikes. In the game, a player descends a procedurally generated dungeon with a single life to find the Amulet of Yendor and climb back out. Along the way the player faces hundreds of monster types and items whose effects often have to be discovered by trying them. The whole world is rendered as an 80×21 grid of characters, and the game is played with vi-style keys: hjkl to move, single letters to quaff, read and wield, and #pray to pray.

Figure 1. What a player sees with fog of war, after a short walk through the first floor. Hover over any symbol, or tap it on a phone, to see what it is; the descriptions come from NetHack's own look command, run on this exact game. You can also focus the map and move a cursor with the arrow keys. Blank squares are rock or places the hero hasn't seen yet, and the settings gallery below shows the same floor with the whole map revealed.

The game is hard for people too: of the 46,796 complete human games we decoded from the NetHack Learning Dataset, 1.8% end in an ascension, and winning games run for tens of thousands of moves. Most of a run is spent deciding when to go deeper, because each staircase down brings tougher monsters, and a player who descends faster than they gain experience dies.

The NetHack Learning Environment (NLE) made NetHack a reinforcement learning benchmark by wrapping the original C program in a gym API, which exposes a step function and an observation but provides almost nothing underneath. There is no way to capture a game's state, so any algorithm that wants to branch or look ahead has to replay every action from the start. Because the game also keeps its state in global variables, one loaded copy of the library can hold exactly one game, and starting a second game in the same process without reloading the library crashes stock NLE.

The workaround is to load a private copy of the whole library for every game. When we first plugged NetHack into PufferLib that way, on our x86 cluster, a cold reset took about 217 ms, which was 86% of wall-clock time under a random policy. Adding games to a process also made it slower, going from 44,000 steps per second with one game to 22,500 with sixteen because copies of the same 4 MB of code fought over the CPU caches (Figure 7 shows the same collapse measured directly in C on a laptop). NLE also exposes no difficulty settings at all, so when an agent dies you can't easily ask whether a locked door or the fog killed it. These three problems (one game per copy of the library, no snapshots, and nothing to adjust) share one root cause, which is that the game's state is scattered through the code.

Figure 2. What one process holds. Stock NLE copies libnethack.so into memory and loads it again for every game, so each game has its own code and data, and its monsters and objects land in the process's shared heap. NetHack+ loads the library once; each thread keeps a thread-local pointer, current_nle_ctx, to the game it is stepping, and each game is four blocks it owns. Numbers from the benchmark in Figure 7 (12 vCPUs, linux/arm64).

Memory and threading

Where one game's state lives

We compiled vanilla NLE 0.9.0 and listed every writable variable in its data segment, the part of a program's memory that holds global variables. There are 658 of them, 255 KB in all, defined across 95 source files: 220 globals, 318 file-level statics, and 120 statics declared inside individual functions, where no header will tell you they exist. The hero animation at the top of this page draws every one of them, in address order.

Figure 3. Vanilla NetHack's writable state by category. 658 variables from nm and readelf on an x86-64 build of NLE 0.9.0 with debug info; categories assigned by variable name and source file. Hover a bar for its largest variables.

A few things in that list surprised us. The single biggest variable is not NetHack's at all but NLE's own settings struct, which takes 49 KB, mostly in fixed-size path strings. NetHack's biggest is level, which holds three 80×21 grids of map cells, object pointers and monster pointers. The random number generator's state (two ISAAC64 generators, 8,288 bytes) is bigger than the entire hero struct u (2,392 bytes). The monster encyclopedia and the item catalogue are writable too, and the catalogue really is per-game state, because every game shuffles item appearances ("a bubbly potion") and records what you have identified. Finally, every level the hero is not standing on lives in a file on disk.

All of this state has to become per-game, and we ended up doing that migration twice.

First attempt: by hand

We started in May with an audit that estimated 120 to 150 mutable globals, and we first tried two shortcuts. The first copied about 50 KB of globals into and out of a per-game blob on every step, which was correct but too slow and only worked with one thread. The second marked the globals thread-local, which gives one copy per thread rather than per game and broke every static table that stores the address of a global, so it crashed with eight or more threads. We therefore moved the globals into a per-game context struct by hand, one batch at a time, starting with the easiest piece, the RNG. Each migrated global became a hand-named field and a macro that redirects the old name, so that most game code didn't have to change:

upstream NLE, src/decl.c and decl.h (abridged)NEARDATA long moves = 1L, monstermoves = 1L;
E NEARDATA long moves, monstermoves;
/* and the hero, a 2,392-byte struct */
NEARDATA struct you u;
by hand, May 2026 (abridged)/* moves/monstermoves migrated direct (stage 9' batch A).
 * Init to 1L,1L deferred to init_nle / c_reset path. */
#define moves        (current_nle_ctx->nle_moves)
#define monstermoves (current_nle_ctx->nle_monstermoves)
#define u (*current_nle_ctx->u_ptr)
Figure 4. The same globals before and after the hand migration. Every moves in the game still compiles; it now reads the current game's context.

Each batch was checked against a recorded 16-seed, 1,000-step replay, and writable global storage, measured on PufferLib's copy (whose NLE settings struct had already been shrunk), fell from 198,740 bytes to 19,018. Much of the editing was done by coding agents working in parallel, each in its own git worktree, and several things broke along the way:

The end state had about 440 hand-written fields and 1,389 references to the context pointer across 132 files. This engine ran, and it is the one behind the experiments in part 2, but we could never prove that nothing had been missed, and we later found that something had. When we wrote a test that restores snapshots and compares every observation, this engine failed it, because a little terminal and RNG state survived a restore. Prime-Agent-Explore reseeds after every restore, which limits what stale RNG state can do, although we have not yet re-run those experiments on the newer engine. That failed test is the reason we redid the migration rather than patching it.

Second attempt: mechanical

In September we reset the game sources byte-for-byte to upstream NLE and redid the migration with a tool. collect_globals is about 2,300 lines of Python on top of libclang, the C compiler's parser packaged as a library. The tool reads the real syntax tree and the reference build's object files, finds every object with static storage duration and every reference to it, including references inside macro bodies, and generates a single struct nh_globals. It found 668 objects, made up of 208 globals, 290 file-level statics and 170 function-level statics. The count is 668 rather than the 658 in the compiled library because it leaves out NLE's own 13 wrapper variables and 5 deliberately shared tables. It also adds 28 read-only tables of addresses, like worn[], that had to become per-game once the objects they point to did. Of these, 509 keep their names through accessor macros, while the 159 whose names would collide with something else, like level, flags and objects, are rewritten at every use as NH_G(name). Initial values become const templates that are copied into each new game.

Figure 5. What collect_globals found and how it rewrote each object, next to the hand migration. 668 objects with static storage duration, split by kind and by how references to them were rewritten; the hand migration in May ended with about 440 hand-named fields and missed some terminal and RNG state.
upstream, src/cmd.c randomkey()static unsigned i = 0;
...
c = extcmdlist[i++ % SIZE(extcmdlist)].key;

/* src/apply.c */
struct obj *mshelter =
    level.objects[mtmp->mx][mtmp->my];
mechanical, generated/* i: per-env nh_g->l_cmd_c_randomkey_i */
...
c = extcmdlist[NH_G(l_cmd_c_randomkey_i)++
               % SIZE(extcmdlist)].key;

/* src/apply.c */
struct obj *mshelter =
    NH_G(level).objects[mtmp->mx][mtmp->my];
Figure 6. A static hidden inside a function, and a name collision, after the mechanical rewrite. By hand, the second line required renaming level.objects to level.objs everywhere; the tool leaves the field alone and rewrites only references to the global.

The generated diff is smaller than the hand-written one (134 files, +7,010/−4,824 lines, against roughly +8,400 by hand), but what the tool really added was a way to check completeness. A build check lists every symbol in a writable section of the compiled objects and fails on anything not on a short, justified whitelist; function-level statics show up there too, so a missed variable cannot hide. A second check replays several seeded games in three ways (alone, interleaved with other games, and through snapshot and restore) and requires identical observations at every step. A third check classifies every field of the struct by how it survives a checkpoint.

One arena per game

Globals are only the roots of a game's state, since the rest of it (monsters, objects, traps and level data) is allocated as the game runs. Each game therefore gets its own arena, a 64 MiB virtual mapping that is reserved but only paid for as pages are touched and is handed out by a pointer that only moves forward. free() does nothing for arena memory, and space comes back when a restore rewinds the pointer or the game ends.

/* src/alloc.c, the fast path of alloc() */
size_t need = (lth + NLE_ARENA_ALIGN - 1) & ~(size_t) (NLE_ARENA_ALIGN - 1);
nle_ctx_t *ctx = current_nle_ctx;
if (ctx) {
    if (!ctx->arena_base)
        nle_arena_per_env_init(ctx);
    size_t offset = ctx->arena_used;
    if (offset + need > ctx->arena_cap)
        panic("nle_arena: per-env out of memory");
    ctx->arena_used = offset + need;
    return (long *) (ctx->arena_base + offset);
}

NetHack's window port is C++, so operator new and delete are overridden to use the arena too, and two linker details make that safe. First, the overrides are hidden symbols, because otherwise the dynamic loader would route every C++ allocation in the process, PyTorch's included, through NetHack's bump allocator. Second, the library links with -Bsymbolic-functions so that its own calls bind to them. (These two details are also why the engine doesn't build natively on macOS.) A fresh game's arena holds 16.5 KB, which grows to 181 KB after 40,000 steps.

Is it faster?

Per step, NetHack+ is no faster than the original: when we measured stock NLE 0.9.0 against NetHack+ with the same compiler flags, random keys, one game and one thread, both took about 6.3 microseconds per step. The refactor therefore changes how many games a process can hold rather than how fast each one runs.

Figure 7. Aggregate steps per second as games are added to one process. Random keys, auto-reset on death, Docker linux/arm64 with 12 vCPUs on an Apple M4 Max laptop. Stock NLE needs a private copy of the library per game; NetHack+ shares one. Right: 1,024 NetHack+ games spread over threads.

With 64 games on one thread, NetHack+ runs 142,000 steps per second against stock NLE's 66,000, because the games share one copy of the code instead of evicting each other from cache. Adding a game to a process takes 0.25 to 0.4 ms instead of 20 to 26 ms (stock NLE has to copy and load the library again), so a thousand games start in under half a second. Because one process can hold them all, it also scales across threads: 1,024 games on 12 threads run at 622,000 steps per second, 6.1× one thread, and 4,096 games reach 636,000. One caveat we found while measuring this is that on our arm64 build a game must be stepped on the thread that started it, although training is unaffected because PufferLib already pins games to threads.

Stock NLE can also use every core by running several processes, and we measured that too, through PufferLib's own environment loop on a 16-core AMD EPYC. With 64 games, 16 single-threaded stock processes ran 669,000 steps per second against 212,000 for NetHack+ in one process on 16 threads, because four games per process stay in cache and no thread waits at a per-step barrier. With 256 games the two were close (306,000 against 331,000), and with 1,024 games NetHack+ was 2.1× faster (456,000 against 218,000). In that same loop, stock NLE crashed whenever more than one thread stepped its copies, and a single process could not reliably hold more than about 150 of them. What it cannot do is put thousands of games inside one training process, which is what PufferLib's vectorized loop needs. (The 217 ms reset we started from was the cost of our own first binding, not of NLE itself.) On the hand-migrated engine, PufferLib trained 4,096 games at about 125,000 steps per second on one A100. Roughly 80% of that time was spent in the policy update (PPO) rather than in the game. Two small changes mattered a lot on that engine: prefetching the first 256 bytes of each game's context, where the hottest fields sat, raised training throughput by 50 to 60%. Sharing one read-only copy of the game's data files cut a Python-level reset from 35 ms to 1.3 ms.

Memory is where the per-game layout pays off most. Stock NLE needs about 3.2 MB for each game, most of it the private copy of the library's code, so 256 games took 832 MB. A NetHack+ game needs about 340 KB, so 256 games take 93 MB and 4,096 games take 1.4 GB, which is about nine times less. That figure depends on one fix we found while measuring it. On Linux kernels that use transparent huge pages for every mapping, the first write into a game's 64 MiB arena faulted in a whole 2 MB page, so each game cost 2.4 MB and the saving shrank to 1.4×. Marking the arena MADV_NOHUGEPAGE brings a game back to 340 KB with no change in steps per second. (Resident memory from /proc/self/smaps_rollup after 1,000 random steps per game, in Docker on linux/arm64.)

Is it still NetHack?

A refactor of this size makes it easy to change the game without meaning to, so we ran 20 seeds with the same stream of actions through stock NLE and NetHack+ for up to 1,000 steps each. On all 20, the map glyphs, characters, colors and messages were identical at every step and episodes ended on the same step, and on 16 of the 20, every observation field matched. The other four differed in a single number, the HP shown during the death sequence, where stock NLE shows a stale value and NetHack+ shows 0, most likely because our engine refreshes the status line every step. These runs used random keys and rarely left the first floor, so they don't exercise level files, shops or deep levels. The stronger test, which we will run next, is to replay the agents' own recorded keystrokes, which reach dungeon levels 5 to 8 on average, through both engines.

Checkpointing

Forking a game is a memory copy

We didn't set out to build checkpointing, but once a game's state is a context block, an arena, a stack and a few buffers, saving it is a memcpy of each. A snapshot holds:

Figure 8. What an in-memory snapshot holds, for a game still on its first floor. Only the arena grows as the game runs (16.5 KB at the start, 181 KB after 40,000 steps); everything else is a fixed size. Level files for floors the hero has left are copied in addition and are not shown. Hover a segment for what it contains.

There is no pointer relocation, because a snapshot is restored into the same game at the same addresses, so every interior pointer lands on the same bytes again. This makes restoring fast, but it also means a snapshot can never be written to disk, so checkpoints that cross processes use NetHack's own level and save formats instead. Level files turned out to be the subtle part: NetHack trusts a flag saying a level file exists and treats a missing file as cheating, which is an instant death. Early on, this meant that revived games died the moment they took the stairs because the game had deleted its level files when it first died.

Figure 9. Getting back to a game state: restore a snapshot, or replay every action from the seed. Hover to compare the two at each measured point. Mean of 200 calls at each point, one game on dungeon level 1. Even after a single step the two differ by about 23×, because replaying means resetting the game, which builds the first floor again, while a restore only copies memory. Snapshots of games that have visited several levels also carry those level files, which we have not timed.

For a game still on its first floor, a restore takes about 20 microseconds however long it has run; a game that has visited other floors also copies those floors' files, which we haven't timed yet. Replaying the same game from its seed takes 5.7 ms after 1,000 steps and about 740 ms after 40,000, so at that point a restore is about 34,000× cheaper. To get different futures from one snapshot, we restore and then reseed, since without reseeding every branch replays the same rolls. We also froze the wall clock, since NetHack checks the moon phase and Friday the 13th, which otherwise made the same seed play differently on different days.

Breaking the game

Turning constants into settings

The same refactor also made difficulty adjustable. Settings are declared once in a single list that a macro expands into the struct, the defaults and the Python names. Adding one therefore means writing one line there and one read where the game makes that decision. Each read is guarded so that a setting's vanilla default leaves the random stream untouched, which keeps the default game byte-identical to vanilla:

/* src/mklev.c (abridged): is a new door locked? vanilla rolls rn2(6) */
int lock_mod = 6;
if (nle_tuning.locked_door != 1.0) {
    double k = nle_tuning.locked_door;
    lock_mod = k <= 0.0 ? 100000 : (int) (6.0 / k + 0.5);
}
if (!rn2(5))
    levl[x][y].doormask = D_ISOPEN;
else if (!rn2(lock_mod))
    levl[x][y].doormask = D_LOCKED;

There are 17 settings in total. Six of them shape a floor when it is generated (room density, room size, corridor connectivity, trap density, locked doors and starting monsters). The other eleven are read every turn and can change mid-game: vision radius and full-map reveal; damage taken and dealt, HP on level-up, HP regeneration, hunger and experience gain; and ongoing spawns, monster speed and monster difficulty. Alongside them are 13 bounds-checked edits to the hero (HP, max HP, gold, experience level, a level-up, hunger, the six attributes, luck) and a jump to any dungeon level. Together they let us set up an exact scenario without playing into it.

Figure 10. What the settings do, rendered by the real engine (the WebAssembly build on this site, seed 33). The map steps through each value of the chosen setting, and the panel next to it compares all of its values at once. Clicking a value stops the cycle on that value, and hovering over the map or the charts pauses it and shows the details.

The engine also has a six-floor version of the game: the first three floors of the Dungeons of Doom joined directly to the last three of Gehennom (dungeon levels 48 to 50), which lets an agent practise the late game without playing dozens of floors to reach it.

Sandboxes

Sandboxing architecture

The final system splits one evaluation across a model provider and four boxes, and the most important choice is what is not in the agent's box, namely the game. Scroll through the diagram to see what crosses each boundary.

Play is split across the agent's container, a tool server on the host, the game's own sandbox, and a fresh sandbox that grades the game by replaying it. The agent reaches the game only through tool calls, and nothing it can touch holds the score.

The model only ever talks to the agent harness, Prime Agent or Claude Code, which runs in a bare Python container with one CPU and no network once it has been set up. The agent's box holds no game, no record of play and no score, so there is nothing in it to tamper with. We moved to this design after an unconfined agent in an earlier harness searched its scratch directory and printed a live access token into its own trace.

From the agent's side, a skill call looks like an ordinary function call, such as nethack_np_move_to({"x": 8, "y": 2}), which goes over MCP (Model Context Protocol) to the tool server on the host. The tool server enforces the call budget, which the agent never sees, so a spent budget simply reads as [the game is over].

The tool server forwards each call to the game's sandbox as one JSON line. There NetCore plans an A* path and the engine consumes three keys, l l u, which go into an append-only record that only the tool server writes. In code mode, the agent's own skill code runs here as well, as an unprivileged user in its own namespaces, limited to 200 keys and 60 CPU seconds per run.

When the game ends, a fresh sandbox with no network replays the keystroke record from the seed and computes BALmin, a balanced progress score defined in part 2, so the reward is whatever those keys actually do. Milestones come from the engine's own end-of-game state rather than from message text, because a player can engrave a fake victory message on the floor.

Figure 11. Sandboxing architecture for one evaluation. Every skill call is recorded as the raw keystrokes the engine consumed, and the score comes from replaying that record in a fresh box, not from anything the agent or the live game reports.

The model then gets back what happened during the call, in the order it happened:

[Executing skill 'move_to 8 2'. Skill 'move_to 8 2' completed: Reached position (8, 2).]

=== GAME MESSAGES (printed during this call, in order) ===
  You swap places with your little dog.  $ - 2 gold pieces.

=== STATUS ===
HP: 16/16  AC: 6  Dlvl: 1  Turn: 4  XP: 1  $: 2  Pos: (8,2)
Character: valkyrie (human, neutral)

(Abridged: a line pointing at the full-map call is omitted.)

There is also no "descend" tool, so going down means pressing > on a staircase, as a human would.

In code mode, the agent edits NetCore's Python skills and runs them. That code executes inside the game's sandbox as an unprivileged user in its own network, process-communication and filesystem namespaces, with a system-call filter and empty temp directories. It is limited to 200 keys, 60 CPU seconds and 90 wall seconds per run. It can read the screen, inventory and monsters through a game object and press keys, and nothing else.

Skills

Chunking actions

In robotics, computer use and other long-horizon domains, chunking actions has turned out to matter a great deal. Predicting a chunk of k actions at once shrinks the effective horizon k-fold, and in ACT's ablation it took bimanual manipulation from 1% success with single actions to 44% with chunks of 100. π0 calls its model once every 16 to 25 control steps, and on computer-use benchmarks, code actions that chain several steps together raise success while taking fewer of them (CodeAct; CoAct-1 reaches 60.8% on OSWorld in about 10 steps per task, against 15 for GUI agents). Chunks are therefore not only cheaper, because the model is called less often, but also better. NetHack points the same way. Hierarchy and options have helped RL agents on the game (Piterbarg et al.; Henaff et al., with the right option rewards), NetPlay gave an LLM a library of interruptible skills, and Astra's ascension ran on movement batches of up to 24 keys that the agent wrote for itself and that stop on damage, a level change or a nearby creature.

A successful game runs to tens of thousands of moves, and one meaningful action, like walking to the next room, can take dozens of key presses, so a harness has to chunk actions. Every way of doing that makes decisions on the model's behalf, and several of them were wrong in early versions of our harness:

Existing NetHack frameworks make these choices differently, and the table below compares them on the questions that mattered most for us.

FrameworkSkillsInterruptibleGame prompts reach the modelObservationRNG controlSnapshots
NLE (2020)✗ raw keys–◐ default tasks skip --More-- and escape most yes/no questions◐ arrays: map, stats, message, inventory✓ seedable✗
BALROG (2024)✗ one key per call–◐ --More-- pressed for the model; yes/no and menus shown◐ ASCII map and inventory, last 16 turns◐ random seed, recorded✗
NetPlay (2024)✓ about 30✓ level change, new glyph, HP below 40%◐ --More-- skipped; menus make skills fail◐ text summary of rooms, 500-token memory✓ seedable✗
Glyphbox (2026)✓ Python code◐ helpers only; the model's code runs to the end✗ answers yes/no and escapes menus for the model✓ full screen, monsters, inventory✗ no seed control✗
NetHack+ with NetCore✓ 8✓ level change, hostile near, HP drop, food spoiling✓ never answered for the model; every message returned✓ screen and status; map and inventory one free call away✓ seeded, reseeded after restore, clock frozen✓ fork in about 20 µs
Figure 12. How NetHack agent frameworks handle chunked actions, prompts, observations and randomness. ✓ yes, ◐ partly, ✗ no. Read from each framework's code at a pinned commit (NLE 2319f29, BALROG b7afe79, NetPlay 6acb90d, Glyphbox 4d6eace) and from the papers. NLE's Challenge task exposes every prompt; its default tasks do not.

NetCore: compressing actions without taking decisions away

NetCore is our default skill set: eight skills built on NetPlay's implementations, plus search and an observe call that returns the full map without spending game time.

SkillArgumentsWhat it does
np_explore_levelnoneExplore the current floor: rooms, corridors, hidden doors
np_move_tox, yPlan an A* path and walk it in one call
np_melee_attackx, yPursue and fight the monster at a tile until it dies
np_kickx, yKick a tile, usually a locked door
np_applyitem letterUse a tool from the inventory
np_restcount (default 5)Wait and regenerate until something happens
np_praynonePray
np_press_keykeyAny raw key or sequence; how the model answers the game's own prompts

One call can run for up to 100 game turns. NetCore nevertheless hands control back as soon as something needs a decision: a level change or teleport, a hostile monster within seven squares (or one that keeps closing in), HP falling below half or dropping by a quarter in one call, or carried food about to spoil. It never answers a game prompt or menu on the model's behalf; instead, the observation says the clock is frozen and lists the legal replies. Every message the game printed during the call also comes back in order, so nothing that happened mid-skill is hidden.

Conclusion

Infrastructure determined what was cheap to try. On NLE, getting back to an earlier state means replaying the game from its seed, which is affordable for an LLM agent that waits seconds per call. It is far too slow, however, for RL or tree search that restore thousands of times. Running thousands of games in one training process and changing a setting mid-game were not possible on NLE at all.

Try it

Play NetHack+ in your browser at liu-jonathan.com/nethack. To run the evaluation yourself, see NetHack-hub.

Next, part 2 uses this engine to measure how five current language models play, and how far a search over forked games takes one of them.