Baking a Fast Lambda MicroVM: Lessons Learned in the Trenches

I built a browser-based Claude Code sandbox that runs each user in their own AWS Lambda MicroVM, with a persistent home directory on an Amazon S3-backed filesystem. If you want the build story and why MicroVMs are the right primitive for it, that’s a separate post: Lambda MicroVMs + S3 Files: My Claude Code Sandbox for iPad.

This post is the performance story. It works its way out from one specific annoyance, but the lessons apply to any snapshot-booted MicroVM and to optimizing S3 Files mounts in general.

One thing worth knowing: I figured all of this out by pairing with Claude Code. We researched the platform together, formed hypotheses, and tested them through trial and error until things got fast. I built a sandbox to run a coding agent, and then the coding agent helped me make the sandbox fast. Everything in this post came out of that collaboration. Make of that what you will.

The challenge was straightforward. Every fresh VM made the user stare at “Mounting workspace…” for about 26 seconds. I got that down to about 11 seconds end to end, and the interesting part is that the fix had nothing to do with the mount.

First, measure, don’t guess

The obvious suspect was the mount itself. It’s NFS over TLS to an S3-backed filesystem, so surely that’s the slow part. Before I optimized a thing, I timed it on a warm VM:

time mount -t s3files -o accesspoint=fsap-... fs-... /mnt/test
# real  0m0.6s

Sub-second. Even after dropping the caches to force a cold read, the mount syscall itself was never the problem. The network path was a red herring.

The real timeline came from something simpler: file modification times on a cold VM. The /run lifecycle hook arrived about 4 seconds in, but my “mount finished” marker didn’t land until roughly 26 seconds after that, for a mount that succeeded on its first attempt. That’s 26 seconds of apparently doing nothing, on an operation that takes 0.6 seconds when warm. That gap is the whole story.

The learning: snapshot demand-paging

To see what’s happening in that gap, you need one fact about how these VMs boot. A MicroVM starts from a memory and disk snapshot captured at image build time. Restoring a snapshot is what makes the boot fast. But there’s a catch that shapes every performance decision on the platform.

Disk pages are lazy. Memory is restored eagerly at boot, but disk content is demand-paged: the first time any file is read on a fresh VM, its bytes get pulled from snapshot storage on the fly. That much is documented platform behavior. What the docs don’t tell you is the speed. In my measurements, that first touch ran at roughly 3 MB/s. (Treat that number as an observation from my VMs in mid-2026, not a spec; it may well improve.)

Now look at what mount -t s3files actually is. It’s a Python script. Before it can mount anything, the VM has to fault in a lot of cold bytes:

• the Python 3.13 interpreter and standard library, about 67 MB
• efs-proxy, the TLS proxy binary, at 24 MB
• the mount helpers, mount.nfs, the shells, and so on

That’s roughly 100 MB of first-touch reads at 3 MB/s. There’s your 25-plus seconds. The mount command wasn’t slow. It was paging in the entire toolchain it needed before it could do a job that takes half a second. It was building the track before it could run on it.

Trick 1: tmpfs rides the memory snapshot

To understand the fix, you need to know that Linux keeps two different kinds of content in RAM, and the snapshot treats them very differently.

The first kind is cached copies of files. When you read a file, Linux keeps a copy in memory (the “page cache”) so the next read is fast. But the file itself still lives on disk; the RAM copy is just a convenience. As far as I can tell, the platform throws those cached copies away when it captures the snapshot, which makes sense: why store the same bytes twice when the disk already has them? I say “as far as I can tell” because this isn’t documented behavior; it’s what my experiments consistently showed: every file I pre-read at build time to “warm it up” was cold again by the time a VM booted. Pre-reading files into the cache at build time bought me nothing, every time I tried it.

The second kind is memory that has no file behind it: program state, heap allocations, RAM disks. Linux calls this “anonymous” memory because it doesn’t correspond to any file on disk. The snapshot can’t drop this (there’s no disk copy to fall back on), so it must be preserved. And the platform documentation is explicit that memory is restored eagerly at boot: it’s all there the moment the VM starts, no demand-paging.

And that is the key: tmpfs, the standard Linux RAM-disk filesystem, is anonymous memory.

Files you put in a tmpfs look like normal files to every program, but their bytes live in RAM with no disk behind them. Which means a file in tmpfs rides the memory snapshot, arrives fully resident at boot, and never pays the 3 MB/s demand-paging toll.

So the move is: at build time, before the snapshot is captured, copy the cold-path toolchain into a tmpfs. Every VM then boots with those bytes already in memory. I do this in the image’s entrypoint (which, on this platform, runs once during the image build right before the snapshot is captured, not on every boot). The entrypoint copies the mount toolchain into a tmpfs and bind-mounts each copy over the original path (a bind mount overlays one path onto another, so programs still find everything where they expect it):

mount -t tmpfs -o size=320m tmpfs /opt/warm-mount-stack
for src in /usr/lib64/python3.13 /usr/sbin/efs-proxy ...; do
  real=$(readlink -f "$src")          # bind the target, not the symlink
  dst="/opt/warm-mount-stack/$(echo "$real" | tr / _)"
  cp -a "$real" "$dst" && mount --bind "$dst" "$real"
done

The paths don’t change (Python is still at /usr/lib64/python3.13) but the bytes now live in memory that arrives with the snapshot. The first mount on a fresh VM runs immediately instead of paging for 25-plus seconds.

I use the exact same trick on the Claude Code CLI bundle, which is about 240 MB and used to cost 60 to 90 seconds on its first launch. Same disease, same cure.

A few things I learned doing this that will save you time:

• Make per-file failures non-fatal. If a single path fails to copy or bind, let it stay on disk. Slow but correct beats a boot that breaks because one file moved between base-image versions.
• Resolve symlinks first. A bind mount over a symlink does not do what you want. Follow it to the real target and bind that.
• Include the interpreter and the shells, not just your app. I warm node too, because the hooks server runs on it. In my testing, even a process that was already running at capture time re-faulted its executable pages from disk on resume. (Again observed, not documented: a running process’s code pages are file-backed, so they appear to get dropped at capture like any other cached file.)
• Budget memory honestly. Between the mount stack (a 320 MB tmpfs) and the Claude CLI bundle (about 240 MB in a 512 MB tmpfs), my warm set costs over half a gigabyte of RAM that could otherwise go to the workload. That’s a real cost, and it’s worth it here, but it also grows the memory snapshot, and a bigger memory snapshot restores slower on every run and resume. Warming everything in sight would eat your own boot time from the other side. Measure your workload’s actual cold path and warm only that.

Trick 2: don’t let warmup fight the critical path

Once the mount was fast, I hit a subtler version of the same problem. I run a background warmup off the /run hook that executes the real startup path once (claude --version, git --version, the login shell) to fault in the pages a real session will touch. Good idea in isolation. But the /run hook is also what triggers the mount, so both were firing at the same moment, and demand-paging bandwidth is shared across the whole VM.

The symptom was unmistakable: with the warmup running concurrently, the mount took 10+ seconds longer; sequenced, it didn’t. I can’t see the platform’s internals, but the behavior says demand-paging bandwidth is shared across the VM. My nice-to-have warmup was competing with the mount for the same pipe, and the mount is the thing the user is actually waiting on. My background optimization was slowing down the foreground task.

The fix was to stop parallelizing them and sequence them instead. The warmup now waits for the mount’s “ready” marker before it starts, with a 60-second cap so a failed mount can’t wedge it forever:

// Wait for the home mount to finish before warming anything else.
// Demand-paging bandwidth is shared (~3MB/s), and the mount is the
// user-blocking path, warmup running concurrently starves it.
const iv = setInterval(() => {
  if (!fs.existsSync('/tmp/home-ready') && Date.now() - waitStart < 60000) return;
  clearInterval(iv);
  runWarmup();
}, 250);

When two page-in-heavy tasks share one scarce pipe, running them in order beats running them at once. There’s a corollary I learned earlier and the hard way: a broad find | cat prefetch is worse than running the actual binary once. The blanket prefetch pages in everything, including things a session never touches, and it competes with the user’s real first launch. Warm exactly what the startup path needs, and nothing else.

A note on telemetry

While reading the mount logs, I found two errors on every single mount. Before you read on: neither of these is Lambda or CloudWatch misbehaving. Both are consequences of choices in my image, and I’m sharing them because if you customize your image the way I did, you’ll hit them too.

The first one I caused directly. My image installed botocore for the mount helper’s CloudWatch logging, but it installed it under Python 3.9, the stock Amazon Linux 2023 interpreter. Later, the image swaps the system Python to 3.13 for other tooling my sandbox needs, and mount.s3files runs under that 3.13, where no botocore exists. Two build steps, each fine alone, that quietly disagreed about which Python they were talking about, and the helper’s log upload was broken from the start. The second is narrower: the particular efs-utils build in my image derives a CloudWatch log-stream name containing a colon, which CloudWatch’s naming rules (correctly) reject. Different trigger, same category: something my image ships, not something the platform does wrong.

I disabled both rather than “fixing” them. Installing botocore for 3.13 would have put a sizable cold import (the package and its dependencies are on the order of 90 MB on disk, right back onto the mount’s critical path, re-buying the exact problem I had just paid to remove. The local diagnostics under /var/log/amazon/efs/ are untouched, so I didn’t give up anything I was actually using. Sometimes the honest fix for broken telemetry is to admit you never had it, and stop paying to pretend you do.

To be clear, this is not “no boto3 in the sandbox.” The rule is narrower: keep the system interpreter lean, because that’s the one the mount helper runs under. If a project inside the VM needs boto3, install it as a project dependency, a virtualenv in the home directory. The mount path never imports from there, and as a bonus it lands on the persistent home rather than the snapshot disk, so it survives VM recycles too. System-wide for the platform, per-project for you.

The results

The before and after, measured hook-to-ready on a cold VM. One honest caveat for all the numbers in this post: the documented platform behavior is that memory restores eagerly while disk demand-pages. The rest (the paging speed, the page-cache-dropped-at-capture behavior, the shared bandwidth) is what I observed and measured on my VMs in mid-2026, not a guarantee. The platform is young; the numbers may improve, and the techniques matter more than the constants.

Path	Before	After
Cold boot, hook to workspace ready	~26s	~11s
Suspend to resume remount	n/a	~2s
The mount syscall itself	~0.6s	~0.6s (never the problem)

What applies where

A useful way to file these tricks away is to ask which technology each one actually belongs to. Some are about snapshot boot and apply to any snapshot-booted VM with no S3 Files in sight. Some are about S3 Files and apply even on a plain EC2 instance. And the headline lesson of this post lives specifically in the overlap.

Snapshot tricks. These hold for any snapshot-booted compute, whatever you’re mounting:

1. “Cold start” is mostly first-touch disk I/O. Profile it with file modification times and drop-caches timing before you blame the network. The syscall that looks slow is often waiting on bytes.
2. tmpfs is your snapshot-warm escape hatch. Anonymous memory survives capture; the page cache does not. If something has to be fast on first touch, put it in memory before the snapshot is taken. Remember, the entrypoint runs at build time, so it’s your one chance to precompute state into the snapshot.
3. Demand-paging bandwidth is scarce and shared. Sequence background warmup behind whatever the user is blocked on, rather than running both at once and splitting the pipe. And warm the real startup path, not a blanket prefetch.
4. The memory snapshot is a budget, not a free lunch. Memory restores eagerly, so every megabyte you warm makes every run and resume a little slower. Warm the measured cold path, nothing more.
5. The ready hook decides what makes it into the snapshot. The platform captures the snapshot only after your /ready hook reports success, and readyTimeoutInSeconds is the budget for that. In my image the entrypoint does the tmpfs copies before it starts the hooks server, so /ready physically cannot answer until the warm set is in place. I registered a generous 180-second ready timeout to leave room for it. If ready answered before the copies finished, the snapshot could capture a half-warmed stack and every VM would inherit it. Whatever build-time state you want baked in, sequence it ahead of ready.

S3 Files tricks. These hold wherever you mount S3 Files, snapshot or not:

1. The mount syscall is cheap. Sub-second on a warm system. If mounting looks slow, the cost is somewhere else.
2. Keep hardlink-dependent tools off the mount. The filesystem rejects hardlinks, and tools like uv rely on them for caches. Point their state at a local disk path instead of the NFS home.
3. Access points are your multi-tenancy primitive. One filesystem, one access point per user, each scoped to its own directory, giving isolation without one filesystem per user.

The intersection. S3 Files inside a snapshot-booted VM, where this post lives:

1. The mount toolchain is the cold start. mount.s3files is Python plus a TLS proxy, and on a fresh VM all of it demand-pages before any network work begins. That’s what the tmpfs trick is really for here.
2. Bake what’s shared, defer what’s per-user, and fail safe when the per-user part is missing. The snapshot is shared across every user’s VM, so the per-user mount can’t live in it. It has to happen at run time, from the lifecycle hook. And when the per-user piece is absent, degrade safely instead of guessing.
3. Answer the lifecycle hook fast; do the slow work behind a readiness marker. The hook has a timeout measured in seconds; the mount has a retry budget measured in tens of seconds. Return 200 immediately, mount in the background, and gate the user-facing shell on a ready file.
4. Design for resume, not just run. Only the /run hook carries your payload; /resume arrives empty-handed. My /run hook persists the access-point id to a local file so /resume can re-mount the same user’s home without being told whose VM it is. Get this right and resume is the fast path it should be: the toolchain is already resident from the memory snapshot, so the remount takes about 2 seconds instead of a cold boot’s 11.

All of these tricks came out of a real, working project: a browser-based Claude Code sandbox I use every day. If you want to see the full build, that’s the companion post: Lambda MicroVMs + S3 Files: My Claude Code Sandbox for iPad.

I would love to hear what you learn baking your own VMs. You can find all my socials at edjgeek.com.

#ServerlessForEveryone