Git as a Datastore: Building a versioned AI File System

My team and I were all set to build an agentic canvas, a visual Claude Code of sorts that lives in the canvas. It seemed like an obvious thing and we were wondering why no one had done it. Only after I started architecting the solutions did I get a hint of why.

So what is this canvas that we want to bring forth? We defined it as a distributed file system that is durable with certainty (lots of digits), has properties of isolation for security and quick read/write properties for code execution and manipulation in real-time. While walking through the different ways we could make this happen, I realised this is a much deeper rabbit hole.

It definitely was an achievable problem, the obvious ideas were to use the VM file system, maybe Postgres as a virtual text-based file system. I was familiar with rsync and boto which could mirror S3 on a local file system. However, walking through all of the solutions made me realise they were all flawed in some way. Elaborating:

1. code sandboxes, any kind; E2B, docker, CodeSandbox.io

This was the first obvious exploration. We used E2B sandboxes before and felt that this would be the natural path for making our canvas come true. There were several issues but the one that we couldn’t come to accept was state snapshotting. E2B imposes a hard 24-hour maximum lifecycle limit on a sandbox. But people’s projects and conversations live indefinitely. This meant we had to constantly serialize the user’s state, transfer it over the wire to hydrate the sandbox, and then extract the modified state back out to some hypothetical persistent storage before the sandbox died.

async def handle_agent_turn(user_project_id, agent_code):
    zip_buffer = await fetch_project_state(user_project_id)

    # or get an existing live one
    sandbox = await e2b.Sandbox.create()

    # requirements for idempotency otherwise branched logic, ugh, ugly
    await sandbox.upload(zip_buffer)

    await sandbox.run(agent_code)

    # mandatory persistence after every job/turn otherwise no guarantees.
    new_zip = await sandbox.download_workspace()

    await save_project_state(user_project_id, new_zip)

flowchart LR
    FE(Frontend) -->|Agent Turn| BE(Backend)
    BE -->|1. Fetch State| DB[(S3 Storage)]
    DB -.->|Massive Zip| BE
    BE -->|2. Hydrate & Execute| E2B{E2B Sandbox}
    E2B -.->|Modified Zip| BE
    BE -->|3. Persist State| DB

This was a surface area which would produce ugly code and emergency alerts at 02:00 AM, which I couldn’t accept, moreover the cold-start latency issues worsened this contender, moving on.

2. postgres virtual file system; text documents miraged as files

I knew that Postgres was blazing fast for text reads, searching, and joins. I thought I could use these properties if virtually abstracted. We already had decided the contract of the file system that we would give to the agent. I started working on it, I had to ensure POSIX-compliance on top of Postgres, even hearing it as it is makes you realise this sort of thing needs a dedicated team for maintenance, fixing bugs regularly, that sort of thing. I was getting the feeling this was far from the solution.

CREATE TABLE virtual_fs (
    id UUID PRIMARY KEY,
    project_id UUID,
    file_path TEXT,
    content TEXT,
    is_dir BOOLEAN,
    last_touched TIMESTAMPTZ --- imagine the bugs that would be introduced if there are mutation issues here
)

--- how would we deal with relative/absolute paths? how do we allow arbitrary commands over this, handle different pipe creations based on the operation. absolute insanity

Postgres is rock-solid. It has transactional guarantees and control over however we want to implement versioning. However, a virtual file system is as its name suggests, virtual. I never even got to the part on how agents would run arbitrary commands over the layer.

3. what we did instead; S3Files (S3 backed EFS over NFS)

At that time, AWS was still in the early stages of its new S3Files product. It almost felt like they tailor-made this solution for platforms that require a native file system for their agent. Although I have my own gripes about the POSIX-compliance of this new product; a lot of our user data was already on S3, and so we made the bet on this.

S3Files is basically a remix of their old EFS product, but there is a constant rsync-like mirroring happening in the background. You should read more in their documentation, but giving a basic gist below:

• Paths are synonymous with S3 object keys
• Only the first time you touch a directory, S3Files imports the metadata of the files inside, but only bytes of files under the import threshold (configurable, 128 KiB by default)
• Anything larger exists as a name, a size and a mode, there is no data. S3Files only streams the data directly from S3 when you actually call a read() on that file. stat() and open() never touch file bytes
• The writes to the NFS file system are synced over to S3 through a 60s inactivity interval, this is a major gotcha that we have addressed later

Reading all of this in their release article was a joy, since it was almost the perfect solution for our use case. I got started playing around with it immediately.

not done yet - we need time travel

The storage challenge with a native file-system interface is solved, it is POSIX-compliant for the attributes that we care about and is as durable as S3 (lots of digits). But a raw file system is forgetful, it only knows what is currently there.

Our product is a canvas, and one of the basic requirements is the ability to traverse across different actions done in the canvas. If some wrong move happens, or if the agent makes a wrong change, you expect to go back to the previous state with just a keystroke. So we needed a way to snapshot the different versions of the canvas, without the expensive duplication.

So to redefine what we wanted, a versioned, branchable file system; kind of sounds like a tool that we use daily and never realise the beauty of what it enables for us.

how git became everything for us

It was beautiful at first, we ran git init on every isolated workspace on the NFS mount and automatically got versioning. We made a commit every time an agent or a human made any change in the canvas. If someone wanted to peek at a previous version, we simply checked it out. If someone wanted to revert, we made a revert commit on top. We could trivially give awareness to the agent that things have changed by simply running a git diff with the latest commit that it had seen.

But conceptually beautiful things often meet a shattering reality. And ours shattered because of the NFS metadata stat() penalty.

git was not meant for network-based file systems

If you have a NAS setup or have used any file system over the network, you need to understand the brutal truth: streaming bytes for reading is fast, but reading metadata is punishing.

My first naive implementation was shelling out to subprocesses:

def commit_canvas_state(repo_path: str, message: str):
    subprocess.run(["git", "add", "."], cwd=repo_path, check=True)
    subprocess.run(["git", "commit", "-m", message], cwd=repo_path, check=True)

The problem lies in what git add . has to do to find the answer to “what changed?”. Git’s dirty model is built on trusting stat(): it walks the entire working tree, lstat()s every tracked file, compares size, mtime and inode against the index, and readdir()s every directory to find untracked files. It doesn’t read content by default, only when stat() disagrees with the index it is tracking. This is fine on a local SSD and also happens within a few milliseconds. It’s not that efficient on NFS, all of those calls have a network overhead.

It gets worse though. The NFS client caches attributes for anywhere between 3 and 60 seconds, and S3Files mounts with the defaults. Basically for about a minute, calls after you start walking the project are cheap, then go cold. An interactive canvas open almost always lands on a cold cache. On a 300-file project, we measured the plain lstat() loop at 1-2ms warm and 238-660ms cold; a 1000-file project pays 2 to 3 seconds. The cost is linear in file count, but every unit is a round trip and it’s paid every time git builds its index when it needs to find untracked/changed.

It gets even worse yet again. Remember the 128 KiB quirk from above? It turns a slow answer into a wrong answer. git decides a file is unchanged from stat() alone, without ever knowing the bytes are not resident. When it decides to hash something, it reads exactly the number of bytes stat() reported. If the NFS attribute cache is stale, or the data is still arriving, the read hits EOF early and hashing fails with an error message that ends in a bare colon and an empty errno.

On local machines this is not visible, but when we started testing on large scale mock data, we saw the fingerprint of all of the above gotchas and errors around 10-15% of the time (I am a huge proponent of observability and testing). We quickly came to the conclusion that there had to be some workaround for production.

ripping out the subprocess: pygit2 and threadpools

We knew we couldn’t bend the git binary to our will, so we looked for a library, pygit2 is a wrapper over libgit2, a C library which is almost a 1:1 implementation of git, even more performant in some very specific cases.

Before we could control how git works though, there was a separate nightmare: stalling the event loop. Our backend is heavily asynchronous. If pygit2 blocked the thread to write a commit to the NFS mount, it stalled the entire server, disconnecting active sockets.

The saving grace of pygit2 is that because it’s a C binding doing heavy I/O, it drops the GIL. So, we wrapped all of our git operations inside an asyncio.to_thread pool. This created a dedicated fleet of background threads doing the heavy, synchronous NFS disk I/O, allowing our main async event loop to keep streaming tokens and handling WebSockets without skipping a beat.

This made us non-blocking. It did not help us with performance. repo.status() in libgit2 does the exact same walk that git does, just in C, and when NFS is cold, it costs the same round trips. We had only moved the slow thing off our main thread.

working around the NFS cost

There were two separate problems hiding in that one git add ., and they needed two separate fixes.

The first is the walk. git has to discover what changed, so we started discovering things for git. We already know which paths an action has touched, because it is us who wrote them. So instead of git add ., we stage exactly those paths and nothing else. Two files changed means two stats, not a walk over the whole tree. The signature (paths_modified and paths_deleted parameters) in the code below is the difference between seconds and milliseconds.

The second is trusting stat(). Even for those two paths, we do not let pygit2 hash for us. Each selected file is read once to EOF, captured as an immutable blob, and the blob id is what gets staged and compared against HEAD. Files over 1 MiB spool to local temp files. If the bytes are not resident yet, the read waits for them instead of hashing a size that was never true.

Finally, after all that, we had a versioned file system committing directly to an S3Files mount without choking the event loop. Not memory speed, the index and every blob still live on NFS, but the cost is now proportional to what has actually changed, rather than to the size of a directory/project/canvas.

async def commit_scoped(
    self, 
    paths_modified: list[str], 
    paths_deleted: list[str], 
    message: str
) -> str:
    def _do_commit() -> str:
        # open a fresh handle inside the thread
        repo = pygit2.Repository(self.repo_dir) 
        index = repo.index

        # refresh the in-memory index of pygit2
        index.read()

        # we only touch what has been mutated: no walk, no discovery
        for p in paths_modified:
            index.add(p) # we actually read the whole file to EOF and stage it ourselves as a blob id, omitting that for simplicity
        for p in paths_deleted:
            try:
                index.remove(p)
            except KeyError:
                pass

        tree_oid = index.write_tree()
        new_oid = repo.create_commit(
            "HEAD", sig, sig, message, tree_oid, [head.id]
        )

        index.write() # persist it so the next thread's fresh handle sees the blob ids

        return str(new_oid)

    # offload to a threadpool
    return await asyncio.to_thread(_do_commit)

flowchart TD
    subgraph EventLoop [Main Async Event Loop]
        WS(WebSockets)
        Tokens(Token Stream)
    end

    subgraph ThreadPool [asyncio.to_thread Pool, GIL released]
        Paths[paths_modified / paths_deleted<br/>no tree walk, no stat trust]
        Blob[read each path to EOF<br/>stage as blob id]
        PG[pygit2 / libgit2<br/>write_tree + create_commit]
        Paths --> Blob --> PG
    end

    Mount[(S3Files mount over NFS<br/>.git/index + objects)]

    EventLoop -.->|offload sync I/O| ThreadPool
    Blob -.->|one read per path| Mount
    PG ==>|blobs, tree, commit| Mount

But fast storage is only half the battle. Staging by path only works when the server knows the paths, and it does not when agents are running arbitrary shell commands from within a sandbox. Also, humans and agents were mutating the same repository now, sometimes at the exact same moment. A git index does not help us with those. Read on for the next set of problems.