Most data recovery cases we write about can be handled on a workbench. This one needed an entire rack.
The client was running a three-node BeeGFS cluster when one of the storage targets stopped serving files correctly. Each node contained around 489 TB of chunk data stored across a 32-drive RAID 6 array. Two of the nodes were still operating normally. The third had already been through a filesystem repair, and that repair was what turned the problem into a much more complex recovery case.
An IT technician had run a filesystem check on the XFS volume used by the chunk store. The intention was to restore normal operation, and xfs_repair did exactly what it was designed to do. When XFS encounters directory entries that it cannot validate, it disconnects those objects from their original directory structure and places them in lost+found. This allows the filesystem to mount cleanly again.
The repair completed successfully and the filesystem mounted. However, 82,547,185 files were now located inside lost+found, spread across approximately 1.84 million numbered subdirectories. The filenames were still there, but the original directory paths had been lost.
For BeeGFS, those paths are essential. If the location of a chunk changes, BeeGFS can no longer find it. The cluster could therefore report petabytes of healthy storage capacity while still being unable to access a significant proportion of the data.
This case study explains how we reconstructed that data, the tools we had to develop during the process, and some of the most important lessons from the recovery. Several times, our own reconstruction initially produced the wrong result. What mattered was identifying those errors through validation before they had any effect on the client’s data.
Why the path is the data
It is important to understand why moving files into lost+found is disastrous for a clustered filesystem, while on a general-purpose system it may be little more than an inconvenience.
On a standard Linux server, a file recovered into lost+found with its contents intact can usually be restored. Someone can open it, identify it and return it to the right place. The contents provide the context.
BeeGFS works differently. Its metadata servers store the namespace and directory tree visible to users, while the storage targets contain only opaque chunks. A chunk is identified entirely by its location:
chunks/u<uid>/<hash1>/<hash2>/<parentEntryID>/<entryID>
The metadata server calculates this path from the file’s attributes and requests that exact location from the storage target. There is no search, fallback or inspection of the contents. If the chunk is not where expected, BeeGFS considers the file missing, even if the bytes remain intact on the same disk only a few directories away.
As a result, xfs_repair preserved 100% of the data while making 100% of it unusable. Every byte remained. None of it was accessible.
That distinction defined the whole recovery. We were not recovering the data itself. The data was intact. We were rebuilding its addresses.
Step zero: copy everything before touching anything
Before we ran a single command on the client’s hardware, every drive in the affected node was imaged onto an identical drive. Thirty-two source drives, thirty-two destination drives, copied sector by sector.
On a case like this, that is more than a precaution. Every method described below involved some degree of experimentation, and several failed on the first attempt. Working from a byte-for-byte copy meant a failed test cost us a few hours rather than causing permanent data loss.
It also meant that when we later needed to scan the raw volume for deleted structures, we still had access to its pre-repair state. That became crucial during Phase 3, months into the recovery, although nobody could have predicted it at the start.
There is another reason that matters just as much. Once a verified image exists, you can work more freely. Recovery slows down when every command might make the situation worse. Engineers become cautious, add extra checks and avoid tests that could otherwise answer a question in minutes.
With a verified image available, the risk changes. You can test an approach that may fail because the consequences of getting it wrong are limited.
If there is one principle to take from this article, it is this: image first, diagnose second. One of the costliest mistakes in enterprise recovery is running a repair tool against the only copy of the data.
Recovering an environment of this size required a carefully controlled server data recovery process that preserved the original disks while the storage structure was reconstructed.
Rebuilding a 500 TB RAID 6 by hand
The array consisted of 32 Seagate 18 TB enterprise drives, each providing 16.37 TiB of usable capacity. A commercial recovery suite detected the array automatically and showed a volume with an XFS partition beginning at sector 2048. That was promising, but at this scale the GUI tools could not do much with it. Scans stalled, jobs failed to complete, and progress bars ran for a day before resetting. We needed direct, scriptable access to the assembled volume.
So we built our own user-space RAID 6 mapper.
On paper, the detected parameters looked straightforward: left-symmetric layout, 256 KB stripe unit, parity delay 1, P first and Q second, and Galois field polynomial 0x11d. The drive order shown in the RAID panel was correct. Yet the reconstructed volume was still wrong.
The problem was the rotation origin. On stripe 0, this array places P at position 30 and Q at position 31, with the data continuing forwards from P+2. Linux md’s left-symmetric convention places P at position 31 on stripe 0. Both models are internally valid; they simply count the rotation from different starting points. The fix is a one-position shift. Set rotation_shift = 1, or rotate the drive order one position to the right and leave the shift at zero. Both approaches produce the same mapping.
This sort of detail can cost days when tackled by trial and error because an almost-correct RAID mapping can look convincing. Partition tables parse, the first superblock reads correctly and directories appear. Only further into the volume does the output collapse into noise, by which point you may have spent hours analysing data built on the wrong foundation.
We verified the mapping independently in two ways before trusting it:
We sampled stripes at random across the volume, calculated the P and Q syndromes for each candidate layout and checked which one balanced correctly. Only one passed: 60 out of 60 stripes matched for P, and 60 out of 60 for Q.
All other candidate layouts failed on most of the tested stripes. This provides strong validation because P and Q are calculated differently. P uses a simple XOR, while Q uses a Galois-field weighted sum, so a layout that matches one by chance is extremely unlikely to match the other.
We took a small reference image produced by the commercial tool and compared 256 KB chunks from the raw drives against it by content. The result was the same, reached without relying on the parity calculations.
The more important confirmation came next. XFS stores a superblock at the beginning of every allocation group. This volume contains 492 of them, distributed evenly across all 540 TB. Each secondary superblock holds the same UUID, block size, AG size and total block count as the primary. If the RAID geometry is wrong by even one position, those secondary copies appear in the wrong locations and no longer parse correctly.
We tested eight of them at intervals extending to 536 TB. Every one parsed successfully and reported the same values: 4 KB blocks, 492 allocation groups and a single consistent UUID. This verified the block mapping across the full length of the array rather than only its first few gigabytes.
We now perform this check before proceeding with any large array reconstruction and recommend the same approach to others. Filesystems that distribute redundant metadata throughout the volume, such as XFS allocation groups, ext4 backup superblocks and ZFS uberblock arrays, provide useful checkpoints for validating RAID geometry at scale. Use them. A RAID mapping checked only at the beginning of a volume has not been fully verified.
For reference, the actual volume size is 539,969,999,339,520 bytes. That equals 540.0 TB in decimal units, or 491.1 TiB. The difference matters when calculating how much external storage is needed for the image, and it led to a real procurement discussion before the recovery began.
Understanding the scale of the problem
Once the array was rebuilt and readable, we could properly assess the damage. A directory listing from the storage root made the situation clear:
Two figures stood out. The lost+found directory had a link count of 1,843,952, indicating around 1.84 million subdirectories, while the directory inode itself was 514 MB. This was clearly not a filesystem in a normal state.
The other notable detail was the empty buddymir directory. BeeGFS supports buddy mirroring, which stores a second copy of each chunk on a partner target. It was configured on this cluster but was not being used. Three storage nodes, but only one copy of the data. We will return to that later.
Scanning the volume into a database gave us the exact count: 82,547,185 files under lost+found, on a filesystem that should have contained roughly 147 million.
The tooling choice that made everything else possible
Before looking at the recovery phases, one decision is worth explaining because it shaped the entire process: almost all of our analysis was carried out in DuckDB.
Handling tens of millions of path records is not a job for shell scripts. Simple approaches quickly fail when it comes to join performance. Our first relocation script exported the complete route list again and rescanned an ever-growing log file for every batch of 200 files. That O(N²) design would have taken weeks on this dataset, something we discovered only because we tested it first on synthetic data.
Using a columnar analytics engine that runs on one machine, reads Parquet natively and can join hundreds of millions of rows turned days of scripting into straightforward SQL queries. During the recovery, we joined:
43 million reconstructed routes with 408 million metadata records
39 million unrouted files with 278 million entry IDs from healthy nodes
1.1 million scanner results with 99 million live filesystem entries
Each of these operations completed within minutes on a single workstation. Doing the same work with scripts would not have been practical.
The wider lesson is that recovery at this scale is as much a data-engineering problem as a storage-engineering one. Reading bytes from disk is not the difficult part. The real challenge is combining tens of millions of facts from four independent sources and deciding which one to trust when they conflict. That became the central question of the entire project.
Phase 1: working backwards from the repair log
xfs_repair produces detailed output. As it detaches objects, it records what happened. In practice, that log gave us a partial history of the directory tree that had existed before the repair.
We parsed the repair output and reconstructed two key pieces of information: the original path of each orphaned object and the parent-child relationships between directories. We loaded these into DuckDB alongside a complete scan of the current filesystem state, creating three databases totalling around 10 GB.
The routing process combined three lookups:
The first pass produced target paths for 43,222,159 files. We then identified the structural pattern that made the remaining work manageable:
xfs_repair moved entire directories, rather than individual files.
This meant the physical location beneath lost+found/<inode>/… and the reconstructed destination path shared the same suffix:
source /brick/storage/lost+found/302914319544/4/1A70-691C4D29-1/323F-691C4D56-2
└────────── suffix ─────────────┘
target /brick/storage/chunks/u80D/691C /4/1A70-691C4D29-1/323F-691C4D56-2
When we tested this pattern, 42,870,846 of the 43,222,159 routes, or 99.19%, matched it. The remaining 351,313 were nested files where the basenames agreed but the immediate parent directory names differed. This was not random noise, but a specific and explainable conflict between two sources.
We used that agreement as a filter. Only routes supported by both the reconstruction and the physical evidence were included in the first move. The 351,313 disputed cases were left untouched. It seemed cautious at the time, but became one of the most important decisions in the project.
Restoring the matching set took 1 hour 10 minutes for 42.87 million files, averaging about 10,000 files per second, with two conflicts and no errors.
How the mover works
A few notes on the mechanics, because at this scale the implementation details matter.
Moves use link() + unlink(), never copying. Creating a hard link and then removing the source is effectively a rename: the inode stays the same, no file data is read or written, and ownership, permissions, timestamps and extended attributes are preserved. It also provides the atomic behaviour we need. link() returns EEXIST if the destination is occupied, so an existing chunk cannot be silently overwritten. mv -n may look equivalent, but it uses stat followed by rename, leaving a window where a concurrent write could be lost.
Every move is recorded in a ledger before the next batch starts. That ledger makes the operation reversible. A rollback script reads it, reverses the source and destination paths and restores everything. At this scale, having an undo path is essential. We tested it on a sample corpus and confirmed that the directory tree produced identical hashes before and after a complete move-and-rollback cycle.
The mover reads only a flat route list. There is no database, locking or per-file query. Progress is stored as a byte offset in that file after each batch, so an interrupted run resumes within 20,000 routes of its previous position and a crash costs only seconds.
Worker processes make syscalls rather than forks. The first version launched mkdir, ln and rm for every file, creating three processes for each of 42 million files. Throughput was 215 files/sec. Switching to Python workers calling os.link() and os.unlink() directly reduced the same test from 4 minutes 38 seconds to 3 seconds. That was a 130× improvement simply from removing process creation, with no change to the underlying logic.
Testing the worker count on real hardware showed that 32 workers comfortably outperformed 8, while 64 performed worse than 32. Beyond a certain point, additional concurrency only creates contention on the XFS log and directory locks. It is better to measure than assume.
Fast turnaround times for business-critical data
Phase 2: reverse-engineering the BeeGFS inode table
Phase 1 covered the files recorded in the repair log, leaving around 39 million with no route information. For those, we needed another reliable source, and BeeGFS provided one through its own metadata.
A week before the repair, the client had run beegfs-fsck, which exported the metadata servers’ view of the filesystem into several dense binary tables. The table we needed, fileinodes, was 45.7 GB. Its format is not publicly documented, so we decoded it manually from hex dumps.
The structure consisted of a 4,176-byte header followed by fixed 112-byte records. Within each record, these were the fields we needed:
From those fields the chunk path falls straight out:
chunks/u{uid:X}/{parent-timestamp hex[0:4]}/{parent-timestamp hex[4]}/{parentEntryID}/{entryID}
Before relying on the formula, we tested it against real paths from the client’s filesystem. This included the two paths that had appeared as conflicts during Phase 1, and it reproduced both exactly.
BeeGFS environments often use large and complex storage configurations. Learn more about enterprise RAID data recovery.
Three times we got it wrong
The parser did not work on the first attempt, and its failures were the most useful part of the process. We designed it to stop whenever a record failed validation instead of skipping the record and continuing. That approach exposed three separate errors that could otherwise have produced convincing but incorrect results at scale.
- Error one: mistaking a flags field for a signature. We initially treated the u32 at +28 as a fixed 0x20000623 value and used it as a record marker. In the real file, 6.46% of records failed that test. Examining them revealed 0x20000622, differing by one bit, with otherwise valid data. It was a flags field rather than a signature. Both record variants were legitimate.
- Error two: zeroed parent fields. Some records contain 0-0-0 in the parent field, while the actual parent is stored in the original parent field. Our formula would have created chunks/u7D1/0/0/0-0-0/… for all of them without reporting an error. That is the dangerous kind of failure: plausible, silent and wrong. We caught it because we first tested the parser on a 200 MB sample and reviewed the distribution before processing the complete file.
- Error three: variable stripe width. After correcting the first two issues, one worker still stopped on 18 records out of 50.9 million at a perfectly aligned offset. The hex contained 0x20000423 instead of 0x20000623, with two entries in the stripe target list rather than three. Byte +29 was not part of a signature. It represented the byte length of the stripe target list. Those 18 records belonged to files striped across only two of the three targets, valid data that our parser had rejected.
We found each issue using the same rule: validate everything, stop on anomalies, inspect them, understand the cause, then expand the rule. Simply skipping records that failed to parse would have created a dataset that appeared complete while silently omitting millions of records.
The final result was 407,833,733 validated records stored in a 6.3 GB Parquet file. We kept only integers and generated path strings during queries, because materialising those strings would have tripled the file size without providing any benefit.
Trust the experts with proven results
Then we checked it against reality, and it failed
This is where the project became particularly interesting.
We now had two independent reconstructions: the repair-log results from Phase 1 and the inode table from Phase 2. They disagreed on 765,220 files. The obvious approach was to trust the more reliable source, but we had no way of knowing which one that was.
The client’s two healthy nodes gave us the answer.
Because files are striped across all three targets using the same relative path, the correct directory structure for the damaged node still existed intact on brick001 and brick002. We asked the client’s team for a listing. They provided full scans containing sizes, microsecond-precision timestamps and inode numbers, with a separate database for each node.
Together, these contained 10,972,742 real chunk directories and 277,743,284 distinct entry IDs, with no ambiguity. No entry ID pointed to two different directories on either node.
Before treating this as ground truth, we ran a control test. We took 196,510 placements where both reconstructions agreed and checked whether those directories existed on the healthy nodes:
196,510 of 196,510. 100.00%.
That result gave us confidence in everything that followed. It showed that the healthy-node directory set had no meaningful gaps, so a directory missing from it was genuinely not a valid BeeGFS directory rather than the result of an incomplete scan.
We then repeated the test on the 414,896 disputed files:
The inode table failed. More than half of the parent directories it suggested for those files did not exist anywhere on the working filesystems.
This was not what we expected. We had reverse-engineered the inode table, validated the path formula against real data and seen it agree with physical evidence on another subset 350,281 times against 990. We had good reason to trust it. The healthy nodes showed otherwise, and unlike our results, they were not a reconstruction.
Closer inspection explained why. All 414,896 were bare files located directly under lost+found/<inode>/ with no surviving parent directory. In those cases, the inode table had no genuine parent information but still produced a value. For nested files where the parent directory survived, it remained accurate.
Two conclusions followed. First, the 42.87 million files moved during Phase 1 were correctly placed and required no reversal. Second, a source can be reliable for one group of files and unreliable for another, while overall accuracy figures hide that difference. Applying the inode table uniformly would have placed 414,896 files into directories that did not exist.
The 351,313 we held back
We applied the same validation to the disputed files from Phase 1. Of the 111,858 that could be checked against ground truth:
- Repair-log reconstruction: 0 correct. Not a single one.
- Inode table: 111,590 correct, or 99.76%.
Holding these files back from the first move had been the right decision, and we now had the evidence to confirm it. They were routed using the inode table during Phase 2.
Measuring accuracy rather than assuming it
Of the 20.3 million nested files that Phase 1 could not route, 5,327,469 had a matching file on a healthy node. This allowed us to compare the inode table results directly against ground truth.
5,285,754 of 5,326,860 correct. 99.23%.
That figure made Phase 2 defensible. It was not an assumption or a theoretical argument, but a measurement across more than five million files against a working filesystem, using the same group of files we were preparing to move.
The 0.78% that failed was concentrated rather than randomly distributed: 41,106 errors came from just 8,863 incorrect directories, with the largest single bad mapping affecting 7,496 files. The inode table fails at the parent-record level, not per file. This allowed us to create two safety filters based on the verified sample:
- The proposed parent directory must exist on a healthy node. This alone detected 30,863 of the 41,106 known errors, or 75%.
- The proposed parent must not appear on a blocklist of 8,863 directories that the inode table was proven to map incorrectly.
Together, these filters reduced the expected error rate from 0.78% to approximately 0.19%.
Phase 2 execution
The move was carried out in two separate tiers so we could verify the more reliable set before starting on the less certain one:
- Tier 1, ground truth. Files whose correct locations came directly from a working filesystem, with no inference involved. 5,439,318 files, 9 conflicts, 23m42s.
- Tier 2, inode table with both safety filters applied. 14,522,074 files, 13 conflicts, 40m44s.
The conflict rates were significant: 9 in 5.4 million and 13 in 14.5 million. If the inode table had been assigning files to incorrect locations, we would have seen far more collisions with files already in the right place. The execution results supported our earlier measurement.
Running total: 62,832,238 of 82,547,185 files restored, 76%.
That left 19,013,790 files sitting directly in lost+found with eleven-character names containing no dashes. xfs_repair had renamed them using their inode numbers, so their entry IDs, the only key linking a chunk to its file, were lost. Another 701,135 files were deliberately left untouched because no reliable destination could be established: 659,655 had proposed parents missing from every healthy node, 36,209 matched the blocklist, and 5,271 were completely absent from the inode table.
We do not make guesses with a client’s data. A file placed in the wrong location is worse than one left in lost+found because it appears to be correct.
What striping means for recovery
During Phase 2, the client mentioned something that changed part of our analysis. They had taken a chunk file, added .jpg to its name, and it opened as a complete image.
That would not happen if every chunk were only a fragment. It occurs when a file is small enough to fit within a single 512 KB chunk, meaning the entire file sits on one target and the first chunk contains the header.
Looking at one directory across all three nodes made the pattern clear:
The same filenames appeared on each node because the chunk name is the file’s entry ID, which belongs to the file rather than the target. Sizes differed because each target stores different byte ranges. There were also gaps because smaller files exist on only one target.
2,097,152 is exactly 4 × 512 KB, while 1,572,864 is 3 × 512 KB. These are complete stripe multiples, while the irregular values are tails.
This explained a discrepancy we had been investigating. Nested unrouted files appeared on healthy nodes 26.2% of the time, compared with only 2.8% for files from the Phase 1 group. That was almost a tenfold difference between two populations from the same disk.
The reason was file size. Nested files averaged 8.49 MB, while loose files averaged 1.62 MB. A file reaches a second target only after exceeding one chunk, so larger files are naturally more likely to have a counterpart elsewhere. Both measurements were correct. We had simply compared populations with different striping behaviour and treated the difference as significant.
It is a small detail, but overlooking details like this can lead to a confident but incorrect conclusion.
The practical consequence
A striped file requires its chunk from every target. Recovering 62.8 million chunks does not mean that 62.8 million files are now readable. A file can only be opened when all of its chunks are present. If even one is missing, the file remains damaged regardless of how many other chunks were restored.
To measure this, we built a small utility. Using the file size, chunk size and number of targets, it calculates how many bytes should exist on each target and compares that with what is actually present.
file size : 5242880 bytes (5.00 MB)
stripe count : 10
expected bytes per target:
target 0 : 2097152 (4 stripes)
target 1 : 1572864 (3 stripes)
target 2 : 1572864 (3 stripes)
observed vs expected:
target 0 : observed 2097152 expected 2097152 match
target 1 : observed 1572864 expected 1572864 match
target 2 : observed 0 expected 1572864 DIFFERS
MISSING 1572864 bytes (30.0% of the file)
Chunk counts measure the recovery process. This measures the condition of the file, which is what matters to the client.
Phase 3: the 150 TB that had disappeared
With most of the directory tree rebuilt, we could properly compare capacity across the cluster. The two healthy nodes were almost identical:
They matched to within 0.002% by capacity and 0.020% by file count. BeeGFS had distributed the data almost perfectly evenly, making either healthy node a reliable model for what the third should contain.
After both recovery phases, the repaired node contained:
Around 30% of the target was simply missing. It existed in neither chunks/ nor lost+found/, with no remaining filesystem record.
The missing files averaged 2.97 MB, compared with 3.17 MB on the healthy nodes. That was close enough to suggest a representative cross-section rather than one specific type of data. It was not simply a case of large files disappearing or one user’s directory being lost.
Where did it go? Most likely through a combination of the xfs_repair operation and earlier recovery attempts made before the client contacted us. Blocks were freed, metadata was discarded and some data was later overwritten. This is why advice to stop further repair attempts matters: every additional intervention can reduce the amount of data that remains recoverable.
The disk contains more than the filesystem can see
A partial scan of the original image, covering about 5% of the volume, identified 1,111,960 files within the raw structures. Of these:
- 202,009 also existed on the live filesystem, with 100.00% agreement on size. This fully validated both the scanner output and our join method.
- 909,951 appeared in neither chunks/ nor lost+found/. Together they represented just under 2 TB. The filesystem had lost track of them, but the data was still on disk.
The next question determined whether Phase 3 was practical: could we identify where those files belonged?
840,479 of 909,951, or 92.37%, had a target path that could be calculated from the inode table. Of those, 833,264 (99.14%) pointed to a parent directory that existed on a healthy node.
That figure captures the recovery model. These files had no directory entry, path or filesystem record, yet we could still place 92% of them because the metadata servers had stored the parent relationship independently of XFS, and the result could be checked against a working filesystem.
Extrapolating from the 5% sample suggests roughly 18 million files and 40 TB, around 38% of the missing files by count and 27% by volume. Whether the remainder is recoverable depends on how representative that sample was, which is something we are still measuring rather than assuming.
Building an XFS scanner for a half-petabyte volume
The challenge in Phase 3 was that none of our existing tools could handle the job. Commercial recovery software struggles with an XFS volume of this size. xfs_db is designed for debugging rather than scanning, and a simple brute-force scan of 540 TB at normal speeds would take weeks without producing useful partial results along the way.
So we built our own scanner.
It reads XFS v4 and v5 directly from a raw image or block device. Nothing is mounted and nothing is written back to the source. Crucially, it also uses two independent methods to locate every inode, because on a damaged volume the metadata trees cannot automatically be trusted.
The scanner works in three separate passes.
Pass 1 - the inode B+tree (fast)
Each allocation group contains an inode B+tree that records which inode chunks are allocated. By walking that tree and reading only the blocks it references, we can reconstruct the live namespace in inode-bytes rather than volume-bytes. On a 540 TB filesystem, that means reading a few hundred gigabytes instead of 540 terabytes. Hours rather than weeks.
Pass 2 - directory blocks, read in disk order
This pass delivers the biggest performance gain, mainly by reducing the amount of work rather than making each operation faster.
The obvious method is to walk the directory tree, opening each directory, reading its blocks and recursing. On spinning disks at this scale, that creates huge amounts of random I/O. Every directory can mean another seek, and there are millions of them.
Pass 1 therefore does not read directory blocks. It only records their locations. Pass 2 gathers all directory block offsets, sorts them into physical disk order and reads them sequentially, linking each block back to its owning inode. XFS v5 allows this because the directory block header stores the owner inode number. Each block identifies its owner, so the directory tree itself is not needed to interpret it.
Same data and same result, but sequential rather than random. In practice, this is typically 10 to 50 times faster.
Pass 3: a raw sweep focused only on free space
Passes 1 and 2 reconstruct what the filesystem still knows about. Pass 3 looks for what it has lost: deleted inodes, orphaned directory blocks and filenames that remain in unused areas of live directory blocks.
When a file is deleted from an XFS directory, its record is marked as unused, but the underlying bytes are not immediately erased. The inode number and filename often remain readable until new data overwrites them. Our parser examines these unused regions and carves out the surviving records. This allows a deleted file to recover its name instead of becoming a numbered blob, which in this case can mean the difference between an identifiable chunk and an anonymous one.
A raw sweep is expensive, so we made it selective. Before scanning, the tool walks the free-space B+tree and identifies the exact byte ranges not allocated to any file. Deleted metadata can only remain in free space. Limiting the sweep to these areas reduced the work on this volume by roughly threefold. The tool also reports the ratio beforehand, making it clear whether the scan is likely to involve hours or days.
Making the reads themselves fast
Work is divided across processes, each with its own file descriptor, so access is not serialised through a shared handle.
Reads use 64 MiB blocks and O_DIRECT where available. There is little value in filling the page cache with half a petabyte of data that will only be read once.
Physically adjacent inode chunks are combined into larger reads.
The magic-number check examines one byte in every 512 using a strided vector comparison, making candidate detection negligible compared with the I/O cost. Full parsing is performed only on matches.
XFS v5 inodes contain their own inode number and the filesystem UUID. Comparing both values with the location where an inode was found reduces false positives to almost zero, which is essential when scanning trillions of byte offsets.
Results stream to disk in row groups instead of building up in memory. Our first version buffered each worker’s entire slice and was terminated by the OOM reaper on a 32 GB machine. The streaming version keeps memory use stable regardless of volume size.
The results are written to sharded Parquet or TSV files, which DuckDB can read without a separate import stage.
The final stage uses SQL: full paths are resolved iteratively from the root inode and then classified. Live and reachable. Allocated but unreachable, the true orphans. Deleted, with a recovered filename and parent path where both survived. Finally, name-only recoveries where the filename remains but the inode is gone.
We validated the full pipeline using a synthetic XFS v5 image designed to include every case we needed to handle: a normal tree, a short-form directory, an inode in a freed B+tree slot, a deleted directory-entry remnant and an inode discoverable only through the raw sweep. Every case was recovered successfully.
What this case should teach the rest of us
Do not run a repair tool directly on the backing store of a distributed filesystem. xfs_repair did nothing wrong. It behaved as intended and the filesystem mounted cleanly afterwards. But a BeeGFS chunk store is not a general-purpose filesystem. The path forms part of the data’s identity. Preserving file contents while removing those paths leaves BeeGFS with nothing it can use. A clustered filesystem should be repaired through its own tooling. If that cannot resolve the problem, it is better to stop than move down to a lower-level repair tool.
Striping without mirroring still leaves a single point of failure. In this cluster, files were striped across all three targets while buddy mirroring was disabled. The buddymir directories were empty. Any file with a chunk on the damaged node therefore became unreadable while that node was offline, regardless of the condition of the other two. Three nodes of infrastructure, but only one copy of the data. It is worth checking on any BeeGFS deployment because mirroring is easy to postpone and costly to leave disabled.
Scale changes which tools are practical, not only how long they take. Techniques that work routinely at 10 TB can fail completely at 540 TB. GUIs stall, recursive scans may never finish and brute-force scanning can take weeks. Recovery at this scale becomes a data-engineering problem involving columnar databases, sequential I/O, parallel workloads and results that can be processed incrementally.
Validate RAID geometry across the full volume. A mapping that works at the start of the array but fails near the end can waste weeks of work. XFS provided 492 checkpoints distributed across the volume. Use them.
Stop on anomalies rather than skipping them. Our binary parser refused to produce output when it encountered a record that failed validation. That exposed three separate misunderstandings of the format, any of which could have generated millions of believable but incorrect results. A parser that silently skips what it cannot understand can quietly produce misleading data.
Measure each source against ground truth before trusting it. Our two reconstructions disagreed on 765,220 files, with no reliable way to choose between them. The healthy nodes resolved the question, including cases where they contradicted the source we had built ourselves and expected to trust. Every later decision was based on measured accuracy for the specific files being moved rather than general confidence in a method.
Overall accuracy can hide failures within specific groups. The same inode table was 99.23% accurate for nested files but only 46% accurate for bare files. A single overall figure would have been misleading. Validation should be segmented in the same way as the data.
Make every operation reversible and test the rollback. Every move in this recovery was recorded in a ledger, and the rollback process was verified by hashing the directory tree before and after a complete move-and-undo cycle. With 20 million files, systematic errors cannot be spotted manually. What matters is knowing that if one is discovered later, the operation can be safely reversed.
The work continues
At the time of writing, 62.8 million of 82.5 million files have been returned to their correct locations, while a full-volume scan is searching for the missing 149.62 TB. The remaining loose files, whose original names were replaced by inode numbers, are more difficult. The most promising approach is the raw sweep, which can recover deleted directory entries from free space and potentially restore those filenames.
Whether the client ultimately recovers close to 490 TB or less depends on how much of the missing data was simply detached and how much was actually overwritten. The full scan will provide the answer. What we already know is that 92% of the orphaned files in the initial sample have a computable destination. At this stage, the recovery limit is determined by what physically remains on disk and what was previously overwritten, rather than our ability to identify where the files belong.
Facing a large-scale storage failure?
Frequently Asked Questions
Why did moving files into lost+found make them inaccessible?
On BeeGFS storage targets, a chunk file’s location forms part of its identity. The metadata servers expect each chunk at a precise path calculated from details including the user ID, parent entry ID and entry ID. Even if the chunk itself remains intact, BeeGFS cannot access it once it is no longer stored at the expected path.
How many files were affected?
We identified 82,547,185 files inside lost+found. By the end of the first two main recovery phases, 62.8 million had been returned to defensible locations, representing approximately 76% of the affected set.
How can deleted files still be recovered from XFS?
Deleting a file often removes its filesystem references before the underlying data is overwritten. The custom scanner used in this case searches inode structures, directory blocks, unused directory-entry areas and selected free-space regions for surviving records that can identify files and recover their original names.
What should I do if my BeeGFS storage target is online but files are still missing?
Stop unnecessary writes and do not run xfs_repair again. A BeeGFS target may be online while files remain inaccessible because their chunk paths have been altered or lost. Preserve the drives first, then carry out diagnosis from an image.