This article was originally written in Japanese and translated into English with AI assistance. Please note that some expressions may carry nuances from the original Japanese.
🇯🇵 日本語版はこちら / Japanese version
Series: DataMigrator Development Journal (Part 1)
In my previous blog posts, I described how, over the four months from January to April 2026, I could barely get to my main business of AI development, and how instead three unplanned products got off the ground — and how, on top of that, I kept getting absorbed in other new ideas, leaving my main business stuck.
Starting with this article, I’d like to write up the development record of the first of those products — “DataMigrator” — step by step. This first installment is about why I decided to build the tool in the first place. I’ve touched on it briefly in earlier blog posts, but here I’ll go into a bit more concrete detail.
To be honest, the development itself isn’t finished yet. Even whether it will ever be finished is, in my own mind, about a fifty-fifty proposition. It’s less “I’m writing this because I expect to complete it” and more “I want to leave a record of the whole process, stalls and all.” Hello — I’m Kanazawa, the representative of Soul Resonant Works.
The Day the MacBook Pro Arrived
At the start of 2026, I bought a new MacBook Pro M4 Max. As I’ve mentioned in earlier blog posts, I bought it for my main business: developing local AI systems.
For someone who has used Macs for years, the moment a new Mac arrives is a small ritual. It’s not something you buy often — it’s a once-every-few-years event — so I unboxed it while noticing how the packaging had changed. Lately, as with the iPhone and the like, there was tape sealing the gap between the base and the lid to guarantee it hadn’t been opened.
The iMac I’d been using had hit its performance limits, so I was counting on finally being able to start implementing the seven AI systems for which I’d only written the design documents — and on a dramatic performance boost for the other apps that lean on processor power and memory (video and the like).
But within a few hours, that excitement was overwritten by a different feeling.
“Wait — how am I going to get all the data I’ve been using over onto the new Mac?”
It should have been simple. Apple has a handy feature called Migration Assistant that moves your data wholesale from the old Mac to the new one at the push of a button. I’d used it on previous upgrades. But this time, I couldn’t bring myself to just run it.
Twenty Years of “I’ll Sort It Out Later” Had Piled Up
There was one reason: the old iMac’s contents — and the storage devices around it — were in a completely disorganized state.
Originally, because I’d been using external HDDs to store photos and videos, I had the iMac’s internal drive plus external HDDs, and on top of that a separate external HDD for Time Machine that backed up the whole thing.
Since the free space on the internal drive directly affects the Mac’s performance, I’d been moving files around as needed.
My storage setup at the time was:
1. An internal Fusion Drive, 3TB
2. External HDD 1: a 4TB drive I’d named “4TB-01”
3. External HDD 2: a 4TB drive I’d named “4TB-02”
4. External HDD 3: a 12TB drive to back up all of 1–3
Drives 2 and 3 were housed together inside a single HDD enclosure.
Since the enclosure had no RAID function, it amounted to two separate 4TB volumes — 8TB in total.
Across this storage, files I’d offloaded for reasons like “the iMac was running out of space, so I moved it,” “I copied it as a backup,” and “I figured it’d be safe to stash here” had quietly piled up.
An enthusiastic amateur at many things, I had stage photos and performance videos from theater, band recordings, photos I’d been accumulating since around 2004, and more — over twenty years’ worth of digital debris scattered across multiple drives.
The worst of it: there were things I’d copied as backups but then, out of laziness, never deleted from the original location. At first I’d felt something like “I’m not sure it copied correctly — what if the copy failed?” — and then I’d lose track of which one was the master, leaving huge folders and files existing in multiple separate places.
There were multiple sets of duplicate data ranging from hundreds of gigabytes to terabytes in size.
Photos were sorted out fine thanks to iCloud, but video files and the like were another story, and multiple copies had ended up existing.
If I ran Migration Assistant in this state, I’d unquestionably end up with a “copy of the mess” on the new Mac. It wouldn’t all fit capacity-wise either, so some kind of cleanup was needed beforehand.
A Scale No Amount of Manual Work Could Handle
At first, I thought I’d manage it by hand.
To start, I’d take the latest version of the Time Machine contents as the baseline and delete what wasn’t needed.
The process of judging whether something is unneeded is deeply unproductive, and I just can’t warm to it.
Whether the latest Time Machine backup truly reflected the most current state, whether all the files were gathered in one place — just sorting out questions like these took an enormous amount of time.
Then I set about hunting through multiple HDDs for copies of what was in that backup, and checking whether they were perfectly identical.
But I could never get real certainty that “this data truly, genuinely has no differences.”
On top of that, the macOS file system has all sorts of traps you can’t see from Finder. Right-click a folder and hit “Get Info” to see its size, and the spinner just starts turning and never comes back. Folders synced to iCloud have placeholder files mixed in whose contents haven’t been downloaded, so their sizes can’t be trusted. There’s a concept called “purgeable” space, where the displayed free space and the actual free space don’t always match. Invisible files like .DS_Store, and the OS’s auto-generated search indexes, also get in the way of diff comparisons.
I also tried dropping into the terminal to handle it with commands. Using rsync to compare folders and surface the differences; using the find command to extract files matching specific conditions; using shasum to compute hashes and judge whether contents were the same — I tried each one, one at a time, consulting AI along the way.
It works up to a point. But after rsync spits out tens of thousands of lines of logs, a different problem emerges: how do you read those logs? When information like “this file is newer, but that one over there is older” stretches on for tens of thousands of lines, a human simply can’t process it anymore. You could write a script to parse the logs — but I didn’t have the knowledge to do that, and trying to pull it off together with AI would have been a major undertaking in itself.
Even trying to feed the log files to an AI, I’d immediately hit the character limit, and the AI would simply refuse to read and analyze the whole thing — the obstacles were piled high.
And even checking the properties of each folder and file one by one, I couldn’t determine whether they were “truly, genuinely the same file.”
One comparison method the AI taught me was computing hash values. By reading the entire contents of an individual file and converting it into a fixed-length string (a hash value), you can determine whether two pieces of data are perfectly identical. Because the method disregards information like the file name and creation date and works only on the actual data portion, it can reliably capture whether the underlying data is the same or not.
For someone like me — the type who’s scattered the same data across multiple places — I thought this was the best approach.
But when there’s too much data, and each individual file is too large (videos especially), computing the hashes could easily take a day or two. This was partly a matter of the Mac’s processor power, and partly the slow transfer speed of the external HDDs (around 100 MB/s effective over USB 3.0), so things crawled along and only the waiting time piled up.
And the biggest problem was that no matter how carefully I worked, the fear that “I might accidentally delete a file I need” was always with me. Some of it — like the theater stage photos — can never be recaptured. The one thing I had to avoid was a folder I’d been thinking “I’ll sort it out later” turning out to be the only original. Because of this fear, the time my hands sat frozen grew longer than the time I spent actually working.
The Day I Vented to an AI
Then one day, I found myself explaining the situation to Claude. It started as a technical question. Something like “rsync is producing logs like this — how should I read them?” or “what’s a good way to compute hashes?” — about that level, I think.
But as we talked, it gradually turned into something closer to venting. I’d bought a new Mac for the sake of data migration, yet the migration itself was nowhere near done. Manual work had its limits. None of the existing tools fit my problem exactly. At this rate, I’d never get to my main business of AI development —
And then it struck me: “Could I actually build a tool to solve a situation like this myself?”
Of course, there are tools out there that address similar problems, but they inevitably fall short of what I had in mind in some respect.
Even if a tool could answer “compare files A and B,” it couldn’t address the separate question of “which directories does my data even live in? Is anything being missed?”
On Mac and Windows alike, when you migrate data, there’s an anxiety around “have I really covered all of it?”
Your data is made up of system data, libraries, personal-attribute data, the apps themselves, user data, and so on. It would be great if something could guarantee “copy just this directory and you’re absolutely fine; there’s definitely no necessary data stored anywhere else” — but individual apps tend to squirrel away important data in their own separate locations, so this becomes a major challenge too.
You also need a feature that scans for “where is my data in the first place?”
Claude’s answer was, roughly, to this effect. Technically, most of it could be done with Python’s standard library. A three-stage approach seemed realistic: first narrow down candidates by file name and size, then compare quickly using partial hashes (a hash of the first few kilobytes of a file), and only take a full hash of the suspicious ones to confirm. The GUI could be built with something like PySide6. There are commercial utilities, but surprisingly few tools go far enough to handle Mac-specific issues (iCloud, purgeable space, invisible files, and so on). So there did seem to be value in building it myself.
Even hearing that, whether I could actually build it was another matter. After all, I have almost zero programming experience. As I’ve written in earlier blog posts, I washed out on C back in my student days.
Even so, I felt myself waver just a little. I had, after all, just finished writing the design documents for the seven AI systems through dialogue with Claude in the very same way. Maybe, using that same method, I could this time build a “working app” — or so I thought.
What I Meant as a “Byproduct” Became the Prelude to the Main Business
And so the data migration itself went on pause, and development of a tool to assist with data migration began. Thinking about it calmly, this is a pretty absurd story. I’d bought a new Mac in order to get to work on my main business — the seven AI systems — and yet here I was, starting development of a different piece of software as preparatory work for moving data onto that Mac.
That said — and this may sound like an excuse — within my own mind it wasn’t such a contradictory decision. There were three reasons.
The first was a calculation: weighing the time to sort everything out by hand against the time to build a tool, the latter might be the lesser evil. Of course, there was a chance I’d stall partway through building it. Even so, since I couldn’t picture a future of continuing the manual work, I figured it was worth the gamble.
The second was a guess that the problem troubling me is probably troubling others too. Among people who’ve used Macs for over twenty years and juggle multiple storage devices, I surely can’t be the only one. If it came together, it might serve as Soul Resonant Works’ first product — something others could actually use. Before putting out AI-related products, gaining the experience of “seeing a smaller product all the way through to the end” once felt like it wouldn’t be a bad thing.
The third was that it was just the right scale to serve as practice for the development style called vibe coding. The seven AI systems are each complex, beyond what I could handle on my own. But a data migration tool — if I kept the features focused — might be manageable for one person (plus AI). The idea was that the experience I’d gain here should pay off for my main business of AI development.
Of course, all of these reasons are after-the-fact rationalizations — “I suppose you could put it that way, in hindsight.” The moment I actually decided to build the tool was, I think, far more impulsive. The ratio was something like eighty percent “I just can’t stand continuing this manual work anymore,” and the remaining twenty percent the logic I laid out above.
Be that as it may, from this impulsive decision, a tool named DataMigrator began to take shape. And as I’ve written in earlier blog posts, this decision was the beginning of “the main business receding into the distance.”
Next Time
Next time, I’ll bring you the story of how I started writing the requirements definition for DataMigrator. Rather than diving straight into code, I began by defining “what to build” in words. I first started writing it using an AI other than Claude, but when I had Claude review it, I was told “the problem is framed in high resolution, but the solution is rough” — and as a result, I ended up revising it five times, from Rev.0 through Rev.5. It’s a story about what it means to have an AI write a requirements definition, and what it means to have an AI shoot down your work over and over.
About Soul Resonant Works
Soul Resonant Works is a solo venture developing seven local AI systems. Starting from zero programming experience, the development is progressing through collaboration with AI.
🌐 Soul Resonant Works:
→ https://www.sr-works.net/en/
📝 This blog publishes the entire development process as a serialized journal.
If you found this article useful, please share it.
Leave a Reply