{"id":41,"date":"2026-06-09T23:09:11","date_gmt":"2026-06-09T14:09:11","guid":{"rendered":"https:\/\/sr-works.net\/blog-en\/?p=41"},"modified":"2026-06-09T23:09:11","modified_gmt":"2026-06-09T14:09:11","slug":"falling-into-data-migration-hell-while-setting-up-for-local-ai-development","status":"publish","type":"post","link":"https:\/\/sr-works.net\/blog-en\/falling-into-data-migration-hell-while-setting-up-for-local-ai-development\/","title":{"rendered":"Falling into Data Migration Hell While Setting Up for Local AI Development"},"content":{"rendered":"\n

This article was originally written in Japanese and translated into English with AI assistance. Please note that some expressions may carry nuances from the original Japanese.<\/em><\/p><\/blockquote>\n\n\n\n

\ud83c\uddef\ud83c\uddf5 \u65e5\u672c\u8a9e\u7248\u306f\u3053\u3061\u3089 \/ Japanese version<\/a><\/em><\/p>\n\n\n\n

\n\n\n\n

Series: DataMigrator Development Journal (Part 1)<\/p>\n\n\n\n

\n\n\n\n

In my previous blog posts, I described how, over the four months from January to April 2026, I could barely get to my main business of AI development, and how instead three unplanned products got off the ground \u2014 and how, on top of that, I kept getting absorbed in other new ideas, leaving my main business stuck.<\/p>\n\n\n\n

Starting with this article, I\u2019d like to write up the development record of the first of those products \u2014 \u201cDataMigrator\u201d \u2014 step by step. This first installment is about why I decided to build the tool in the first place. I\u2019ve touched on it briefly in earlier blog posts, but here I\u2019ll go into a bit more concrete detail.<\/p>\n\n\n\n

To be honest, the development itself isn\u2019t finished yet. Even whether it will ever be finished is, in my own mind, about a fifty-fifty proposition. It\u2019s less \u201cI\u2019m writing this because I expect to complete it\u201d and more \u201cI want to leave a record of the whole process, stalls and all.\u201d Hello \u2014 I\u2019m Kanazawa, the representative of Soul Resonant Works.<\/p>\n\n\n\n

The Day the MacBook Pro Arrived<\/h2>\n\n\n\n

At the start of 2026, I bought a new MacBook Pro M4 Max. As I\u2019ve mentioned in earlier blog posts, I bought it for my main business: developing local AI systems.<\/p>\n\n\n\n

For someone who has used Macs for years, the moment a new Mac arrives is a small ritual. It\u2019s not something you buy often \u2014 it\u2019s a once-every-few-years event \u2014 so I unboxed it while noticing how the packaging had changed. Lately, as with the iPhone and the like, there was tape sealing the gap between the base and the lid to guarantee it hadn\u2019t been opened.<\/p>\n\n\n\n

The iMac I\u2019d been using had hit its performance limits, so I was counting on finally being able to start implementing the seven AI systems for which I\u2019d only written the design documents \u2014 and on a dramatic performance boost for the other apps that lean on processor power and memory (video and the like).<\/p>\n\n\n\n

But within a few hours, that excitement was overwritten by a different feeling.<\/p>\n\n\n\n

\u201cWait \u2014 how am I going to get all the data I\u2019ve been using over onto the new Mac?\u201d<\/p>\n\n\n\n

It should have been simple. Apple has a handy feature called Migration Assistant that moves your data wholesale from the old Mac to the new one at the push of a button. I\u2019d used it on previous upgrades. But this time, I couldn\u2019t bring myself to just run it.<\/p>\n\n\n\n

Twenty Years of \u201cI\u2019ll Sort It Out Later\u201d Had Piled Up<\/h2>\n\n\n\n

There was one reason: the old iMac\u2019s contents \u2014 and the storage devices around it \u2014 were in a completely disorganized state.<\/p>\n\n\n\n

Originally, because I\u2019d been using external HDDs to store photos and videos, I had the iMac\u2019s internal drive plus external HDDs, and on top of that a separate external HDD for Time Machine that backed up the whole thing.<\/p>\n\n\n\n

Since the free space on the internal drive directly affects the Mac\u2019s performance, I\u2019d been moving files around as needed.<\/p>\n\n\n\n

My storage setup at the time was:<\/p>\n\n\n\n

\u30001. An internal Fusion Drive, 3TB<\/p>\n\n\n\n

\u30002. External HDD 1: a 4TB drive I\u2019d named \u201c4TB-01\u201d<\/p>\n\n\n\n

\u30003. External HDD 2: a 4TB drive I\u2019d named \u201c4TB-02\u201d<\/p>\n\n\n\n

\u30004. External HDD 3: a 12TB drive to back up all of 1\u20133<\/p>\n\n\n\n

Drives 2 and 3 were housed together inside a single HDD enclosure.<\/p>\n\n\n\n

Since the enclosure had no RAID function, it amounted to two separate 4TB volumes \u2014 8TB in total.<\/p>\n\n\n\n

Across this storage, files I\u2019d offloaded for reasons like \u201cthe iMac was running out of space, so I moved it,\u201d \u201cI copied it as a backup,\u201d and \u201cI figured it\u2019d be safe to stash here\u201d had quietly piled up.<\/p>\n\n\n\n

An enthusiastic amateur at many things, I had stage photos and performance videos from theater, band recordings, photos I\u2019d been accumulating since around 2004, and more \u2014 over twenty years\u2019 worth of digital debris scattered across multiple drives.<\/p>\n\n\n\n

The worst of it: there were things I\u2019d copied as backups but then, out of laziness, never deleted from the original location. At first I\u2019d felt something like \u201cI\u2019m not sure it copied correctly \u2014 what if the copy failed?\u201d \u2014 and then I\u2019d lose track of which one was the master, leaving huge folders and files existing in multiple separate places.<\/p>\n\n\n\n

There were multiple sets of duplicate data ranging from hundreds of gigabytes to terabytes in size.<\/p>\n\n\n\n

Photos were sorted out fine thanks to iCloud, but video files and the like were another story, and multiple copies had ended up existing.<\/p>\n\n\n\n

If I ran Migration Assistant in this state, I\u2019d unquestionably end up with a \u201ccopy of the mess\u201d on the new Mac. It wouldn\u2019t all fit capacity-wise either, so some kind of cleanup was needed beforehand.<\/p>\n\n\n\n

A Scale No Amount of Manual Work Could Handle<\/h2>\n\n\n\n

At first, I thought I\u2019d manage it by hand.<\/p>\n\n\n\n

To start, I\u2019d take the latest version of the Time Machine contents as the baseline and delete what wasn\u2019t needed.<\/p>\n\n\n\n

The process of judging whether something is unneeded is deeply unproductive, and I just can\u2019t warm to it.<\/p>\n\n\n\n

Whether the latest Time Machine backup truly reflected the most current state, whether all the files were gathered in one place \u2014 just sorting out questions like these took an enormous amount of time.<\/p>\n\n\n\n

Then I set about hunting through multiple HDDs for copies of what was in that backup, and checking whether they were perfectly identical.<\/p>\n\n\n\n

But I could never get real certainty that \u201cthis data truly, genuinely has no differences.\u201d<\/p>\n\n\n\n

On top of that, the macOS file system has all sorts of traps you can\u2019t see from Finder. Right-click a folder and hit \u201cGet Info\u201d to see its size, and the spinner just starts turning and never comes back. Folders synced to iCloud have placeholder files mixed in whose contents haven\u2019t been downloaded, so their sizes can\u2019t be trusted. There\u2019s a concept called \u201cpurgeable\u201d space, where the displayed free space and the actual free space don\u2019t always match. Invisible files like .DS_Store<\/code>, and the OS\u2019s auto-generated search indexes, also get in the way of diff comparisons.<\/p>\n\n\n\n

I also tried dropping into the terminal to handle it with commands. Using rsync<\/code> to compare folders and surface the differences; using the find<\/code> command to extract files matching specific conditions; using shasum<\/code> to compute hashes and judge whether contents were the same \u2014 I tried each one, one at a time, consulting AI along the way.<\/p>\n\n\n\n

It works up to a point. But after rsync<\/code> spits out tens of thousands of lines of logs, a different problem emerges: how do you read those logs? When information like \u201cthis file is newer, but that one over there is older\u201d stretches on for tens of thousands of lines, a human simply can\u2019t process it anymore. You could write a script to parse the logs \u2014 but I didn\u2019t have the knowledge to do that, and trying to pull it off together with AI would have been a major undertaking in itself.<\/p>\n\n\n\n

Even trying to feed the log files to an AI, I\u2019d immediately hit the character limit, and the AI would simply refuse to read and analyze the whole thing \u2014 the obstacles were piled high.<\/p>\n\n\n\n

And even checking the properties of each folder and file one by one, I couldn\u2019t determine whether they were \u201ctruly, genuinely the same file.\u201d<\/p>\n\n\n\n

One comparison method the AI taught me was computing hash values. By reading the entire contents of an individual file and converting it into a fixed-length string (a hash value), you can determine whether two pieces of data are perfectly identical. Because the method disregards information like the file name and creation date and works only on the actual data portion, it can reliably capture whether the underlying data is the same or not.<\/p>\n\n\n\n

For someone like me \u2014 the type who\u2019s scattered the same data across multiple places \u2014 I thought this was the best approach.<\/p>\n\n\n\n

But when there\u2019s too much data, and each individual file is too large (videos especially), computing the hashes could easily take a day or two. This was partly a matter of the Mac\u2019s processor power, and partly the slow transfer speed of the external HDDs (around 100 MB\/s effective over USB 3.0), so things crawled along and only the waiting time piled up.<\/p>\n\n\n\n

And the biggest problem was that no matter how carefully I worked, the fear that \u201cI might accidentally delete a file I need\u201d was always with me. Some of it \u2014 like the theater stage photos \u2014 can never be recaptured. The one thing I had to avoid was a folder I\u2019d been thinking \u201cI\u2019ll sort it out later\u201d turning out to be the only original. Because of this fear, the time my hands sat frozen grew longer than the time I spent actually working.<\/p>\n\n\n\n

The Day I Vented to an AI<\/h2>\n\n\n\n

Then one day, I found myself explaining the situation to Claude. It started as a technical question. Something like \u201crsync is producing logs like this \u2014 how should I read them?\u201d or \u201cwhat\u2019s a good way to compute hashes?\u201d \u2014 about that level, I think.<\/p>\n\n\n\n

But as we talked, it gradually turned into something closer to venting. I\u2019d bought a new Mac for the sake of data migration, yet the migration itself was nowhere near done. Manual work had its limits. None of the existing tools fit my problem exactly. At this rate, I\u2019d never get to my main business of AI development \u2014<\/p>\n\n\n\n

And then it struck me: \u201cCould I actually build a tool to solve a situation like this myself?\u201d<\/p>\n\n\n\n

Of course, there are tools out there that address similar problems, but they inevitably fall short of what I had in mind in some respect.<\/p>\n\n\n\n

Even if a tool could answer \u201ccompare files A and B,\u201d it couldn\u2019t address the separate question of \u201cwhich directories does my data even live in? Is anything being missed?\u201d<\/p>\n\n\n\n

On Mac and Windows alike, when you migrate data, there\u2019s an anxiety around \u201chave I really covered all of it?\u201d<\/p>\n\n\n\n

Your data is made up of system data, libraries, personal-attribute data, the apps themselves, user data, and so on. It would be great if something could guarantee \u201ccopy just this directory and you\u2019re absolutely fine; there\u2019s definitely no necessary data stored anywhere else\u201d \u2014 but individual apps tend to squirrel away important data in their own separate locations, so this becomes a major challenge too.<\/p>\n\n\n\n

You also need a feature that scans for \u201cwhere is my data in the first place?\u201d<\/p>\n\n\n\n

Claude\u2019s answer was, roughly, to this effect. Technically, most of it could be done with Python\u2019s standard library. A three-stage approach seemed realistic: first narrow down candidates by file name and size, then compare quickly using partial hashes (a hash of the first few kilobytes of a file), and only take a full hash of the suspicious ones to confirm. The GUI could be built with something like PySide6. There are commercial utilities, but surprisingly few tools go far enough to handle Mac-specific issues (iCloud, purgeable space, invisible files, and so on). So there did seem to be value in building it myself.<\/p>\n\n\n\n

Even hearing that, whether I could actually build it was another matter. After all, I have almost zero programming experience. As I\u2019ve written in earlier blog posts, I washed out on C back in my student days.<\/p>\n\n\n\n

Even so, I felt myself waver just a little. I had, after all, just finished writing the design documents for the seven AI systems through dialogue with Claude in the very same way. Maybe, using that same method, I could this time build a \u201cworking app\u201d \u2014 or so I thought.<\/p>\n\n\n\n

What I Meant as a \u201cByproduct\u201d Became the Prelude to the Main Business<\/h2>\n\n\n\n

And so the data migration itself went on pause, and development of a tool to assist with data migration began. Thinking about it calmly, this is a pretty absurd story. I\u2019d bought a new Mac in order to get to work on my main business \u2014 the seven AI systems \u2014 and yet here I was, starting development of a different piece of software as preparatory work for moving data onto that Mac.<\/p>\n\n\n\n

That said \u2014 and this may sound like an excuse \u2014 within my own mind it wasn\u2019t such a contradictory decision. There were three reasons.<\/p>\n\n\n\n

The first was a calculation: weighing the time to sort everything out by hand against the time to build a tool, the latter might be the lesser evil. Of course, there was a chance I\u2019d stall partway through building it. Even so, since I couldn\u2019t picture a future of continuing the manual work, I figured it was worth the gamble.<\/p>\n\n\n\n

The second was a guess that the problem troubling me is probably troubling others too. Among people who\u2019ve used Macs for over twenty years and juggle multiple storage devices, I surely can\u2019t be the only one. If it came together, it might serve as Soul Resonant Works\u2019 first product \u2014 something others could actually use. Before putting out AI-related products, gaining the experience of \u201cseeing a smaller product all the way through to the end\u201d once felt like it wouldn\u2019t be a bad thing.<\/p>\n\n\n\n

The third was that it was just the right scale to serve as practice for the development style called vibe coding. The seven AI systems are each complex, beyond what I could handle on my own. But a data migration tool \u2014 if I kept the features focused \u2014 might be manageable for one person (plus AI). The idea was that the experience I\u2019d gain here should pay off for my main business of AI development.<\/p>\n\n\n\n

Of course, all of these reasons are after-the-fact rationalizations \u2014 \u201cI suppose you could put it that way, in hindsight.\u201d The moment I actually decided to build the tool was, I think, far more impulsive. The ratio was something like eighty percent \u201cI just can\u2019t stand continuing this manual work anymore,\u201d and the remaining twenty percent the logic I laid out above.<\/p>\n\n\n\n

Be that as it may, from this impulsive decision, a tool named DataMigrator began to take shape. And as I\u2019ve written in earlier blog posts, this decision was the beginning of \u201cthe main business receding into the distance.\u201d<\/p>\n\n\n\n

Next Time<\/h2>\n\n\n\n

Next time, I\u2019ll bring you the story of how I started writing the requirements definition for DataMigrator. Rather than diving straight into code, I began by defining \u201cwhat to build\u201d in words. I first started writing it using an AI other than Claude, but when I had Claude review it, I was told \u201cthe problem is framed in high resolution, but the solution is rough\u201d \u2014 and as a result, I ended up revising it five times, from Rev.0 through Rev.5. It\u2019s a story about what it means to have an AI write a requirements definition, and what it means to have an AI shoot down your work over and over.<\/p>\n\n\n\n

\n\n\n\n

About Soul Resonant Works<\/h2>\n\n\n\n

Soul Resonant Works is a solo venture developing seven local AI systems. Starting from zero programming experience, the development is progressing through collaboration with AI.<\/p>\n\n\n\n

\ud83c\udf10 Soul Resonant Works: \u2192 https:\/\/www.sr-works.net\/en\/<\/a><\/p>\n\n\n\n

\ud83d\udcdd This blog publishes the entire development process as a serialized journal.<\/p>\n\n\n\n

\n\n\n\n

If you found this article useful, please share it.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

This article was originally written in Japanese and translated into English with AI assistance. Please note that some expressions may carry nuances from the original Japanese. \ud83c\uddef\ud83c\uddf5 \u65e5\u672c\u8a9e\u7248\u306f\u3053\u3061\u3089 \/ Japanese version Series: DataMigrator Development Journal (Part 1) In my previous blog posts, I described how, over the four months from January to April 2026, I […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-41","post","type-post","status-publish","format-standard","hentry","category-ai-development-journal"],"_links":{"self":[{"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/posts\/41","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/comments?post=41"}],"version-history":[{"count":2,"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/posts\/41\/revisions"}],"predecessor-version":[{"id":43,"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/posts\/41\/revisions\/43"}],"wp:attachment":[{"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/media?parent=41"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/categories?post=41"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/tags?post=41"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}