{"id":138,"date":"2026-08-31T22:00:00","date_gmt":"2026-08-31T13:00:00","guid":{"rendered":"https:\/\/sr-works.net\/blog-en\/?p=138"},"modified":"2026-08-30T21:26:44","modified_gmt":"2026-08-30T12:26:44","slug":"the-day-utf8conv-was-born-vs-code-wasnt-lying-and-the-mojibake-wouldnt-go-away","status":"publish","type":"post","link":"https:\/\/sr-works.net\/blog-en\/the-day-utf8conv-was-born-vs-code-wasnt-lying-and-the-mojibake-wouldnt-go-away\/","title":{"rendered":"The Day utf8conv Was Born \u2014 VS Code Wasn’t Lying, and the Mojibake Wouldn’t Go Away"},"content":{"rendered":"\n
\n

This article was originally written in Japanese and translated into English with AI assistance. Please note that some expressions may carry nuances from the original Japanese.<\/em><\/p>\n<\/blockquote>\n\n\n\n

\ud83c\uddef\ud83c\uddf5 \u65e5\u672c\u8a9e\u7248\u306f\u3053\u3061\u3089 \/ Japanese version<\/a><\/em><\/p>\n\n\n\n

\n\n\n\n

Series: utf8conv Development Journal (Part 1)<\/p>\n\n\n\n

\n\n\n\n

Starting with this installment, I’m beginning the development journal for another product that grew up alongside DataMigrator: “utf8conv” (UC for short).<\/p>\n\n\n\n

utf8conv is a very simple GUI tool that does nothing more than batch-converting files saved in a non-UTF-8 encoding into UTF-8. The name is exactly what it sounds like: “utf-8 converter.” Unlike DataMigrator, there’s no lengthy design document behind it, and in scale the two aren’t remotely comparable \u2014 utf8conv is much smaller.<\/p>\n\n\n\n

Still, being small didn’t mean it was easy. Right from day one I ran into a compatibility problem between uv and tkinter that ate half a day, and partway through I struggled under a “zero additional packages” constraint I’d set for myself \u2014 there were a fair number of stumbles along the way.
This first installment starts with the story of why I decided to build this tool in the first place.<\/p>\n\n\n\n

While Talking with Claude, Mojibake Was Quietly Happening<\/h2>\n\n\n\n

Around February 2026, alongside developing DataMigrator, I was also doing other work while chatting with Claude \u2014 tweaking HTML files here and there, tidying up old documents, that kind of miscellaneous task.<\/p>\n\n\n\n

One day, in the middle of asking Claude for advice on something, I found a garbled file on my PC. Opening it, the Japanese parts had turned into something like “\u8b41?\uff6d\u601c\u55a7\u7e3a” \u2014 a string of symbols that unmistakably screamed “the character encoding is broken.” Having used a Mac for 20 years, this kind of mojibake is a familiar sight I’d run into more than a few times.<\/p>\n\n\n\n

Normally I’d have shrugged it off with an “ugh, again” and dealt with it however. But this time, since I happened to be in the middle of migrating data from my old iMac to a MacBook Pro, and had Claude on hand to ask, I tried “How do I fix this file?” Claude’s answer was straight out of the textbook: “The original file is probably saved in a different encoding, like Shift-JIS. Please re-save it as UTF-8 in VS Code.”<\/p>\n\n\n\n

So I did as told: opened it in VS Code, chose “Save with Encoding” \u2192 UTF-8, saved it, and reopened it in the browser.<\/p>\n\n\n\n

\u2014It didn’t fix it.<\/p>\n\n\n\n

On screen, the mojibake was still dancing around, unchanged. VS Code’s own status bar at the bottom said “UTF-8,” so the file was surely UTF-8 now. When I told Claude, “I saved it but it’s still not fixed,” Claude thought for a moment and suggested, “Try explicitly specifying the encoding again when you reopen and save it.” I tried that too. Still no fix.<\/p>\n\n\n\n

At this point, doubts were starting to creep in: “Is this a VS Code bug?” “Is the AI telling me something wrong?” Doubting one or the other is one thing, but once you start doubting both at once, you lose your footing entirely.<\/p>\n\n\n\n

So, What Is a Character Encoding, Anyway?<\/h2>\n\n\n\n

Let me back up for a moment and lay out some groundwork.<\/p>\n\n\n\n

Inside a computer, every character is stored as numbers \u2014 a sequence of bytes. The rule for how to read that byte sequence and map it to characters is what we call a “character encoding.” There isn’t just one such rule; many have been created over the years, for different languages and eras \u2014 Shift-JIS or EUC-JP for Japanese, GBK for Chinese, CP1252 for Western European languages, and today’s standard, UTF-8.<\/p>\n\n\n\n

The same character will produce a different byte sequence depending on which rule you use. Put the other way around: read the same byte sequence under a different rule, and you get something completely different. That’s mojibake.<\/p>\n\n\n\n

Let’s actually try it. Take a string saved under one language’s character encoding, then read it as if it were a different language’s encoding, and here’s what you get:<\/p>\n\n\n\n

Below, a Japanese file is read as Chinese (GBK), a Chinese file as Japanese (EUC-JP), and a German file as Cyrillic (CP1251).<\/p>\n\n\n\n

Japanese \u3053\u3093\u306b\u3061\u306f \u6587\u5b57\u5316\u3051     \u2192  \u5059\u50ab\u5075\u506a\u507c \u66a5\u5e24\u58d4\u5057\nChinese  \u4f60\u597d \u4e16\u754c \u7f16\u7801          \u2192  \u4f4e\u632b \u5f0a\u9806 \u5712\u9df9\nGerman   Gr\u00fc\u00dfe Stra\u00dfe Fu\u00dfball  \u2192  Gr\u044c\u042fe Stra\u042fe Fu\u042fball<\/code><\/pre>\n\n\n\n
Look at the German example. Gr\u00fc\u00dfe<\/code> has become Gr\u044c\u042fe<\/code>. The alphabetic letters survive intact; only the characters carrying an umlaut or an eszett break, turning into Cyrillic letters. Unlike the Japanese case, where the whole string turns unreadable, here only part of it corrupts. Exactly which parts the two rules share, and which parts differ, shows up directly in the result.<\/p>\n\n\n\n
And the mojibake that opened this article runs in the opposite direction. Take something saved in UTF-8 and read it as if it were Shift-JIS, and you get this:<\/p>\n\n\n\n
Japanese \"mojibake\" (\u6587\u5b57\u5316\u3051)  \u2192  \u8b41\u87c4\u601c\u55a7\u7e3a<\/code><\/pre>\n\n\n\n
In both directions, the file itself isn’t broken. The byte sequence is intact, safely saved exactly as it was. What’s broken is the “how to read it.”<\/p>\n\n\n\n
The Cause: A Mismatch Between the Byte Sequence and the Declaration<\/h2>\n\n\n\n
Digging into the cause together with Claude, here’s what it eventually turned out to be.<\/p>\n\n\n\n
The problem file was an HTML file. HTML files carry a declaration line near the top of the file that says, in effect, “this file is written in this encoding” \u2014 specifically, something like <meta charset=\"Shift_JIS\"><\/code>. The browser sees this declaration and decides, “ah, then I’ll read it as Shift-JIS.”<\/p>\n\n\n\n
When you choose “Save with Encoding: UTF-8” in VS Code, the file’s contents \u2014 the actual byte sequence \u2014 really are converted to UTF-8. VS Code isn’t lying about that part. But the line that says <meta charset=\"Shift_JIS\"><\/code> is left exactly as it was.<\/p>\n\n\n\n
That’s because an editor’s “save with a different encoding” feature converts the file’s byte sequence \u2014 it doesn’t read the document’s contents and rewrite them. So the string <meta charset=\"Shift_JIS\"><\/code> just gets stored, unchanged, inside a UTF-8-encoded file.
And I don’t think that’s careless of the editor \u2014 I think it’s the correct behavior. An editor that took it upon itself to rewrite the body text because “this declaration doesn’t seem to match reality, so I’ll fix it for you” would be far more alarming.<\/p>\n\n\n\n
Still, the upshot is that the file is left holding nothing but a contradiction.<\/p>\n\n\n\n
The result is a mismatch like this: the file’s actual contents are UTF-8. But the top of the file declares “this is Shift-JIS.” The browser trusts the declaration first. Believing it’s Shift-JIS, it tries to interpret the UTF-8 byte sequence as Shift-JIS. Naturally, the result isn’t meaningful Japanese. That result is the mojibake from the start of this article.<\/p>\n\n\n\n
The moment I heard this explanation, two thoughts rose up in me at once: “I see,” and “wait, couldn’t this be turned into a tool?” The problem I’d run into was probably one that plenty of VS Code users out there, somewhere, had also run into \u2014 and given that it even included the situation of “an AI kindly walking you through the wrong fix,” it seemed like something with real reproducibility. In particular, since I was already building DataMigrator, a product for organizing and migrating data files, with the underlying premise of getting file migration between my own Macs properly sorted out, I was going to need to correctly interpret the encodings of a large, disorganized pile of files anyway. That’s also where requirements like being able to do a dry run in advance, actually changing the encoding properly, handling a large number of files at once, and having a safety net to revert if needed, all found their way into the spec.<\/p>\n\n\n\n
To begin with, a tool that just batch-converts files from non-UTF-8 to UTF-8 is the kind of thing that could reasonably have existed for a long time already. But one that’s a GUI, needs no extra installation, can safely do a dry run (check only), and correctly rewrites HTML\/XML declarations too \u2014 all of that together \u2014 I couldn’t find, within the range I searched.<\/p>\n\n\n\n
By the way, this problem \u2014 mojibake caused by a mismatch between the declaration and the actual encoding \u2014 wasn’t limited to browsers. The same thing happens in the Quick Look preview you get on macOS Finder by selecting a file and pressing the space bar. I confirmed this later by making four test files.<\/p>\n\n\n\n
A file whose contents are UTF-8 but whose declaration says “Shift-JIS” \u2014 garbled, even in Quick Look. A file where both the contents and the declaration are UTF-8 \u2014 readable. One where both are Shift-JIS \u2014 also readable. And a file with the declaration removed entirely \u2014 readable, correctly.<\/p>\n\n\n\n
In other words, Quick Look can actually work out the right answer just by looking at the contents. Yet even though it can work it out, when a declaration is present, it trusts the declaration instead and breaks. It displays correctly only when there’s no declaration at all \u2014 a slightly perverse result.<\/p>\n\n\n\n
From here on, this is less about character encoding itself and more about how the software is built. Software that handles character encodings is, for the most part, built to “auto-detect by default, but let you choose manually if needed.” Browsers work this way too. But the lightweight preview feature built into the OS \u2014 the one that doesn’t even count as launching an app, that opens instantly when you select a file \u2014 doesn’t seem to have that manual entry point anywhere. Maybe it exists somewhere if I looked hard enough. But the act of hunting for it is a hassle in itself, and even if I found the setting and changed it, some other encoding would stop displaying correctly instead \u2014 which would defeat the purpose.<\/p>\n\n\n\n
Rather than fiddling with the display side to fix it, fix the file itself. That, in the end, is what utf8conv is trying to do.<\/p>\n\n\n\n
UC’s Direction<\/h2>\n\n\n\n
And so, right as DataMigrator’s M1 wrapped up, another small project \u2014 “utf8conv” \u2014 got underway.<\/p>\n\n\n\n
I wrote a requirements document for it too, but it never grew past 30-plus chapters the way DataMigrator’s did. It’s a modest requirements document, running to just a few chapters at most. Even so, I settled a few important policies right at the start.<\/p>\n\n\n\n
The first was “zero additional packages” \u2014 a constraint that everything had to be done using nothing but Python’s standard library. I initially considered using a well-known library called chardet<\/code> for encoding detection, but I judged that skipping chardet would buy me benefits like “it runs the moment it’s installed,” “no dependency headaches,” and “distribution stays simple.” Detection accuracy might suffer a little as a result, but I figured I’d make up for that with the “try encodings in order from the top” approach described later.<\/p>\n\n\n\n
The second was “dry run first.” By default, the tool doesn’t convert anything \u2014 it just runs in a mode that logs what would happen if you converted, and only performs an actual conversion once the user explicitly chooses “really convert.” Just like DataMigrator’s “never delete” philosophy, this leans toward the safe side. Since undoing a character-encoding conversion once it’s done is a hassle, I judged that a confirm-first mechanism was indispensable.<\/p>\n\n\n\n
The third was “automatic backups.” When an actual conversion runs, the pre-conversion file is automatically moved into a backup folder. Even if a user chooses to skip the dry run and go straight to a real conversion, having a backup means they can still roll back \u2014 a last line of defense.<\/p>\n\n\n\n
Written out, these three policies come to only three lines, but they ended up defining utf8conv’s character down the line.<\/p>\n\n\n\n
On Day One, uv and tkinter Get Into a Fight<\/h2>\n\n\n\n
Once the requirements document was done, I started actually moving my hands in Claude Code. Same as with DataMigrator: create the project’s box, spin up a Python environment, get a minimal GUI showing \u2014 that route. For the GUI, in keeping with the zero-additional-packages policy, I decided to use tkinter from Python’s standard library.<\/p>\n\n\n\n
But when I tried to run it, the window wouldn’t open. The error wasn’t particularly helpful either \u2014 all I got back was a message to the effect of “Tcl\/Tk not found.”<\/p>\n\n\n\n
Having Claude Code dig into it, the cause became clear. uv (the Python environment manager I’d adopted during DataMigrator’s M1) ships its own Python binaries, and those binaries don’t bundle Tcl\/Tk \u2014 the GUI library tkinter relies on internally. The Python that uv distributes is, strictly speaking, “just the Python interpreter” \u2014 it doesn’t include GUI-related extra runtimes. That’s a reasonable design choice on uv’s part, but it’s a trap if you’re trying to use tkinter.<\/p>\n\n\n\n
The fix was to install Python and python-tk via Homebrew, then rebuild uv’s virtual environment on top of that Homebrew Python. Written as commands, the flow looks roughly like this:<\/p>\n\n\n\n
brew install python@3.12 && brew install python-tk@3.12\nuv venv --python \/opt\/homebrew\/bin\/python3.12<\/code><\/pre>\n\n\n\n
Written down, it’s only a couple of lines, but it took about half a day of trial and error to pin down the cause. Neither uv nor tkinter is at fault \u2014 it’s the specific combination that’s unusual, the kind of problem that’s genuinely hard to reach on a first encounter.<\/p>\n\n\n\n
Still, this experience paid off later. From then on, every Python project with a GUI \u2014 both utf8conv and DataMigrator \u2014 got standardized on this “uv virtual environment built on Homebrew Python” setup, and I never ran into the same problem again. One more piece of experience banked, so I wouldn’t hit the same wall twice.<\/p>\n\n\n\n
The End of Day One<\/h2>\n\n\n\n
By the end of day one, a directory structure suited to software development was in place, and the source file utf8_converter_gui.py<\/code> was up and running.<\/p>\n\n\n\n
Launching utf8_converter_gui.py<\/code> opened a tkinter window. The conversion logic itself was already written by this point. I could already select which file extensions to target for conversion, and there was already an “add custom extension” input field where I could add my own. On top of that, the GitHub Actions test automation was running, and the first push to the private repository was done.<\/p>\n\n\n\n
That said, the current source code has the conversion logic and the GUI mixed together in a single file. In this shape, there’s no way to pull out just the conversion logic and verify it on its own. In fact, the only test I managed to write that day was a single one checking whether the version number could be read correctly. Splitting these apart so each piece could be tested individually \u2014 that became the next day’s homework.<\/p>\n\n\n\n
Oddly enough, I didn’t get the same heavy sense of “taking the first big step into a large development effort” that I’d felt when M1 wrapped up on DataMigrator. utf8conv’s first day felt lighter than that \u2014 more like an oddly optimistic “maybe this’ll be done in three days.” Maybe it’s because the smaller scope let me picture the finished product right away.<\/p>\n\n\n\n
And in fact, utf8conv’s development would go on to produce a working MVP within a genuine handful of days after this. Three days including a weekend, six chat sessions (U1 through U6), and all the features were implemented \u2014 that was the pace. Along the way, I went through a few design decisions that pushed a bit further than typical vibe coding, like “separate the core from the GUI,” “put the latin-1 fallback last,” and “write tests as data instead of code.” I’ll get into those starting next time.<\/p>\n\n\n\n
Coming Up Next<\/h2>\n\n\n\n
Next time is the story of utf8conv’s second day (session U2). As of day one, the conversion logic and the GUI were still mixed together in one file. Since that makes it impossible to write tests (you can’t check the logic without launching the GUI), I end up separating the core (logic) from the GUI (screen). Changing a design for the sake of “so we can write tests” was, for me, a first. Along the way, I also get into an idea called the “latin-1 fallback,” for implementing character-encoding detection using nothing but Python’s standard library.<\/p>\n\n\n\n
\n\n\n\n
About Soul Resonant Works<\/h2>\n\n\n\n
Soul Resonant Works is a solo venture developing seven local AI systems.
Starting from zero programming experience, the development is progressing through collaboration with AI.<\/p>\n\n\n\n
\ud83c\udf10 Soul Resonant Works:
\u2192 https:\/\/sr-works.net\/en\/index.html<\/a><\/p>\n\n\n\n
\ud83d\udcdd This blog publishes the entire development process as a serialized journal.<\/p>\n\n\n\n
\n\n\n\n
CubePlot (free version available)<\/h2>\n\n\n\n
CubePlot is the first product from Soul Resonant Works \u2014 it turns a CSV into a 3D scatter plot you can rotate and zoom. No install, no sign-up: it’s a single HTML file you open in your browser, and it works offline. CubePlot itself does not send the data you load outside your machine \u2014 external network traffic is blocked at the browser level (CSP). Start with the free version.<\/p>\n\n\n\n
\u25b6 Product page: https:\/\/sr-works.net\/en\/cubeplot\/<\/a>
\u25b6 Get it (commercial license, USD $39 + tax where applicable): https:\/\/soulworks8.gumroad.com\/l\/pzcij<\/a><\/p>\n\n\n\n
\n\n\n\n
If you found this article useful, please share it.<\/p>\n","protected":false},"excerpt":{"rendered":"
This article was originally written in Japanese and translated into English with AI assistance. Please note that some expressions may carry nuances from the original Japanese. \ud83c\uddef\ud83c\uddf5 \u65e5\u672c\u8a9e\u7248\u306f\u3053\u3061\u3089 \/ Japanese version Series: utf8conv Development Journal (Part 1) Starting with this installment, I’m beginning the development journal for another product that grew up alongside DataMigrator: “utf8conv” […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-138","post","type-post","status-publish","format-standard","hentry","category-ai-development-journal"],"_links":{"self":[{"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/posts\/138","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/comments?post=138"}],"version-history":[{"count":3,"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/posts\/138\/revisions"}],"predecessor-version":[{"id":140,"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/posts\/138\/revisions\/140"}],"wp:attachment":[{"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/media?parent=138"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/categories?post=138"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/tags?post=138"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}