{"id":166,"date":"2026-10-17T10:23:57","date_gmt":"2026-10-17T01:23:57","guid":{"rendered":"https:\/\/sr-works.net\/blog-en\/?p=166"},"modified":"2026-10-17T10:23:57","modified_gmt":"2026-10-17T01:23:57","slug":"implementing-the-declaration-rewrite-the-day-i-wrote-tests-as-data-not-code","status":"publish","type":"post","link":"https:\/\/sr-works.net\/blog-en\/implementing-the-declaration-rewrite-the-day-i-wrote-tests-as-data-not-code\/","title":{"rendered":"Implementing the Declaration Rewrite \u2014 The Day I Wrote Tests as Data, Not Code"},"content":{"rendered":"\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p class=\"wp-block-paragraph\"><em>This article was originally written in Japanese and translated into English with AI assistance. Please note that some expressions may carry nuances from the original Japanese.<\/em><\/p>\n<\/blockquote>\n\n\n\n<p class=\"wp-block-paragraph\"><em><a href=\"https:\/\/note.com\/sr_works\/n\/n2aa2565754ce\">\ud83c\uddef\ud83c\uddf5 \u65e5\u672c\u8a9e\u7248\u306f\u3053\u3061\u3089 \/ Japanese version<\/a><\/em><\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Series:<\/strong> utf8conv Development Journal (Part 6)<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p class=\"wp-block-paragraph\">Last time was a technical explanation: what converting a character encoding is actually doing, and why, in a file like HTML that declares its own character encoding by itself, &#8220;I saved as UTF-8 but it&#8217;s still broken&#8221; happens. This time I return to the timeline.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This is the story of March 9, the day after I rewrote the requirements document as Rev.2. I did two things that day. I actually built the feature I wrote about at the end of last time, the one that rewrites the declaration inside a file. And I changed the tests so that they are written as data rather than as code. After that, I will go on as far as the small additions of the following day, March 10.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Building the Declaration Rewrite<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">On this day, I added one function to the conversion internals (<code>core.py<\/code>) that rewrites declarations. In the flow of a conversion, after the byte sequence has been rewritten as UTF-8, this function is called automatically.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">What it rewrites had been written as a table in the requirements document Rev.2. What I built that day follows that table exactly.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>HTML <code>&lt;meta charset&gt;<\/code> \u2192 rewrite<\/li>\n\n\n\n<li>The charset in HTML <code>&lt;meta http-equiv=\"Content-Type\"&gt;<\/code> \u2192 rewrite<\/li>\n\n\n\n<li>XML <code>&lt;?xml encoding?&gt;<\/code> \u2192 rewrite<\/li>\n\n\n\n<li>CSS <code>@charset<\/code> \u2192 rewrite<\/li>\n\n\n\n<li>Python <code># -*- coding: -*-<\/code> \u2192 do not rewrite<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">The first four are all self-declarations saying &#8220;this file is written in this character encoding.&#8221; After the body has been rebuilt as UTF-8, if only this declaration is left behind pointing at the name of the original character encoding, the reader believes the declaration and the text comes out garbled. So they are rewritten. The last one, the declaration written at the top of Python source code, plays the same role but is not rewritten. What this line means is as I wrote in last time&#8217;s technical explanation.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The implementation takes the form of regular expressions (a notation for writing out a pattern of characters and finding the places that match it). It searches the body of the file for the declaration line by pattern, rewrites something like <code>Shift_JIS<\/code> to <code>UTF-8<\/code>, and then saves. That day&#8217;s journal records that differences between uppercase and lowercase could be absorbed on the pattern side, and that because the attributes in an HTML <code>&lt;meta&gt;<\/code> tag vary in order, the pattern was designed carefully. I wrote 8 tests for this function.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Writing Tests as Data Instead of Code<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">In parallel with the declaration rewrite, the other big piece of work that day was a change to how the tests are built.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Until then, each test had been written as a Python function, one by one. Five tests for detection, four tests for collecting the target files, and so on: one function for each thing to be checked. With this approach, if you want more tests, you have to write that much more code. As I wrote two installments ago, the test files (fixtures) had been prepared by auto-generation, 518 of them. Writing these out one at a time as test functions is not realistic.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">What I used instead was a mechanism in pytest (the tool that runs the tests) called &#8220;parametrized tests.&#8221; Roughly speaking, it is &#8220;hand it a list of data, and it repeats the same test as many times as there are items in the list.&#8221;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In utf8conv, I set it up so that pairs \u2014 a file before conversion, and the answer file saying what it should become after conversion (<code>.expected<\/code>) \u2014 are collected automatically from the <code>tests\/fixtures\/<\/code> folder and fed into this mechanism. The test itself is a single function. It checks just one thing: &#8220;does the result of converting a given before-conversion file match the contents of the answer file paired with it?&#8221; Into this function flow, one after another, the 252 pairs out of the 518 files that have an answer paired with them.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The result: of the 252 pairs, 210 passed and 42 were skipped. Besides these, there are 2 tests confirming that files already in UTF-8 pass straight through without being converted, and 3 tests confirming that broken files are rejected. The tests up to the day before had numbered 25, so the number of tests jumped in a single day. But what grew was &#8220;data,&#8221; not &#8220;code.&#8221; There is one test function, and every time you place one more pair of answer files, the tests grow by one.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This experience took on a slightly philosophical meaning for me. I had assumed that &#8220;the only way to add tests is to write code,&#8221; when in fact there was a world where &#8220;tests grow just by placing data.&#8221; To put it in vibe coding terms, it may be close to the idea of &#8220;expressing the specification in the form of data.&#8221; Instead of &#8220;writing the rules in words,&#8221; you &#8220;put down many examples,&#8221; and that is how you convey what you expect.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The 42 Skipped Pairs, and the Characters I Put Back In<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The reason 42 pairs were skipped is as I wrote in last time&#8217;s technical explanation. Some of the files made in Western and Eastern European character encodings had contents only in the ASCII range, so they could not be distinguished from UTF-8, and were treated as &#8220;detected as UTF-8, no conversion needed.&#8221; That is not a weakness in detection but correct behavior, so in the README I wrote it not under Limitations but under How It Works. That much is last time&#8217;s story.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">On this day, there is a sequel to it. Into the latin-1 fixtures, I deliberately put non-ASCII characters back in (characters outside letters and digits, such as <code>\u00e9<\/code>). With contents that are ASCII only, it does not amount to a test that confirms &#8220;a file saved in latin-1 gets converted to UTF-8.&#8221;<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">The Next Day, March 10 \u2014 A Small Addition, and One That Was Already In<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The next day, March 10, I ran another round of work and cleared up three small features. \u300c\u8ffd\u52a0\u6307\u5b9a\u300d (add custom extension), which lets you add target extensions yourself; a feature that saves the conversion results log to a file; and a feature that converts just one file rather than a folder. All of them had been listed as &#8220;not yet started.&#8221;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But as soon as I started, <strong>it turned out that \u300c\u8ffd\u52a0\u6307\u5b9a\u300d (add custom extension) was already in.<\/strong> The GUI I ran on day one had had an input field for \u300c\u8ffd\u52a0\u6307\u5b9a\u300d (add custom extension) from the very start. It is the field I wrote about in the first installment of this series, the one on day one&#8217;s screen. The requirements document listed it among the &#8220;things to build next,&#8221; and the real thing had long since existed. This was the only item written under &#8220;where I stumbled&#8221; in that day&#8217;s journal. I judged that no tests needed to be added.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In the end, what I newly built that day came to two things. One is saving the log: press the \u300c\u30ed\u30b0\u3092\u4fdd\u5b58\u2026\u300d (Save Log\u2026) button and the conversion results shown on the screen can be written out to a text file. The other is converting a single file: choose one file from the \u300c\u30d5\u30a1\u30a4\u30eb\u2026\u300d (File\u2026) button, and it converts just that file without specifying a folder. As with a folder, a backup is made and the declaration rewrite runs too. I added 5 tests for these two, and reorganized the usage section of the README into four parts: folder, file, \u300c\u8ffd\u52a0\u6307\u5b9a\u300d (add custom extension), and saving the log.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">How Many Days It Took, and a Look Back<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The work on utf8conv reached the declaration rewrite I have been describing in three days: March 7, 8, and 9. Including the additions on the 10th, four days. As reasons it progressed in this many days, I will list three things the records allow me to say.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">One is that the scale was small. There is one thing to do: bring the character encodings of files into line as UTF-8. That is all.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The second is the constraint of &#8220;zero additional packages.&#8221; The policy of limiting the parts used to those that come with Python from the start was set from the very beginning of development, and is written into the requirements document as a policy. The screen is tkinter only, and the detection of character encodings is my own, without using outside parts that detect statistically. There were few things to choose from in the first place.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The third is that I was able to rewrite the requirements document while building. As I wrote two installments ago, the requirements document was rewritten from Rev.1, written on day one, to Rev.2 the next day, March 8. The declaration rewrite I built on March 9 is an item added at that time.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Implementation Ends Here, for Now<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">With this, the implementation of utf8conv is complete. The Mac app form (<code>.app<\/code>), as I wrote two installments ago, has already been exported. What remains is not writing code but the procedures for &#8220;putting it out into the world.&#8221; The signing and notarization with an Apple Developer ID written under &#8220;Future Outlook&#8221; in the requirements document. And then deciding where to distribute it.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Looking back now that it is complete, the development of utf8conv was, for me, &#8220;a small, fast, closed experience.&#8221; A small product that slipped in, like a breath between strokes, in the middle of a large project like DataMigrator. But what I gained was more than I expected. In particular, the vibe coding sense of &#8220;reusing a pipeline you have once pushed through on the next product&#8221; is something I felt for real for the first time with utf8conv. And this sense is sure to pay off, from here on, when I build other products as well.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Whether utf8conv will actually be used by someone and prove useful is something I will not know until I release it. I started from the hypothesis that the problem I was struggling with must be one that someone somewhere in the world is struggling with too, but whether that hypothesis is right is something I am at the stage of confirming from here. &#8220;Building it&#8221; and &#8220;selling it&#8221; are different stories, and this blog covers only as far as the former. About the latter, honestly, anxiety is slightly ahead of confidence at the moment. But, with that included, I want to go ahead with preparing to put it out into the world as one of SRW&#8217;s products.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The &#8220;building it&#8221; story of the utf8conv series ends here. Next time, I will talk about the work that still remains from here on.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">About Soul Resonant Works<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Soul Resonant Works is a solo venture developing seven local AI systems.<br>Starting from zero programming experience, the development is progressing through collaboration with AI.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\ud83c\udf10 Soul Resonant Works:<br>\u2192 <a href=\"https:\/\/sr-works.net\/en\/index.html\">https:\/\/sr-works.net\/en\/index.html<\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\ud83d\udcdd This blog publishes the entire development process as a serialized journal.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">CubePlot (free version available)<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">CubePlot is the first product from Soul Resonant Works \u2014 it turns a CSV into a 3D scatter plot you can rotate and zoom. No install, no sign-up: it&#8217;s a single HTML file you open in your browser, and it works offline. CubePlot itself does not send the data you load outside your machine \u2014 external network traffic is blocked at the browser level (CSP). Start with the free version.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u25b6 Product page: <a href=\"https:\/\/sr-works.net\/en\/cubeplot\/\">https:\/\/sr-works.net\/en\/cubeplot\/<\/a><br>\u25b6 Get it (commercial license, USD $39 + tax where applicable): <a href=\"https:\/\/soulworks8.gumroad.com\/l\/pzcij\">https:\/\/soulworks8.gumroad.com\/l\/pzcij<\/a><\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p class=\"wp-block-paragraph\"><em>If you found this article useful, please share it.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>This article was originally written in Japanese and translated into English with AI assistance. Please note that some expressions may carry nuances from the original Japanese. \ud83c\uddef\ud83c\uddf5 \u65e5\u672c\u8a9e\u7248\u306f\u3053\u3061\u3089 \/ Japanese version Series: utf8conv Development Journal (Part 6) Last time was a technical explanation: what converting a character encoding is actually doing, and why, in a [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-166","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/posts\/166","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/comments?post=166"}],"version-history":[{"count":2,"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/posts\/166\/revisions"}],"predecessor-version":[{"id":169,"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/posts\/166\/revisions\/169"}],"wp:attachment":[{"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/media?parent=166"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/categories?post=166"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/sr-works.net\/blog-en\/wp-json\/wp\/v2\/tags?post=166"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}