Final Project Update – COMPUTATIONAL FABRICATION

Status

So far, I have printed, cleaned, and glued magnets into all of the physical parts I need, and the browser-based scanner has some good progress. To be more specific, I’ve printed 4 baseplates and 144 tiles (for a grand total of 400 magnets!) and the scanner is able to reliably identify all tiles in an image containing a single baseplate.

I have gathered the code and instructions for printing the parts into this github repo, and the in-progress scanner can be found here.

Physical Prints

All four baseplates and 144 tiles, connected and organized.

The baseplates, separated from each other, with some of the tiles scattered on the side.

Two copies of the physical translation key. Note the two copies of each English letter per Zuish symbol, letting the whole key be rotated to see the as-is vs rotated/human-readable versions.

The first in-game Zuish phrase. Above is how it appears in the game (plus a translation in the top left), below is the same phrase made out of physical tiles.

An in-progress photo of a baseplate. These things are big! 144g of filament with 10% infill!

An in-progress picture of the printer putting down black filament onto the white tiles.

Scanner/Translator

With the prints complete, all that’s left is the scanner program. Currently, images with a single baseplate work fine, though curiously it fails when I have multiple baseplates in the same image. Curiously enough though, it appears to not be a problem of image detail, but of js-aruco (the base library) only searching for markers of a certain size relative to the image size. I have a few ideas on how to resolve this, and it’ll be the next immediate goal to tackle.

Another issue I’m facing is making things work nicely on mobile browsers. If the processing time takes too long, my phone will reload the page, meaning I need to rework the program to allow the workload to be spread out over time. And of course, there’s the problem of adjusting the html for mobile devices, so that everything fits nicely on-screen.

Immediate short-term hurdles aside, there’s still a fair bit more work to go. The program can identify symbols (even handling the red and blue ones) but now I need to make the translator. Specifically, I need to make it gather the symbols into a structured grid, and translate those symbols into English while accounting for spaces. Then, I also need to have translation in the other direction: give the website some English text and config parameters, and have it generate an image in Zuish.

Deliverables

My deliverables and timeline have not changed, so this is a copy from the project proposal.

A set of small 3D-printed tiles to represent the letters of the Zuish alphabet from FEZ
- There should be multiple copies for each letter
- The duplicate symbols in the letter pairs K & Q and U & V should be differentiated by different filament colors
A set of baseplates which have slots arranged in a grid to put letter tiles into
- Baseplates should be able to connect together to create an arbitrarily large grid
- Baseplates should be designed such that the translator script can identify them together as one collective grid
- The number of slots per baseplate and number of baseplates printed should be enough to write at least 1-2 simple, medium-length sentences in one grid.
A javascript program for translating Zuish by interacting with the 3D-printed tiles
- Given an input image of the grid-placed tiles, output the English translation as text
- Given an input string of English text and other parameters such as max column height, output an image representing the Zuish translation
- If feasible, make the program accessible in-browser through a simplistic webpage, for both desktop and mobile devices

Timeline

Week 1:
- A small printed sample of the baseplates and tiles, enough to demonstrate the symbols and connectable baseplates
Week 2:
- Prototype translation program which can output Zuish images from English text, and progress on Zuish images to English text
Week 3:
- Refined baseplate and tile models printed in full quantities, and a completed translation program

Post-Tuesday Edit

This isn’t going with the rest of the post since the update due date was on Tuesday, but as of Wednesday 6PM, I’ve fixed the multi-baseplate and too-slow-for-mobile issues, and Zuish to English translation is (more or less) complete!

It was struggling with images where the markers were small compared to the image because js-aruco normally sets a minimum marker size based on the width of the image. Simply tweaking it to a fixed minimum instead was all it took to allow images with more baseplates to work.

Image processing is now split apart into steps that get spread out over time, so mobile browsers don’t crash the page. I’ve also tweaked the UI elements to make the page fit nicer on mobile screens, though it’s not perfect.

I can now translate images of the physical pieces, so most of all that’s left to do is converting English text into Zuish images.

Here’s a video demonstration:

Note how it’s quite resilient to rotation and camera tilting, though it can struggle with glare (the missing U in the last image), bad lighting (light from my monitor interfered with the red/blue detection in the webcam portion), and low-quality images (as seen with my terrible webcam). It’s not perfect, though clean images taken by my phone work very well as long as glare is minimized.

There is one other little hiccup though… The webcam (or back-cam, rather) feed doesn’t work quite right for mobile devices. It loads one frame, then no more. After searching around, it seems to be a common problem, and not one that’s easily fixable. Unless I can find a good solution later, mobile devices will be limited to taking pictures and uploading them, instead of using a live feed.