B.S., Boğaziçi University (2018)
S.M., Massachusetts Institute of Technology (2020) Department of Electrical
Engineering and Computer Science

Doctor of Philosophy

February 2024 January 26, 2024

Stefanie MuellerAssociate Professor of Electrical Engineering and Computer
Science

Leslie A. KolodziejskiProfessor of Electrical Engineering and Computer Science
Chair, Department Committee on Graduate Students

# Ubiquitous Metadata: Design and Fabrication of
Embedded Markers for Real-World Object
Identification and Interaction

Mustafa Doğa Doğan

The convergence of the physical and digital realms has ushered in a new era of
immersive experiences and seamless interactions. As the boundaries between the
real world and virtual environments blur and result in a "mixed reality,"
there arises a need for robust and efficient methods to connect physical
objects with their virtual counterparts. In this thesis, we present a novel
approach to bridging this gap through the design, fabrication, and detection
of embedded machine-readable markers.

The vision of mixed reality relies on mobile and wearable devices being aware
of the surroundings to enhance real-world experiences with contextual
information. For individual object identification, machine-readable tags such
as barcodes and RFID labels are typically used. Barcodes, though cost-
effective, tend to be obtrusive, less durable, and less secure than RFID
labels. Regardless of their type, such tags are usually added to objects after
fabrication, rather than being integrated into the original design.

This thesis attempts to overcome the shortcomings of traditional post-hoc
augmentation processes by proposing novel tagging approaches that extract
hidden, integrated features of objects and employ them as machine-detectable
markers. Our research focuses on the design, implementation, and evaluation of
comprehensive systems for embedding and interacting with embedded markers. We
categorize the proposed tagging approaches into three distinct categories:
natural markers, structural markers, and internal markers. Natural markers,
such as those used in SensiCut, are inherent fingerprints of objects
repurposed as machine-readable identifiers, while structural markers, such as
StructCode and G-ID, leverage the structural artifacts in objects that emerge
during the fabrication process itself. Internal markers, such as InfraredTag
and BrightMarker, are embedded inside fabricated objects using specialized
materials. Leveraging a combination of methods from computer vision, machine
learning, computational imaging, and material science, the presented
approaches offer robust and versatile solutions for object identification,
tracking, and interaction.

These markers, seamlessly integrated into real-world objects, effectively
communicate an object’s identity, origin, function, and interaction,
functioning as gateways to "ubiquitous metadata" – a concept where metadata is
embedded into physical objects, similar to metadata in digital files. Across
the different chapters, we demonstrate the applications of the presented
methods in diverse domains, including product design, manufacturing, retail,
logistics, education, entertainment, security, and sustainability. Finally, we
discuss the challenges and opportunities associated with deploying embedded
machine-readable markers at scale, such as integration with mass
manufacturing, privacy considerations, consumer product applications, and
novel interactions for AR personalization.

In conclusion, this thesis presents a comprehensive exploration of embedded
machine-readable markers as a means to connect real-world objects to virtual
worlds to achieve ubiquitous metadata. The thesis forges ahead with a future
where objects come alive, environments become interactive, and virtual worlds
seamlessly merge with our everyday lives.

### Acknowledgments

Reflecting upon the incredible odyssey of doing a PhD at MIT, I am compelled
to acknowledge the exceptional individuals whose support, guidance, and
expertise have shaped my growth and enriched my scholarly pursuits.

First and foremost, I am forever indebted to my thesis advisor, Stefanie
Mueller. Her mentorship, scholarly brilliance, and firm belief in my potential
have been pivotal in my intellectual and professional development. Under her
guidance, I have had the privilege to be part of the distinguished Human-
Computer Interaction (HCI) Engineering group, likely the most international
research team at CSAIL. I am incredibly honored that I got to hang out with
the coolest and most interesting individuals with so many different interests,
both in research and life. Thank you, Cedric, Dishita, Faraz, Junyi, Martin,
Marwa, Ticha, and Yunyi, for being constant sources of inspiration and
encouragement, and not hesitating to offer me last-minute feedback whenever I
needed it. A special note of gratitude goes to Ticha, who has gone above and
beyond to support us, whether it is through offering us fancy teas and snacks,
or hosting cozy meetups with live music. After almost 6 years, it is
astounding to think that time flew so quickly; looking back, I feel privileged
to have worked with brilliant postdocs who have gone on to be at top research
institutions around the globe – Junichi, Michael, Moji, Mackenzie, Yoonji –
thank you for your mentorship at MIT and beyond. And Linda, as the one-and-
only administrative assistant of our lab, the biggest thanks go to you, for
coming to our rescue whenever we struggled.

I would like to express my earnest gratitude to my academic advisor Daniela
Rus, and my thesis advisors and mentors Arvind Satyanarayan and Michael
Nebeling. Their guidance and mentorship have been crucial in shaping my
research journey, and their insights and feedback have deeply enriched my
understanding of the broader scientific landscape.

One of the most fun and rewarding experiences during my time at MIT was having
the opportunity to work with numerous undergraduate researchers, UROPs, as
well as Master’s students, which made me a better mentor. Boundless gratitude
to and admiration for Ahmad Taka, Steven Acevedo, and Veerapatr "Vic"
Yotamornsunthorn, who won Best UROP and Master’s Thesis awards, and made me
and Stefanie so proud.

Steven, you are greatly missed, you will be remembered forever. May you rest
in peace. To the reader of this thesis: Please consider supporting research
for treatments and a cure for DSRCT, an aggressive and rare type of cancer
that took Steven from us, by donating to the MD Anderson Cancer Center through
the Cory Monzingo Foundation.

I extend my heartfelt appreciation to my other esteemed colleagues and friends
at MIT and in Boston, whose camaraderie, collaborative spirit, and
intellectual exchange have been a constant source of strength during
challenging times. I am indebted to Cathy, Dylan, Jack, Kru, Lucio, Miranda,
Raul, Reza, Safa, Tuna, and Ulya for making MIT and Boston feel like a second
home, even during the freezing winters.

I remain genuinely thankful for my friends Ecem and Ege, who have been beacons
of motivation and support since İstanbul Erkek Lisesi, our high school. I am
still moved to tears when I think about all we accomplished together, from
Technology Student Association trophies, to film contests and İcat Çıkar. Also
enormous thanks to my dear friends from the HCI community and beyond,
including Eric, Frederik, Jasper, Jingyi, Mert, and Parastoo, whose presence
has been a pillar of support throughout my PhD journey. And heartfelt thanks
to Marcel, who made Delft feel like a true home during times of uncertainty.

I consider myself extremely fortunate to have had the opportunity to
collaborate with exceptional mentors and role models from institutions around
the world. I am grateful to Sriram Subramanian and Kaan Akşit from University
College London, Elvin Karana, and Zjenja Doubrovski from TU Delft, Thijs
Roumen from Cornell Tech, Alexa Siu, Jennifer Healey, Tong Sun, Chang Xiao,
and Eunyee Koh from Adobe Research, David Kim, Mar-Gonzalez Franco, and Ruofei
Du from Google, and Aakar Gupta from Meta Reality Labs. Their invaluable
contributions have elevated the impact and rigor of my research.

I am also immensely grateful to the professors and mentors from my college
years, with whom I have not only forged a strong academic bond but also
developed lasting friendships over the years. I owe Metin Sitti, İlke Ercan,
Ali Emre Pusane, Donghoon Son, and Ankur Mehta a profound debt of gratitude
for instilling in me the love for learning and the belief in my own potential.
Their mentorship laid the foundation upon which I stand today.

I am profoundly thankful to my parents, Neşe and Rifat, and my sister, Bengü,
I owe an immeasurable debt of gratitude for their unconditional love,
understanding, and belief in my abilities. Their support has been a constant
source of motivation throughout the years.

In closing, I would like to express my deep appreciation to all individuals
and institutions that have played a role in shaping my doctoral journey. Your
collective support and encouragement have been invaluable in realizing my
ambitions and aspirations.

###### Contents

  1. 1 Introduction
    1. 1.1 Design Criteria
    2. 1.2 Categories of Machine-Readable Markers
    3. 1.3 Natural Markers
    4. 1.4 Structural Markers
      1. 1.4.1 Laser Cutting Artifacts as Machine-Readable Data
      2. 1.4.2 3D Printing Artifacts as Identifiers
    5. 1.5 Internal Markers
      1. 1.5.1 Invisible Markers in 3D Printed Objects
      2. 1.5.2 Increasing the Contrast of Markers
    6. 1.6 Structure of the Thesis
  2. 2 Related Work
    1. 2.1 Computer Vision-Based Approaches
    2. 2.2 Use Cases for Markers in HCI
    3. 2.3 Making Markers Less Obtrusive
      1. 2.3.1 Marker Location: Inside, Surface, Structure
      2. 2.3.2 Data Complexity: Identifiers, Information
      3. 2.3.3 Complexity of Tagging Approach: Fabrication, Detection
  3. 3 SensiCut: Material-Aware Laser Cutting Using Speckle Sensing and Deep Learning
    1. 3.1 Introduction
    2. 3.2 Motivation
    3. 3.3 Speckle Sensing Hardware Add-On for Laser Cutters
      1. 3.3.1 Speckle Sensing Working Principle
      2. 3.3.2 Hardware Add-On
    4. 3.4 User Interface and Applications
      1. 3.4.1 Identifying Single Material Sheets
      2. 3.4.2 Identifying Multiple Sheets of Different Materials at Once
      3. 3.4.3 Engraving onto Multi-Material Objects
    5. 3.5 Classification of Materials
      1. 3.5.1 Choosing Material Samples
      2. 3.5.2 Data Capture and Material Speckle Dataset
      3. 3.5.3 Training the Neural Network
    6. 3.6 Evaluation of Material Classification
      1. 3.6.1 Detection Accuracy Results
      2. 3.6.2 Effect of Illumination and Sheet Orientation on Detection
      3. 3.6.3 Generalization to Different Material Batches and Manufacturers
    7. 3.7 Software Implementation
    8. 3.8 Discussion
    9. 3.9 Conclusion
  4. 4 StructCodes: Leveraging Fabrication Artifacts to Store Data in Laser-Cut Objects
    1. 4.1 Introduction
    2. 4.2 Method: Embedding Data in StructCodes
      1. 4.2.1 Identifying Features Suitable for StructCodes
      2. 4.2.2 Structural Embedding of Data Bits
      3. 4.2.3 Encoding Scheme
    3. 4.3 End-to-End Workflow
      1. 4.3.1 User Interface for Embedding StructCodes
      2. 4.3.2 Mobile Interface for Reading StructCodes
    4. 4.4 Applications
      1. 4.4.1 Embedding Identifiers for Static or Dynamic Labels
      2. 4.4.2 Embedding Context: Resources and Instructions
      3. 4.4.3 Embedding Overlaid Media
    5. 4.5 Detection of StructCodes
      1. 4.5.1 Detection of Finger Joints
      2. 4.5.2 Detection of Living Hinges
      3. 4.5.3 Evaluation of the Detection Pipelines
    6. 4.6 Mechanical Evaluation
      1. 4.6.1 Compression Evaluation of Finger Joints
      2. 4.6.2 Bending Evaluation of Living Hinges
    7. 4.7 Software Implementation
    8. 4.8 Discussion
    9. 4.9 Conclusion
  5. 5 G-ID: Identifying 3D Prints Using Slicing Parameters
    1. 5.1 Introduction
    2. 5.2 Method: Labeling and Identifying Objects By Their Slicing Parameters
      1. 5.2.1 Main Benefits of Using Different Slicing Parameters
      2. 5.2.2 G-ID Workflow for an Identification Application
      3. 5.2.3 Labeling Interface (Slicer)
      4. 5.2.4 Identification Interface (Mobile App with Object Alignment)
      5. 5.2.5 Surface and Interior: Detecting Infill Using a Light Source
    3. 5.3 Slicing Parameters Used for Labeling
      1. 5.3.1 Surface Parameters
      2. 5.3.2 Infill Parameters
      3. 5.3.3 Selecting Parameters that Minimize Print Time and Material
    4. 5.4 Detecting and Identifying Slicing Parameters
      1. 5.4.1 Aligning the Object’s Base in Handheld Camera Images
      2. 5.4.2 Detecting Bottom Line Angle and Width
      3. 5.4.3 Error Checking
      4. 5.4.4 Detecting Infill Angle, Width, and Pattern
    5. 5.5 Spacing of Slicing Parameters
      1. 5.5.1 Selecting 3D Models to Evaluate Parameter Spacings
      2. 5.5.2 Determining the Range for Each Slicer Setting
      3. 5.5.3 Slicing with Different Spacings and Capturing Photos
      4. 5.5.4 Results of the Experiments
      5. 5.5.5 Total Number of Instances Possible
      6. 5.5.6 Cross-Validation
    6. 5.6 System Implementation
    7. 5.7 Evaluation
      1. 5.7.1 Different Materials, Lighting Conditions, Thicknesses
      2. 5.7.2 Different 3D Printers
      3. 5.7.3 Camera Distance and Angle
    8. 5.8 Applications
    9. 5.9 Discussion and Limitations
      1. 5.9.1 Other Slicing Parameters
      2. 5.9.2 Rotational Symmetry of Outlines
      3. 5.9.3 Non-Flat Side Surfaces
      4. 5.9.4 Applicability Beyond FDM Printing
    10. 5.10 Conclusion
  6. 6 InfraredTags: Embedding Invisible Markers Using Infrared-Translucent 3D Printing Filaments
    1. 6.1 Introduction
    2. 6.2 The InfraredTags System
      1. 6.2.1 Infrared Filament
      2. 6.2.2 Choosing an Infrared Camera
      3. 6.2.3 Composition of the Tags and Materials
    3. 6.3 Embedding and Reading InfraredTags
      1. 6.3.1 User Interface for Encoding InfraredTags
      2. 6.3.2 IR Imaging Module for Reading the Tags
    4. 6.4 Software Implementation
      1. 6.4.1 UI Implementation
      2. 6.4.2 Mobile IR Imaging
      3. 6.4.3 Image Processing Pipeline
    5. 6.5 Applications
      1. 6.5.1 Distant Augmented Reality (AR) Interactions with Physical Devices
      2. 6.5.2 Embedding Metadata about Objects
      3. 6.5.3 Tangible Interactions: Use Anything as a Game Controller
    6. 6.6 Evaluation of the Detection
    7. 6.7 Discussion
    8. 6.8 Conclusion
  7. 7 BrightMarkers: 3D Printed Fluorescent Markers for Object Tracking
    1. 7.1 Method: High-Contrast Markers Using Fluorescent Filaments
      1. 7.1.1 Fluorescent Filament
      2. 7.1.2 Infrared Imaging of Fluorescence
    2. 7.2 Fabricating, Capturing, and Tracking BrightMarkers
      1. 7.2.1 Adding Markers for Fabrication
      2. 7.2.2 Detection Using Imaging Modules
      3. 7.2.3 Image Processing for Marker Detection
    3. 7.3 Applications
      1. 7.3.1 Rapid Product Tracking
      2. 7.3.2 Wearables for Tracking Human Motion
      3. 7.3.3 Tangible Interfaces for MR Experiences
      4. 7.3.4 Privacy-Preserving Night Vision
    4. 7.4 Evaluation
      1. 7.4.1 Detection Distance, Excitation Intensity, and Surface Color
      2. 7.4.2 Detection Rate and Marker Speed
      3. 7.4.3 Marker Size
    5. 7.5 Discussion
      1. 7.5.1 Concentration of Fluorescent Dye
      2. 7.5.2 Mass Production
      3. 7.5.3 NIR Power
      4. 7.5.4 Embedding Circular Markers
      5. 7.5.5 Occlusion
    6. 7.6 Conclusion
  8. 8 Discussion
    1. 8.1 Scaling Up and Mass Production
    2. 8.2 Privacy
    3. 8.3 Embedded Markers in Consumer Products
    4. 8.4 Interaction and Personalization in AR
  9. 9 Conclusion

###### List of Figures

  1. 1.1 My research vision. Reading metadata embedded unobtrusively into objects and materials in the physical world and reflecting it in the digital world.
  2. 1.2 SensiCut. (a) Many laser cutting materials look alike and are hard to distinguish visually by users or regular cameras. Furthermore, there are many hazardous materials that are often confused for safe ones. (b) SensiCut instead senses the sheet’s unique surface structure using speckle imaging and deep learning. This enables us to select the correct machine settings, which prevents material waste and ensures user safety, as well as (c) precisely engrave multi-material objects.
  3. 1.3 StructCode embeds data in the fabrication artifacts of laser-cut objects, such as the patterns of (a) living hinges and (b) finger joints, to augment objects with data. Here, the embedded StructCodes allow narration for a painting and status updates for a potted plant, among others.
  4. 1.4 G-ID. (a) 3D printed objects inherently possess surface patterns due to the print path. G-ID exploits such features that would normally go unnoticed to identify unique instances of an object. Our mobile app (b) uses image processing to detect them.
  5. 1.5 InfraredTags are 2D markers and barcodes embedded unobtrusively into 3D printed objects and can be detected using infrared cameras (top-right images). This allows new applications for (a) identifying and controlling devices in AR interfaces, (b) embedding metadata such as 3D model URLs into objects, and (c) tracking passive objects for tangible interactions.
  6. 1.6 BrightMarkers are embedded into objects using a NIR-fluorescent filament. (a) When viewed with a NIR camera with the matching filter, the markers appear with high contrast, which allows them to be tracked even when the objects are in motion, e.g., on a conveyor belt. (b) BrightMarker can be used to fabricate custom wearables for tracking, or (c) for transforming physical controls into precise input methods in MR environments.
  7. 2.1 An exaggerated portrayal of an urban environment saturated with QR codes, showing the aesthetic and practical challenges of current object tagging methods. Illustration used with permission from Michael Sloan.
  8. 3.1 Existing material identification approaches: (a) Manually selecting from a database (e.g., ULS [135]) or (b) scanning QR code stickers on sheets (Glowforge). (c) SensiCut uses speckle sensing to identify the material based on its surface structure without the need for additional tags.
  9. 3.2 SensiCut augments standard laser cutters with a speckle sensing add-on that can (a) identify materials often found in workshops, including visually similar ones. (b) SensiCut’s user interface integrates material identification into the laser cutting workflow and also offers suggestions on how to adjust a design’s geometry based on the identified material (e.g., adjusting the size of an earring cut from felt since the kerf for felt is larger than for other materials). (c) Each identified sheet is cut with the correct power and speed settings.
  10. 3.3 Speckle sensing. (a) Laser rays reflect off the material surface and arrive at the image sensor. Phase differences between the rays result in mutual interference and thus dark or bright pixels in the captured image. (b) Different materials viewed by a regular camera, a scanning electron microscope, and our speckle sensing imaging setup. (c) Our speckle sensing add-on consists of a laser pointer, lensless image sensor, microprocessor, and battery.
  11. 3.4 SensiCut UI. (a) The Pinpoint tool allows users to identify the material at a desired location on the cutting bed. (b) The user enters the thickness value corresponding to the detected sheet and starts cutting.
  12. 3.5 Making a face shield. (a) Find flexible sheets from the stockpiles and (b) place in the cutter. (c) Upon identification, the UI labels all 3 materials and gives relevant information on, e.g., their handling and safety. (d) After the cut parts are assembled, the shield can be safely sanitized with alcohol.
  13. 3.6 Rapid material testing for product design. (a) Choosing multiple samples from a material swatch and (b) inserting in the laser cutter. (c) After sensing, the UI matches each shape with the corresponding material. It also warns that the kerf for felt will compromise the design.
  14. 3.7 Adjusting the design for kerf. (a) The user can enlarge the shape to compensate for the thicker kerf for felt. (b) Without the adjustment, details like the hole and fine blades of the leaf disappear in the cut felt. (c) The fabricated prototypes.
  15. 3.8 Engraving a multi-material phone case, which consists of (a) wood-like rubber and leather. SensiCut scans (b) the input design’s outline and (c) splits it into two parts based on the material type detection. (d) Engraved design.
  16. 3.9 Engraving a pattern on a T-shirt that has (a) plastic details on it. (b) SensiCut uses the right combination of laser settings after partitioning the design (middle). Top/bottom shows the outcome for singular settings.
  17. 3.10 Material considerations and evaluation. (a) Most common material types for laser cutting. (b) Visually similar materials. (c) Confusion matrix from our trained classifier for 30 material subtypes.
  18. 3.11 Effect of (a) varying illumination and (b) sheet orientation on detection.
  19. 3.12 Our detection pipeline takes the user’s drawing as input and turns it into target points to capture speckle patterns. The captured images at those points are passed to the CNN to retrieve the material label.
  20. 4.1 StructCode embeds data in the fabrication artifacts of laser-cut objects, such as the patterns of (a) finger joints and (b) living hinges, to augment objects with data. Here, the embedded StructCodes allow narration for a painting and status updates for a potted plant, among others.
  21. 4.2 General design of (a) finger joints and (b) living hinges. To embed bits, we resize both the fingers and gaps in the finger joints, but only the links in the living hinges.
  22. 4.3 Workflow for encoding and decoding information. (a) After the tool highlights the compatible structures, the user selects the hinge and encodes "Red Oak". (b) They use the StructCode mobile application to decode the message.
  23. 4.4 (a) Document folders with encoded personal labels, which the user views in AR to retrieve the right one. (b, c) Status updates inform the user about when the plant needs watering.
  24. 4.5 Providing context. (a) The user can access the fabrication files of the furniture. (b) Running low on cookies, the user adds a new batch to the online shopping cart through the embedded reorder link.
  25. 4.6 Disassembly instructions are linked to the shelf in case the user needs to take it apart for recycling or transport.
  26. 4.7 Overlaying media. (a) In an exhibit, visitors use an AR app to view narrative videos by the artists. (b) The crocodile has StructCodes that describe what part they are located on, which is used for educational applications.
  27. 4.8 Image processing steps for (a) finger joints and (b) living hinges.
  28. 4.9 Curvature of living hinges. (a) As bit lengths may be distorted due to curvature, StructCode encodes them with sufficient tolerance to ensure correct detection. (b) StructCodes can be encoded on various shapes, e.g., conical hinges.
  29. 4.10 Mechanical evaluation of StructCodes: (a) Compressive strength of finger joints and (b) bending of hinges.
  30. 5.1 Different ways to embed tags into 3D models while leaving the surface intact: (a) changing the internal geometry (InfraStructs), (b) varying slicing parameters (G-ID), (c) relying on fabrication imprecisions (PrinTracker).
  31. 5.2 3D printed objects inherently possess surface patterns due to the angle of the print path and the thickness of the trace the 3D printer lays down. G-ID exploits such features to identify unique instances of an object without the need to embed an obtrusive, additional tag. G-ID provides (a) a user interface for slicing individual instances of the same object with different settings and assigning labels to them. After (b) 3D printing, users can (c) identify each instance using the G-ID mobile app that uses image processing techniques to detect the slicing parameters and retrieve the associated labels.
  32. 5.3 G-ID labeling interface: load a 3D model and enter number of unique instances needed.
  33. 5.4 G-ID mobile app for identifying instances: (a) select model from library, (b) face the object, (c) once outlines are aligned, the app automatically takes the image.
  34. 5.5 By adding a small light source, we can also detect variations in infill, such as different infill angles, patterns, and densities, which allow for a larger number of instances. Here, the coffee maker recognizes the mug’s infill and pours the user’s preferred drink.
  35. 5.6 Combinations of different line widths and angles.
  36. 5.7 Different layer angles achieved by rotating the model at the expense of additional support material and print time (a vs. b), and various print qualities (b vs. c).
  37. 5.8 Cross-sections of the mug model show (a) different infill angles and densities, (b) different infill patterns.
  38. 5.9 Image registration & processing pipeline. (a) The captured outline is registered with that of the 3D model model for (b) improved alignment. (c) Its Fourier transform is used to infer line angle $\delta$ and width $d$. (d) The distance $\Delta x$ between intensity peaks on the inclined line is inversely proportional to $d$.
  39. 5.10 Image processing to extract the components of the infill pattern: (a) photo, (b) contrast increased, (c) blurred, (d) binarized, and (e) matched to the template of the respective infill type: grid, triangles, trihexagon from top to bottom.
  40. 5.11 Detection accuracies vs. parameter spacing between pairs of instances for the four slicing parameters (bottom line angle and width, as well as infill angle and width). All plots have the same y-range.
  41. 5.12 Detection accuracies vs. parameter spacing between pairs of instances for the four slicing parameters (bottom line angle & width, as well as infill angle & width). All plots have the same y-range.
  42. 5.13 System overview.
  43. 5.14 Different filament colors vs. minimum illuminance required to correctly detect the traces on the bottom surface.
  44. 5.15 Identifying particular print settings with G-ID: (a) exploring different slicing parameters and (b) printing them, (c) retrieving the best settings using G-ID.
  45. 5.16 Toys-to-life figurines are used to identify the player and display their info in the video game.
  46. 5.17 The hotter the nozzle, the darker the print’s finish.
  47. 5.18 (a) Model’s axis of rotation is orthogonal to the image plane of the virtual camera, (b) rendered target outline for the user to align model, (c) image has parallel traces.
  48. 5.19 A sample from a Formlabs SLA printer captured by a conventional mobile phone camera.
  49. 6.1 Material composition of the tags for a sample ArUco marker. We modify the interior of the object to embed the tag based on (a) single- or (b) multi-material printing. (c) The transmission spectrum of the IR PLA and regular PLA.
  50. 6.2 Determining the shell thickness for a multi-material print with white PLA. As $t_{shell}$ increases, the checkerboard pattern becomes less visible in both (a) the visible camera and (b) IR camera image. Thus, it gets more challenging to (d) identify the contrast in the pattern for humans, and to (c, e) binarize it correctly from the IR view.
  51. 6.3 InfraredTags embedding interface.
  52. 6.4 Infrared imaging module. (a) The module is attached onto a flexible case that can be 3D printed based on the user’s mobile device. (b) The module’s hardware components.
  53. 6.5 Image processing to read the tags. (a) Infrared camera view. (b) Individual processing steps needed to decode the QR code message: "HCI_IR_TEST".
  54. 6.6 Controlling appliances using a mobile AR application. The user points at (a) the home speakers to adjust its volume, and the (b) thermostat to adjust the temperature. The infrared camera in the phone’s case identifies the appliances by reading the embedded QR codes. (c) Pairing a phone with a WiFi router, whose SSID is visible from all sides but the password is visible only from its bottom.
  55. 6.7 Embedded metadata about the object itself: The user is redirected to the Thingiverse model that was used to fabricate the object.
  56. 6.8 Using passive objects (a) as a game controller. (b) This wheel is black under visible light and has no electronic components. (c) The fiducial marker embedded inside is only visible to an infrared camera.
  57. 6.9 Detection evaluation. (a) Maximum detection distance for single- and multi-material ArUco markers. (b) Cases where the IR LED and the visible cut-off filter improve detection.
  58. 7.1 Ways to add tracking capabilities to objects as used in previous HCI projects. External tags rely on adding retro-reflective beads [123, 38], sensor modules [166, 37], or fiducial markers [125], but may result in bulky or visually obtrusive objects. To address this, BrightMarker (rightmost) embeds high-contrast markers using a fluorescent material.
  59. 7.2 Fluorescence and our imaging approach. (a) BrightMarker embeds tracking markers with fluorescent filaments, which "shift" the wavelength of IR radiation. (b) Although the excitation and emission spectra of the filament overlap (top), they can be separated in practice using optical tools (bottom). (c) Our imaging setup filters for the marker’s fluorescence.
  60. 7.3 The design space of possible material colors. Each test slab was printed increasing shell thickness. The left image of each pair shows the visible camera capture, while the right image shows the NIR capture.
  61. 7.4 Marker embedding process. (a) Our tool allows users to (b) uniformly distribute markers based on the object geometry.
  62. 7.5 Hardware modules for tracking BrightMarkers.
  63. 7.6 BrightMarker’s image processing pipeline. (a) The objects are tracked using (b) the outlines in IR capture. (c, d) The localized markers are decoded using a set of filters.
  64. 7.7 Wearable for hand tracking. (a) Rigid and flexible bracelet designs with embedded markers. (b) Fluorescence imaging is used to detect the unique tracking IDs. (c) The user’s motion is digitized.
  65. 7.8 Using a lightsaber as a prop to slice fruits in a game.
  66. 7.9 Privacy-preserving night vision. (a) Regular CCTVs help monitor important objects but may intrude on users’ privacy. (b) Our detection setup allows tracking of solely the fluorescent objects.
  67. 7.10 Excitation intensity required with increasing marker distance from the camera.
  68. 8.1 Adaptive display of related information acquired from unobtrusive tags could be utilized to create a more personalized and assistive mixed reality environment.

###### List of Tables

  1. 4.1 Average peak load comparison of the finger joints.
  2. 4.2 Average bend angle of the living hinges.
  3. 5.1 Each slicing parameter’s range (min, max) and incremental spacing values as determined by our experiments. The last column shows the number of variations that can be realized. (*2 angles reserved for error checking).
  4. 5.2 Polar angle $\theta$ vs. the percentage of identified objects.
  5. 6.1 Thickness values for the shell and code layers.
  6. 6.2 Filter combinations and QR code detection accuracy

##  Chapter 1 Introduction

Mobile and wearable devices promise to enrich our daily interactions by
providing information to us whenever we need it. These devices, especially
those intended for augmented reality (AR), or more broadly, mixed reality (MR)
applications, need to constantly identify what is around us and display
relevant digital content. Imagine yourself in a store, browsing various items,
and desiring further information about the ones that caught your attention:
the unit price, calorie information, recipes, or perhaps more affordable
options. To facilitate such interactions, the objects around us need to carry
labels that describe what they are and communicate this information to digital
devices. While paper-based labels like QR codes are cost-effective to
manufacture, they often prove visually distracting and lack the durability and
security of electromagnetic alternatives like radio-frequency identification
tags (RFID), which can be discreetly embedded within the object. In this
thesis, we study the design, fabrication, and detection of novel, low-cost,
and durable physical tagging mechanisms that allow for unobtrusive
identification of everyday objects, products, and materials to overcome these
challenges.

Figure 1.1: My research vision. Reading metadata embedded unobtrusively into
objects and materials in the physical world and reflecting it in the digital
world.

We envision that one day, every real-world object in our lives will contain
some sort of metadata that describes its identity, look, origin, function, or
instructions on how to use it — and we will be able to access these just by
pointing our phones or augmented reality glasses at the objects (Figure 1.1).
This is familiar to us in the digital ecosystem, where audio or image files
include embedded metadata that helps the user or software seamlessly identify
not just what they are but also when and by whom they were captured or
recorded. We aspire to transfer this paradigm to physical objects – using
embedded physical markers. We call this "ubiquitous metadata."

Ubiquitous metadata refers to the concept of embedding metadata into physical
objects, allowing them to be identified and augmented with relevant
information when viewed through digital devices. This approach could allow
information to weave itself into the "fabric of everyday life" until it is
indistinguishable from it [150]. The implementation of ubiquitous metadata
through physical tagging mechanisms requires robust fabrication and detection
pipelines. If implemented correctly, this can open up new possibilities for
immersive spatial computing, where the paradigm will shift from content inside
the screen, to users inside the content, where one will be able to directly
and seamlessly interact with and contribute to the digital content associated
with physical objects.

###  1.1 Design Criteria

As we progress in our exploration of machine-readable markers, we have
identified pivotal design criteria that serve as the guiding principles for to
our proposed tagging methods. This list is derived from the limitations of
existing solutions and literature, which we will detail in Section 2. The
criteria consider the practicality, robustness, and widespread adoption of the
markers in real-world applications, and are listed as follows.

  1. 1.

Embeddedness: We advocate for these markers to be integral components of
objects right from the fabrication stage, rather than being considered as
afterthoughts or additions. This seamless integration into the physical
object’s design ensures both unobtrusiveness and resistance to tampering. The
embedding strategy may be more opportunistic, implying a reliance on existing
features for identification, or more engineered, involving more sophisticated
marker embedding processes to achieve specific design objectives. Some of our
methods demonstrate the feasibility of integrating the marker embedding
process into existing manufacturing pipelines, which enables the creation of
the object and its marker in a single, harmonious process, thus preserving the
object’s form and appearance.

  2. 2.

Uniqueness: Our proposed tagging mechanisms feature a unique identifier (ID)
or metadata for precise object or product identification. A unique ID can come
in the form of a distinctive pattern in each object instance for facilitating
straightforward identification and accurate data retrieval from digital
databases. On the other hand, a unique metadata approach involves embedding
more comprehensive information directly into the object, providing a richer
context beyond a mere identifier. The choice between these two options depends
on the specific requirements of the application and the desired depth of self-
contained information associated with each physical entity.

  3. 3.

Cost-effectiveness: The methods prioritize cost-effectiveness, leveraging
readily available materials and standard manufacturing processes. By operating
without an embedded power source, i.e., adopting a passive design, these
markers also become more durable and less susceptible to failure,
accommodating diverse usage scenarios and environmental conditions. Further,
this approach reduces the complexity of tag design and streamlines the
object’s fabrication process.

  4. 4.

Detectability: Regardless of their embedded nature, these markers should be
easily detectable by commonly available sensors, such as RGB and near-infrared
cameras on devices. This not only enhances user experience but also promotes
widespread adoption, as it eliminates the need for specialized equipment. The
choice of sensors should align with the intended application to find a balance
between accuracy and accessibility.

  5. 5.

Spatiality: Spatial awareness plays a pivotal role in the successful
integration of ubiquitous metadata, especially in the context of AR. Some of
the markers we will introduce in this thesis provide spatial information about
the objects that they are attached to. Knowing the exact position and
orientation of the objects is critical for accurately overlaying and anchoring
virtual elements in AR applications. Additionally, the markers should retain
their machine-readability from various angles and distances to accommodate the
dynamic interactions characteristic of AR applications.

In summary, the design criteria of the proposed embedded machine-readable
markers encompass important considerations, including the integrated embedding
of markers as part of the object design, the choice between unique ID and
metadata, the cost-effectiveness of passive solutions, detectability through
accessible sensors, and spatial anchoring in AR. Establishing a balance to
meet these criteria is essential for creating effective and versatile systems.

###  1.2 Categories of Machine-Readable Markers

Aligned with the established design criteria, this thesis centers around the
exploration and development of three distinct categories of embedded machine-
detectable markers. These categories are strategically designed to
comprehensively address the identified criteria through different technical
methods.

The categories, presented below based on the design criterion of embeddedness,
range from the most opportunistic, implying a reliance on existing features
for identification, to the most engineered, meaning more sophisticated marker
embedding processes.

  1. 1.

Natural markers: Leveraging the inherent fingerprints of objects and
materials, this category explores the use of natural features for
identification.

  2. 2.

Structural markers: Exploiting the structural artifacts that emerge during the
fabrication process, this category aims to repurpose inherent elements in the
object’s construction as markers.

  3. 3.

Internal markers: Using specialized materials, this category involves
embedding custom markers seamlessly inside the fabricated items.

In the pursuit of these categories, novel methods were developed and
evaluated, optimizing the design of the markers to have minimal impact on the
object’s look and form (i.e., unobtrusiveness). The fabrication processes were
designed to be compatible with low-cost tools, such as commonly available
fused deposition modeling (FDM) 3D printers. Subsequently, the markers can be
detected using readily accessible sensors such as RGB and near-infrared
cameras.

Throughout the following sections, we will study these tagging approaches in
detail and elaborate on the methodologies used to ensure their effectiveness.
By combining techniques from computer vision, machine learning, computational
imaging, and material science, our presented approaches offer robust and
versatile solutions for object identification, tracking, and interaction.

###  1.3 Natural Markers

Natural markers allow us to leverage objects’ natural properties, such as
their micron-scale surface texture, as identifiers.

To make use of naturally available markers, we can employ optical methods such
as laser speckle imaging, where an image sensor detects how the laser reflects
off the object’s raw surface. In this method, differences in surface structure
result in unique speckle patterns for each material type, which allows us to
classify objects. While we can integrate this technology in any digital
device, it comes in handy for tools that already have a laser source, e.g.,
laser cutters.

For the project SensiCut [23], we leverage laser speckle imaging to sense
material sheets in laser cutters without any pre-labeled stickers (Figure
1.2). To identify materials in traditional laser cutting and set the laser
power or speed settings accordingly, users either manually select the type
from a database, or an on-board camera takes pictures of material sheets that
come with a QR code. Our imaging method, on the other hand, can automate this
mundane but risky task, and is not subject to the limitations of conventional
cameras, which may confuse visually similar materials, or QR codes, which may
be cut off from sheets.

Figure 1.2: SensiCut. (a) Many laser cutting materials look alike and are hard
to distinguish visually by users or regular cameras. Furthermore, there are
many hazardous materials that are often confused for safe ones. (b) SensiCut
instead senses the sheet’s unique surface structure using speckle imaging and
deep learning. This enables us to select the correct machine settings, which
prevents material waste and ensures user safety, as well as (c) precisely
engrave multi-material objects.

Our technical evaluation shows that SensiCut can accurately sense laser
cutting materials most commonly used by designers under different
environmental conditions, such as illumination and sheet orientation.
SensiCut’s sensing hardware can be attached to existing laser cutters with our
compact sensing hardware add-on, enabling a new workflow for using laser
cutters and applications, e.g., providing user safety alerts when hazardous
materials are detected, facilitating rapid prototyping, and engraving multi-
material objects.

While SensiCut focuses on addressing these important laser cutting challenges
as identified in recent human-computer interaction literature (Yildirim et al.
[160]), our technique may be used within everyday tools and consumer
electronics such as smartphones, many of which now come with infrared lasers
for facial recognition [50]. More recently, research prototypes have used
speckle imaging hardware directly on mixed reality headsets to amplify
detection capabilities, showing the potential for sensing-enhanced AR
interactions in the future [131].

###  1.4 Structural Markers

A distinct category within embedded markers closely aligns with the
fabrication process itself. These markers, which we refer to as structural
markers, leverage the inherent characteristics and structures that emerge
during the fabrication process to create markers. In this section, we will
introduce two methods that utilize different digital fabrication processes and
enable different use cases.

####  1.4.1 Laser Cutting Artifacts as Machine-Readable Data

StructCode [22] allows users to store machine-readable metadata within laser-
cut objects’ structure by harnessing the fabrication artifacts themselves. By
selectively modifying the lengths of laser-burnt finger joints and living
hinges, StructCode represents different bits of information without requiring
additional parts or materials. These modifications, inherent to the
fabrication process, enable the integration of meta-information such as
labels, instructions, and narration directly into the physical objects (Figure
1.3). StructCode provides a reliable and unobtrusive means of capturing
information, and can be read with standard device cameras, such as those
available on smartphones and AR/VR headsets.

Figure 1.3: StructCode embeds data in the fabrication artifacts of laser-cut
objects, such as the patterns of (a) living hinges and (b) finger joints, to
augment objects with data. Here, the embedded StructCodes allow narration for
a painting and status updates for a potted plant, among others.

We present and evaluate a marker decoding pipeline that is robust to various
backgrounds, viewing angles, and wood types. In our mechanical evaluation, we
show that StructCodes preserve the structural integrity of laser-cut objects.

####  1.4.2 3D Printing Artifacts as Identifiers

Compared to fabrications methods such as laser cutting, 3D printing provides
greater flexibility in achieving complex 3D geometries for objects. To
integrate identifiers into 3D printed items, we may exploit the artifacts
resulting from the extrusion-based FDM process. By manipulating the 3D
printer’s path through custom CNC machine instructions ("G-code"), we can
obtain unique and subtle surface textures for each instance of the same 3D
model.

By strategically adjusting slicing parameters that do not alter the geometry
of the object, our method G-ID [25, 24] creates unobtrusive machine-readable
differences. These variations serve as distinct markers for identifying and
distinguishing similar-looking objects from a single photograph captured using
a conventional smartphone (Figure 1.4).

Figure 1.4: G-ID. (a) 3D printed objects inherently possess surface patterns
due to the print path. G-ID exploits such features that would normally go
unnoticed to identify unique instances of an object. Our mobile app (b) uses
image processing to detect them.

We evaluated how finely these texture-related parameter differences can be
differentiated between, and built a mobile application that uses image
processing techniques to retrieve these parameters from photos and their
associated labels. This enables low-cost, interactive applications for user
identification.

###  1.5 Internal Markers

While G-ID is powerful for distinguishing between different copies of 3D
printed objects, an important limitation is that the objects do not carry a
large amount of data, but rather a single, uniquely identifiable texture. This
restricts users from embedding larger amounts of custom information.

On the other hand, internal markers enable users to embed data in the form of
2D markers directly into 3D printed objects. Previous works in internal
markers require the use of expensive or specialized equipment. For instance,
AirCode [80] uses a camera and projector setup to detect gaps beneath the
object surface. InfraStructs [153] uses a terahertz scanner to detect even
deeper holes. Our goal is to develop internal markers that are easy to
fabricate and that can be detected using low-cost scanning tools.

####  1.5.1 Invisible Markers in 3D Printed Objects

InfraredTags [29, 30] are 2D markers and barcodes imperceptible to the naked
eye that can be 3D printed as part of objects, and detected rapidly by low-
cost near-infrared (NIR) cameras (Figure 1.5). These 2D markers and barcodes
consists of multiple bits, as opposed to a single texture demonstrated in
G-ID. We achieve this by printing objects from an infrared-translucent
filament, which infrared cameras can see through, and by having air gaps
inside for the tag’s bits, which appear at a different intensity in the
infrared image.

Figure 1.5: InfraredTags are 2D markers and barcodes embedded unobtrusively
into 3D printed objects and can be detected using infrared cameras (top-right
images). This allows new applications for (a) identifying and controlling
devices in AR interfaces, (b) embedding metadata such as 3D model URLs into
objects, and (c) tracking passive objects for tangible interactions.

By integrating InfraredTags into 3D printed objects during the fabrication
process, we enable interactive applications for identifying and controlling
devices in AR interfaces or embedding metadata such as links to download the
object’s 3D model. We also show how convolutional neural networks can be used
in conjunction with data augmentation methods to improve the detection of
these tags from near-infrared camera streams [31].

####  1.5.2 Increasing the Contrast of Markers

We built on InfraredTag to create markers that have higher contrast. Called
BrightMarker [27, 26], these markers are are fabricated using NIR-fluorescent
filaments to allow real-time tracking of 3D printed color objects. By using an
infrared-fluorescent filament that "shifts" the wavelength of the incident
light, our optical detection setup filters out all the noise to only have the
markers present in the infrared camera image. The high contrast of the markers
allows us to track them robustly regardless of the moving objects’ surface
color (Figure 1.6).

BrightMarker can be used in a variety of applications, such as custom
fabricated wearables for motion capture, tangible interfaces for AR/VR, rapid
product tracking, and privacy-preserving night vision. BrightMarker exceeds
the detection rate of state-of-the-art invisible marking, and even small
markers (1"x1") can be tracked at distances exceeding 2m.

Figure 1.6: BrightMarkers are embedded into objects using a NIR-fluorescent
filament. (a) When viewed with a NIR camera with the matching filter, the
markers appear with high contrast, which allows them to be tracked even when
the objects are in motion, e.g., on a conveyor belt. (b) BrightMarker can be
used to fabricate custom wearables for tracking, or (c) for transforming
physical controls into precise input methods in MR environments.

The integration of these embedded machine-readable markers into real-world
objects serves as a gateway to virtual worlds, enabling users to access a
wealth of multimedia content, interactive experiences, and valuable contextual
information. We will showcase the applications of our methods across diverse
domains, including product design, manufacturing, entertainment, marketing,
logistics, security, and sustainability.

###  1.6 Structure of the Thesis

In the remainder of the thesis, we first introduce the landscape of related
work in the identification, tagging, and digital fabrication domains (Chapter
2). We then explain each of the previously introduced methods in individual
chapters, including natural markers (SensiCut in Chapter 3, and structural
markers (StructCodes in Chapter 4, G-ID in Chapter 5), internal markers
(InfraredTags in Chapter 6, BrightMarkers in Chapter 7).

Lastly, we address the challenges and opportunities associated with deploying
embedded machine-readable tags at scale (Chapter 8). These considerations
encompass aspects such as generalization to faster fabrication methods,
integration into consumer products, AR interactions, and privacy concerns. By
addressing these challenges, we aim to pave the way for a future where objects
come alive, environments become interactive, and virtual worlds seamlessly
merge with our everyday lives.

In conclusion, this thesis represents a comprehensive exploration of embedded
machine-readable tags as a means to bridge the gap between real-world objects
and virtual worlds. Through our research, we strive to advance a future where
ubiquitous metadata enhances our interactions, enabling us to effortlessly
access information and engage with the physical and digital realms.

##  Chapter 2 Related Work

In the realm of identifying and tagging physical objects, a wide range of
approaches has been explored to bridge the gap between the physical and
digital domains. This section provides an overview of existing methodologies,
and highlights their strengths, limitations, as well as contributions to HCI.

###  2.1 Computer Vision-Based Approaches

Computer vision, as a powerful and versatile tool, has significantly advanced
object recognition capabilities [136] Machine-learning (ML) classifiers,
particularly convolutional neural networks (CNNs) [78], have proven highly
capable at assigning general labels to objects based on visual features [45].
This made various computational solutions possible in the field of semantic
understanding. Techniques like YOLO [106] and SSD [84] provide real-time
object detection by identifying the bounding boxes of objects in the scene.
Semantic segmentation involves classifying each pixel in an image into
predefined categories, while instance segmentation takes this a step further
by distinguishing between individual instances of the same object category.
These techniques, as employed in popular models such as Mask R-CNN [54] and
U-Net [110], as well as frameworks such as MediaPipe [86], allow for more
precise identification and localization of objects.

While these classifiers excel at recognizing and categorizing objects into
broader classes, they often fall short in capturing fine-grained details
necessary for a comprehensive understanding of the scene [82, 59]. The
incapacity to extract specific metadata – such as the subtype of an object,
detailed product information, origin, or owner – poses a challenge in
achieving a deeper and more contextually rich comprehension of the
environment.

Real-world scenarios often demand more granular information about objects that
go beyond generic labels. For instance, in a retail context, being able to
access detailed product information, including specifications or pricing,
enhances the overall user experience. In industrial settings, knowing the
specific model or version of a machine component holds critical importance. As
we explore methods for ubiquitous metadata, we aim to address this limitation
by capturing and associating specific metadata with each object instance,
ensuring a more comprehensive and contextually relevant representation of the
physical environment.

###  2.2 Use Cases for Markers in HCI

Markers have been used to allow instance-level object identification to extend
beyond the capabilities of conventional computer vision [60].

In HCI, visual markers, such as ArUco markers, barcodes, and QR codes have
been used to mark objects and enable different interactive applications with
them. For instance, Printed Paper Markers [162] use different paper structures
that conceal and reveal fiducial markers (i.e., ArUco [109]) to create
physical inputs, such as buttons and sliders. DodecaPen [154] can transfer
users’ handwriting to the digital environment by tracking ArUco markers
attached on a passive stylus. Cooking with Robots [133] uses detachable
markers to label the real-world environment for human-robot collaboration.
Position-Correcting Tools [108] scan QR code-like markers to precisely
position CNC tools while users cut sheets.

While traditional barcodes and QR codes offer a cost-effective identification
solution, such markers are visible to the human eye, which impacts object
aesthetics and may reduce the usable area on the object, as depicted in Figure
2.1. Furthermore, practical usage often sees visually unappealing labels
removed by users, and visible codes may be tampered with by malicious third
parties [94]. Conversely, RFID tags, equipped with unique identifiers, can be
attached to physical objects for efficient tracking [129, 62]. While RFID
provides non-line-of-sight identification and finds applications in various
environments, it has limitations in terms of range and scalability [95]. The
cost of RFID tags, coupled with the size of RFID readers, may pose challenges
in large-scale deployments and integration with the object’s geometry while
objects during fabrication [151].

Figure 2.1: An exaggerated portrayal of an urban environment saturated with QR
codes, showing the aesthetic and practical challenges of current object
tagging methods. Illustration used with permission from Michael Sloan.

###  2.3 Making Markers Less Obtrusive

Researchers have investigated two primary approaches to make markers less
obtrusive: making existing visible markers more aesthetic [105, 10, 104], or
developing unobtrusive markers that can be part of the objects [89, 80].

To make existing codes more aesthetic, researchers have modified traditional
QR codes (halftoned) to look more like an aesthetic image (e.g., a photo)
while still preserving its detectability [105]. ReacTIVision [10] creates
fiducial markers that look like amoeba to create an organic look.

Specifically for 3D objects, there are more specialized ways to abstract away
the marker component. To achieve this, markers can be embedded inside the
objects, or integrated unobtrusively onto the surface of 3D objects so that
users do not perceive them as "markers."

Integrating markers with objects in a seamless and unobtrusive manner can
provide meaningful contextualization and allows for sharing of information
[36]. In HCI, the general idea has been explored for photographs [138], font
characters [155], charts [42], documents [147, 28], and textiles [165]. In the
next sections, we review different types of tags for 3D physical objects
specifically and categorize the approaches based on where the tag is located
on the object, how much data it can store, and how complex the associated
fabrication and detection processes are.

####  2.3.1 Marker Location: Inside, Surface, Structure

Researchers have investigated different locations on objects to embed markers
[8]. One way to embed markers is on the inside of objects. For instance,
AirCode [80], InfraStructs [153], and InfoPrint [69] embed markers by adding
air gaps inside 3D printed objects. Another way is to embed tags on the
surface of objects instead. For instance, Seedmarkers [44] are visual markers
optimized for aesthetics that can be placed on the surface of either laser-cut
or 3D printed objects. ObjGen [92] engraves Data Matrix cells on the object to
store the vector file it originated from. However, since multiple codes are
needed, they typically occupy the whole object surface, resulting in an
unnatural look. Tenmoku et al. place small visual components (e.g., T-shaped
or triangular) on objects to use them as visually elegant tracking markers for
mixed reality, however, these markers do not carry any data [139], Acoustic
barcodes [53] are physical notches that can be etched on the object surface
using a laser cutter and that create unique, identifiable bursts of sound when
swiped. LayerCodes [89] are barcodes on the surface of 3D models printed from
an infrared resin and then detected from smartphone camera images.

####  2.3.2 Data Complexity: Identifiers, Information

A marker is a label that can be used either to identify items or to store
information in the form of data. Kubo et al. [74] distinguish multiple copies
printed with different infill structures by measuring their vibration
characteristics. However, while such methods can store identifiers, they do
not allow users to store information specifically defined by the user, such as
custom texts or metadata, in the tag. AnisoTag [87] increases the tags’ data
capacity by combining different print textures.

To store information, AirCode [80], LayerCode [89], and InfraStructs [153]
embed barcodes and QR codes in the form of air gaps or lines ("physical bits")
within the objects. Since such codes store arrays of characters or digits,
they can be used to embed texts or metadata within the object.

####  2.3.3 Complexity of Tagging Approach: Fabrication, Detection

To embed unobtrusive markers that store information into objects, existing
methods require specialized fabrication equipment or additional post-
processing steps. For instance, LayerCodes [89] require a custom modification
to a 3D printer and a special NIR resin. AirCodes [80] require that objects
are printed in two parts and combined manually after washing away the support
material.

Another important consideration is the availability and complexity of the
detection equipment. For example, InfraStructs [153] employs a large and
costly terahertz scanner, which is not readily available to consumers.
AirCodes [80] use a camera and projector setup that are calibrated to each
other, which takes time to set up and image the tag. InfoPrint [69] uses a
thermal camera to capture patterns embedded into objects. AnisoTag [87] needs
the object to be manually swiped using a collimated laser beam and
photoresistors, making it difficult to integrate it with users’ existing
devices.

To facilitate the implementation of ubiquitous metadata, the markers should
ideally be detectable using commodity hardware, capturing the tags must follow
a simple workflow, and the detection method needs to work across a variety of
different environments.

##  Chapter 3 SensiCut: Material-Aware Laser Cutting Using Speckle Sensing and
Deep Learning

###  3.1 Introduction

While there have been many support tools for laser cutting that help users
with tasks such as automatically packing parts onto sheets [116, 124], systems
that support users with the different material types available for laser
cutting are largely unexplored [8].

For users, working with the various materials available in a workshop comes
with several challenges: First, identifying unlabeled sheets from scrap
buckets or material stockpiles in a shared workshop is challenging since many
materials are visually similar [70]. As a result, users may take the wrong
material from the stack and use it with another material’s power and speed
setting. This can lead to wasted material when the power setting is too low,
causing the outline to not be cut through – or worse the material may catch
fire when the power is too high leading to safety risks. Further, there are
many materials that are not safe to laser cut because they release toxic fumes
[117]. These hazardous materials may easily be mistaken for safe materials due
to similarity in appearance (e.g., PVC vs. acrylic) [102].

Because of the challenges outlined above, laser cutter users desire smarter
machines that can "identify the materials they [are] working with, so that the
system could […] suggest settings based on material" as shown in a recent HCI
study by Yildirim et al. [160]. One naive solution for this is to add a camera
to laser cutters to automatically identify the sheets. However, a conventional
camera can be easily fooled by visually similar materials or materials with
printed decorative textures that imitate another material.

To ensure reliable identification, recent laser cutters use sticker tags
attached to the sheets (e.g., QR codes on Glowforge Proofgrade sheets [66]).
As can be seen in Figure 3.1b, these tags can be detected by a camera even
when materials look similar or are transparent. However, scanning the tags to
detect the material type has its own issues. First, a new tag has to be
attached onto each new material sheet. Second, laser cutter users need to be
careful to not cut off the tag to ensure that the remaining part of the sheet
can later still be identified. These issues exist because using tags for
identification is not inherently material-aware as the laser cutter does not
measure the physical properties of the material.

Figure 3.1: Existing material identification approaches: (a) Manually
selecting from a database (e.g., ULS [135]) or (b) scanning QR code stickers
on sheets (Glowforge). (c) SensiCut uses speckle sensing to identify the
material based on its surface structure without the need for additional tags.

In this project, we investigate how we can identify laser cutting materials by
leveraging one of their inherent properties, i.e., surface structure. A
material’s surface structure is unique even when it is visually similar to
another type. To achieve this, we use speckle sensing. This imaging technique
works by pointing a laser onto the material’s surface and imaging the
resulting speckle patterns. We built a hardware add-on consisting of a laser
pointer and a lensless image sensor, which can be attached to the laser cutter
head using a mount. We then use the captured speckle patterns to identify the
material type with our trained neural network. Our user interface uses the
material type information to support users in different ways, i.e. it
automatically sets the power and speed settings for the detected material, it
warns the user against hazardous materials, it automatically adjusts the shape
of a design based on the kerf for the detected material, and finally, it
automatically splits designs when engraving onto multi-material objects. We
also discuss how speckle sensing can be used to estimate the thickness of
sheets as another material-aware component for future laser cutters.

Figure 3.2: SensiCut augments standard laser cutters with a speckle sensing
add-on that can (a) identify materials often found in workshops, including
visually similar ones. (b) SensiCut’s user interface integrates material
identification into the laser cutting workflow and also offers suggestions on
how to adjust a design’s geometry based on the identified material (e.g.,
adjusting the size of an earring cut from felt since the kerf for felt is
larger than for other materials). (c) Each identified sheet is cut with the
correct power and speed settings.

In summary, by leveraging speckle sensing as an identification technique, we
can improve the material awareness of existing laser cutters. Our work enables
safer and smarter material usage, addresses common material identification-
related challenges users face when laser cutting, and encourages makers to
reuse laser-cut scraps to reduce waste [20, 143]. We note that this chapter
was originally published at ACM UIST 2021 [23]. Our contributions are the
following:

  * •

An end-to-end laser cutting pipeline that helps users identify materials by
sensing the material’s surface structure using laser speckles to, e.g.,
automatically set the corresponding power/speed, warn against hazardous
materials, adjust designs based on material-specific kerf, or split designs
when engraving onto multi-material objects.

  * •

A compact (114g) and low-cost material sensing add-on for laser cutters that
simplifies hardware complexity over prior work by using deep learning.

  * •

A speckle pattern dataset of 30 material types (38,232 images), which we used
to train a convolutional neural network for robust laser cutter material
classification (98.01% accuracy).

  * •

A technical evaluation showing which visually similar materials speckle
sensing can distinguish under various sheet orientation and illumination
conditions.

###  3.2 Motivation

SensiCut addresses an important open challenge in the personal fabrication
literature. A recent field study [160] in HCI revealed an unaddressed user
need for fabrication tools concerning the “awareness of material types”. In
particular, users wished that the tools could “identify the materials they
were working with [and] suggest settings.” The authors conclude that “HCI
researchers could advance [these tools] by leveraging new sensing […]
capabilities”.

To further understand what specific challenges exist, we surveyed five
additional HCI publications [4, 65, 12, 100, 72]. We also conducted formative
interviews with six expert users that we recruited by reaching out to
makerspaces. Each expert user had several years of laser cutting experience
working with different material types. During the 1-hour semi-structured
interviews, we interviewed them about their experiences using different
material types, difficulties they had identifying materials, and how different
material types affected their designs for laser cutting. Additionally, we
performed a study in which we gave 13 novice users, who had used laser cutters
at least once but no more than four times, a list of 30 materials commonly
found in workshops (list of materials in Section 3.5.1), and asked them to
match them to 30 unlabeled sheets. For the interview responses, we took a
bottom-up approach in our thematic analysis to identify four main challenges,
which we report below.

Characterizing unlabeled sheets: We found that users have a hard time
identifying materials. In our study, novices were able to label on average
only 29.23% (SD=6.41) of the sheets correctly. The ones that were correctly
identified by most users were cardboard and cork. The top 10 mislabeled sheets
were all different types of either plastic or wood. However, this is not only
an issue for novices, but also for experts. One senior maker we interviewed
reported that certain materials are too similar to distinguish by only looking
and touching. He added that he checks if a sheet is acrylic or Delrin by
"breaking the sheet and seeing how brittle it is." Identifying materials by
their surface structure eliminates these issues since even similar types of
plastic have different surfaces structures.

Democratizing material knowledge: One way to help novices identify materials
is to ensure sheets are labeled at all times. However, in practice, this is
infeasible to do for all sheets. One expert we interviewed, a manager of a
large workshop, said that "there is no way to keep track of all the sheets
[as] so many people contribute to the scrap piles." Another option if sheets
are unlabeled is that novice makers ask an experienced maker which type of
material it is. However, Annett et al. [4] report that makers with “knowledge
[of] material [were] often difficult to access” and that users need
“intelligent sensing [of] materials.” Hudson et al. [65] show that “early in
the casual makers’ learning process motivation appeared to be very fragile”
and “early failures [can] result in them completely giving up.” A smart system
that provides access to reliable material identification would eliminate this
issue, thereby lowering the entry barrier to laser cutting and democratizing
its use.

Automating mundane work: Laser cutting requires several steps that are mundane
and would benefit from being automated. For instance, in today’s workflow,
users have to identify the sheet, select the correct material type from the
material database, and then verify the power/speed settings. Yildirim et al.
[160] found that professional users want "automated [fabrication tools] that
could pick up menial work, [e.g.] registering materials." They "find it
frustrating when they have to monitor an autonomous [tool]." A material-aware
sensing platform can remove the tedious overhead and allow users to focus on
the essential work.

Enhancing safety of all users: Laser cutting poses both safety and health
hazards [58, 85]. In our interviews, all experts reported that they
experienced multiple fires in the laser cutter at their workspaces. One of
them said that "all of the places [he has] worked at had a fire" and that it
is a "huge safety risk." Concerning health, one of our interviewees, a class
instructor, said someone almost cut a hazardous material that includes
chlorine, which would release toxic fumes and corrode the machine. In
addition, not adhering to the rules would have "revoked all class
participants’ access to the workshop." For safety-critical tasks, HCI
researchers have looked into designing interfaces where the role of users is
“mediated by computer technology” [12, 100]. For fabrication tools
specifically, Knibbe et al. [72] found the “implementation [of] safety alerts
could provide significant benefits within group makerspaces.” A smart sensing
platform can provide such safety alerts and prevent human error by determining
if hazardous materials are used or when users accidentally select wrong laser
settings, which can cause a fire.

###  3.3 Speckle Sensing Hardware Add-On for Laser Cutters

In this section, we first discuss the working principle behind laser speckle
imaging. We will then show how we built a sensing add-on that can be mounted
onto existing laser cutters and highlight the technical contributions of our
add-on over prior work.

####  3.3.1 Speckle Sensing Working Principle

Figure 3.3a illustrates how laser speckle sensing works. It uses a coherent
light source, i.e., a laser, to create the speckles and an image sensor for
capturing them. To create the speckle pattern, the laser light reflects off
the material surface, resulting in a reflectance pattern (speckle) of bright
and dark spots that looks different depending on the material’s surface
structure. This occurs because the tiny features of the material surface lead
to small deviations in the optical path of the reflected laser beam. To show
this, we provide additional electron microscope images of different materials
in Figure 3.3b. Although the materials look visually similar to the human eye,
the electron microscope images clearly show different surface structures,
resulting in different speckle images that can be used for material
identification.

Figure 3.3: Speckle sensing. (a) Laser rays reflect off the material surface
and arrive at the image sensor. Phase differences between the rays result in
mutual interference and thus dark or bright pixels in the captured image. (b)
Different materials viewed by a regular camera, a scanning electron
microscope, and our speckle sensing imaging setup. (c) Our speckle sensing
add-on consists of a laser pointer, lensless image sensor, microprocessor, and
battery.

####  3.3.2 Hardware Add-On

To integrate speckle sensing into an existing laser cutter, we consider (1)
which light source and (2) image sensor to use, as well as (3) how to mount
all required components on the platform. We provide additional specifications
for each component in the appendix.

Laser Pointer: Our initial idea was to utilize the laser pointer present in
our laser cutter (model: Universal Laser Systems (ULS) PLS6.150D), which
conventionally serves as a guide to align the material sheet. However, we
found that laser pointers need to be sufficiently powerful to create speckles
detectable by the image sensor. For this reason, green laser pointers work
best given equal power since most commercial cameras have a Bayer mask with
twice as many green elements as red or blue. Unfortunately, our laser cutter
has a red laser pointer with a power of <1 mW, which according to our
experiments was not sufficient to create detectable speckle patterns. We
therefore decided to use an additional green laser pointer (515nm, <5mW). In
the future, this additional laser pointer may not be necessary if
manufacturers increase the power of their existing laser pointers.

Image Sensor: Our goal was to choose a sensing setup that is compact, i.e.,
uses as few components as possible, yet provides sufficiently high resolution
to detect the speckles. When surveying the related work, we found that
existing setups consist of multiple sensors and/or LEDs [52, 119]. Each
component in these setups helps acquire a unique datapoint related to high-
level statistics, such as the average brightness or overall spectral
reflectivity, which are then input into a classifier as 1D data. We found that
we can reduce the hardware complexity over prior work to only a single image
sensor if feed the raw image, in which the 2D spatial data correlates with the
materials’ surface structure, directly into a convolutional deep neural
network (see Section 3.5.3). Although the 2D image input requires additional
time to compute the prediction result compared to 1D data, it does cause not a
disruption to the laser cutting workflow (0.21s on a 2GHz Intel Core i5
processor).

For the image sensor, we chose an 8MP module (model: Raspberry Pi [99]. As
explained previously, this image sensor, like most commercial ones, has a
higher sensitivity to the green region of the spectrum [128], which is
beneficial for capturing speckle images created by our green laser pointer. We
placed the laser pointer and image sensor as close as possible so that the
speckles’ intensity caused by the laser illumination is high enough when
captured by the image sensor.

Before mounting the image sensor, we removed the lens of the camera module
using a lens focus adjustment tool. We did this because the laser pointer only
illuminates a tiny area on the material sheet (i.e., size of the laser spot).
When imaging the sheet with the off-the-shelf camera module that has the lens
attached, the speckle is present in only a small portion of the entire image
because the lens directs not just the laser light, but all the available light
rays in the scene onto the image sensor. This gives us less speckle data to
work with. When removing the lens from the camera module, however, it is
mainly just the reflected laser light that hits the sensor, causing the
speckle pattern to appear across the entire image (last column in Figure
3.3b). Thus, the camera module with no lens utilizes all the pixels of the
bare image sensor and can capture a higher-resolution pattern.

Microprocessor and Battery: Since commercial laser cutters are closed source,
we had to add a small and lightweight microprocessor and an external battery
pack to allow our add-on to capture images. Since the microprocessor has
limited computational capacity, we send the captured images wirelessly to a
computer for further processing. To make speckle sensing available in future
laser cutters, manufacturers do not need to add these components since laser
cutters already include processing hardware and a power supply.

Mounting on Laser Cutter: To make our hardware add-on compact and easy to use,
we designed and 3D printed the lightweight mechanical housing (60g) shown in
Figure 3.3c. The housing snaps onto the laser cutter’s head and can be mounted
with a small rod. Attaching the sensing hardware to the laser head allows us
to avoid additional calibration since because of the mount, the add-on is
always located at a fixed offset from the laser. All together, the add-on
weighs 114g.

In summary, our speckle-based hardware add-on consists of a laser pointer, a
lensless image sensor, a microprocessor, and a battery pack. However, to
integrate speckle sensing into future laser cutters, manufacturers only need
to add the lensless image sensor – all other components (power and computing
infrastructure, laser pointer) already exist in laser cutters.

###  3.4 User Interface and Applications

In this section, we describe our custom user interface (UI) that integrates
laser speckle sensing into the laser cutting pipeline. In particular, we show
how it helps users identify the material of a single sheet or multiple sheets
at once, and supports users with cutting or engraving multi-material objects.
Additionally, SensiCut can offer safety warnings, provide extra information on
materials, and help with kerf-related geometry adjustments. We also describe
how we use the interface and SensiCut’s material sensing capabilities for
different applications.

Similar to the traditional laser cutting pipeline, users first start by
loading their design (i.e., an SVG file) into the SensiCut UI, which
subsequently shows it on the canvas (Figure 3.4a). Next, users place the
material sheet they intend to use inside the laser cutter.

Figure 3.4: SensiCut UI. (a) The Pinpoint tool allows users to identify the
material at a desired location on the cutting bed. (b) The user enters the
thickness value corresponding to the detected sheet and starts cutting.

####  3.4.1 Identifying Single Material Sheets

To identify a single material sheet, users point SensiCut’s laser pointer to a
desired location on the sheet as shown in Figure 3.4a. They can do this by
first choosing the Pinpoint tool (arrow 1) and clicking the point on the
canvas that corresponds to the physical location in the laser cutter (arrow
2). SensiCut then moves the laser over this location to capture the speckle
pattern and classify the material. The resulting material name is then shown
to the user (arrow 3).

After the user clicks Continue, the shapes in the canvas are color-coded to
reflect the material type (e.g., red corresponds to Cast Acrylic). The user
then enters the material thickness in the text field next to the
classification result (Figure 3.4b). Once ready for laser cutting, they can
hit the Start button and SensiCut automatically retrieves the appropriate
laser power, speed, and pulse per inch (PPI) settings from the material
settings database.

Using the identified material type, SensiCut can further support users via
different functionalities:

Toxic and Flammable Material Warnings: As mentioned in the introduction, there
are many materials that should not be laser cut because they are toxic,
flammable, and/or harmful to the machine. Based on the identified material
type, SensiCut displays a warning whenever the user requests to cut a material
that is hazardous and should not be used in the laser cutter.

Showing Material-Relevant Information: Even though some materials appear
similar, they may exhibit different characteristics, which novice users may
not be aware of. To address this, SensiCut displays additional information on
each detected material to inform users about general characteristics of the
material, ideal uses with sample pictures, and handling/care instructions. We
referred to the laser cutting service Ponoko [67] to retrieve this
information. Workshop managers can edit and extend this information depending
on the workshop type and its users (architecture vs. engineering).

Kerf Adjustments: Design files can have details that are too intricate for
certain material types, especially when the sheets are thin. Cutting these
fine geometries can fuse details together because of kerf. The kerf, i.e., the
amount of the material removed due to the laser, depends on the type of the
material [112, 114]. When details are affected by kerf, SensiCut shows a
warning to the user and then offers three options to address it: SensiCut can
either slightly enlarge the design based on the material type, smooth out too
intricate details, or ask the user to adjust the file manually in the drawing
editor (e.g., Adobe Illustrator).

Application: Fabricating a Face Shield From Different Unlabeled Scraps: In
this application example, we would like to fabricate a face shield and use
transparent plastic materials to ensure clear sight while wearing it. We start
by surveying different designs online and after deciding on one [35], we
download the parts, which consist of a visor and a shield. The design
instructions highlight the importance of using the correct material for each
part. In particular, it is recommended to use a transparent rigid material for
the visor (e.g., acrylic) and a transparent flexible material for the face
shield (e.g., acetate or PETG).

Figure 3.5: Making a face shield. (a) Find flexible sheets from the stockpiles
and (b) place in the cutter. (c) Upon identification, the UI labels all 3
materials and gives relevant information on, e.g., their handling and safety.
(d) After the cut parts are assembled, the shield can be safely sanitized with
alcohol.

We start by browsing through the leftover scrap materials from our workshop.
However, while going through the transparent scrap materials, we notice that
almost all of them are unlabeled. To make the visor, we take the first rigid
transparent material we find in the pile that has a sufficient size and place
it inside the laser cutter. We then open SensiCut’s UI (Figure 3.4a) and
select Pinpoint to identify the material at a desired location. The right-hand
bar shows the material has been detected as cast acrylic, which is a suitable
rigid material for the visor. Once we confirm, SensiCut automatically
retrieves the appropriate power/speed settings for this job and the laser
cutter starts cutting the acrylic.

We repeat the procedure for the shield, which needs to be made from a flexible
transparent material. We go back to the scrap materials and find three
different flexible sheets (Figure 3.5a). We read online that acetate and PETG
may be more suitable than other plastics, but are not sure which one is which.
We take all three and place them in the laser cutter. In the UI, we then
choose Identify sheets and set the number of sheets to 3. Next, we click on
one point on each sheet to instruct SensiCut to identify the material there.
Once the results are displayed, we realize that one of them is a polycarbonate
(Lexan), which SensiCut labels with a "hazardous" warning (Figure 3.5c). The
other two sheets are identified as acetate and (thin) cast acrylic.

To learn more about the difference between the two materials, SensiCut shows
information from its knowledge database and displays it on the corresponding
material. For example, it shows that acetate has high impact strength and is
reasonably flexible, and that, in contrast to acrylic, it can be wiped down
with alcohol, which is important to disinfect the shield. We remove the Lexan
and acrylic sheets from the laser cutter, choose acetate in the UI, and start
cutting. Now that all parts are cut, we can assemble the final face shield
(Figure 3.5d).

####  3.4.2 Identifying Multiple Sheets of Different Materials at Once

SensiCut also allows users to cut multiple sheets of different material types
in rapid succession. The user first loads the design files that contain the
shapes they want to cut from the different sheets. Next, they place the
corresponding material sheets inside the laser cutter and initiate the Scan
multiple shapes mode. This causes SensiCut to go to the location of each shape
and capture an image there for material identification. The resulting material
names are then displayed in the Material Detection sidebar and the shapes are
similarly color-coded based on the material types.

Application: Rapid Testing of Multiple Material Types for Product Design:  In
our second example, we want to rapidly prototype a new earring design in a
white color. We want to fabricate the earrings to test the look and feel of
different materials to determine which one looks best when worn. To evaluate
different material types, we pick a handful of white samples from a material
swatch (Figure 3.6a).

Figure 3.6: Rapid material testing for product design. (a) Choosing multiple
samples from a material swatch and (b) inserting in the laser cutter. (c)
After sensing, the UI matches each shape with the corresponding material. It
also warns that the kerf for felt will compromise the design.

To speed up our prototyping process, we want to cut all the material samples
at once. To do this, we place all our selected material samples on the laser
cutter bed as shown in Figure 3.6b. We then load the earring design. Next, we
position a copy of the earring design in the UI in the location where each
material sample is placed. Next, we choose the Scan multiple shapes option.
After the scan, each earring’s shape is color-coded to reflect the detected
material type: felt, foam board, cast acrylic, and leather (Figure 3.6c).

Figure 3.7: Adjusting the design for kerf. (a) The user can enlarge the shape
to compensate for the thicker kerf for felt. (b) Without the adjustment,
details like the hole and fine blades of the leaf disappear in the cut felt.
(c) The fabricated prototypes.

Next, we enter the thickness for each material sheet: 1mm for felt and
leather, and 3mm for the others. For felt, SensiCut shows a notification that
our design has details that are too intricate for a thickness of 1mm and may
thus fuse together because of kerf (Figure 3.7b). We choose the Make shape
bigger option and enlarge the felt earring so that the minimum feature size no
longer goes below the kerf limit (Figure 3.7a). Once adjusted, we cut and
engrave all sheets in a single job. The finished earring prototypes are shown
in Figure 3.7c.

####  3.4.3 Engraving onto Multi-Material Objects

Compared to individual material sheets, cutting or engraving designs onto
multi-material objects (e.g., the smartphone case in Figure 3.8a) is a
particularly challenging task. It requires a cumbersome workflow where users
first have to split the design into multiple files, one for each material.
More specifically, proper alignment of the shapes in the digital design with
the different parts of the physical object is challenging without knowing
where the material borders are located.

SensiCut facilitates cutting and engraving on multi-material objects by
automatically splitting the design precisely along the border of different
materials by sensing the material type at each point in the design. For this,
users start by loading a single file containing the entire design, insert the
multi-material object into the laser cutter, position the design onto the
multi-material object, and select the Scan multi-material objects mode.
SensiCut then samples points along the laser-cut path to identify the material
at each point. The scanning progress is shown in the SensiCut UI by
highlighting the scanned trajectory. After scanning is completed, SensiCut
splits the design according to the detected material type at each point to
ensure the correct laser settings will be used.

Application: Personalizing Existing Multi-Material Products: In this example,
we want to engrave a custom design at the center of a smartphone case that
consists of two different materials across its surface, i.e., leather and
wooden parts (Figure 3.8a).

Figure 3.8: Engraving a multi-material phone case, which consists of (a) wood-
like rubber and leather. SensiCut scans (b) the input design’s outline and (c)
splits it into two parts based on the material type detection. (d) Engraved
design.

First, we load the design file, position it on top of the phone case, and
select Scan multi-material objects, which then moves the laser head along the
design’s engraving path to detect the material at each point (Figure 3.8b).
Once the scan is complete, SensiCut splits the design into two parts, one for
each of the two materials (Figure 3.8c).

Once SensiCut identified the materials, we realize that the part we had
thought was wood is actually made of silicone rubber with a decorative wood
pattern. SensiCut is not deceived by the disguise pattern because it measures
surface structure and sets the correct laser engraving settings. Once we
confirm, our design is engraved onto our multi-material phone case (Figure
3.8c).

Application: Customizing Multi-Material Garments: Figure 3.9a shows another
multi-material item, i.e. a T-shirt, that we want to engrave with a custom
seagull design. The T-shirt has a plastic iron-on material applied on it. To
engrave our design, SensiCut detects which parts are made of textile and which
are made of plastic. It then splits the seagull design into multiple paths
accordingly and assigns the correct laser power/speed settings for each one
(Figure 3.9b). If we had instead used only one set of laser power/speed
settings for the entire seagull design, i.e., the settings for either textile
or plastic, the lines would either not be visible on the yellow plastic or the
textile would have been burned. Further, it would be particularly difficult to
achieve this without SensiCut: One would have to remove the iron-on plastic
from the fabric itself, engrave the plastic and fabric separately, and put
them back together precisely. This shows how SensiCut could help users further
customize garments that have non-textile parts (e.g., [40]) quickly and on
demand.

Figure 3.9: Engraving a pattern on a T-shirt that has (a) plastic details on
it. (b) SensiCut uses the right combination of laser settings after
partitioning the design (middle). Top/bottom shows the outcome for singular
settings.

###  3.5 Classification of Materials

SensiCut can differentiate between 30 different materials relevant to the
challenges laser cutter users face. In the next section, we discuss how we
built a dataset of speckle patterns of these materials using an automated
script, and how we trained a convolutional neural network (CNN) to be able to
distinguish between them.

####  3.5.1 Choosing Material Samples

For our dataset, our goal was to choose materials that are most representative
of the materials commonly found in makerspaces and workshops, with a
particular focus on the ones that cause confusion because of their
appearances. Figure 3.10a summarizes the list of materials we compiled by
surveying a range of online communities (e.g., Thingiverse [90], Instructables
[6]), educational materials on laser cutting [7], supply vendors [67, 77], as
well as the laser cutter material databases that come with the default laser
cutter control software (e.g., ULS Universal Control Panel [135]).

The resulting material list includes 30 different materials ranging from
different types of paper, plastic, wood, fabric to (engraved) metal. In the
next section, we discuss how these selected materials are representative of
the challenges that laser cutter users face.

Different Laser Cutting Materials with Similar Appearance: As our formative
material labeling study showed, plastics are particularly challenging to
distinguish for users due to their visual similarity (Figure 3.10b). To
represent such cases, we purchased samples of cast acrylic, extruded acrylic,
and Delrin (also known as acetal or POM) of the same color, which require
different settings to properly cut/engrave a design [7]. We also included
samples of different transparent sheets, i.e., acrylic, PETG, and acetate. In
addition, we included other materials that have slightly varying appearances,
but still are difficult to distinguish for non-expert users who are not
familiar with the specific nuances, such as different types of wood (e.g.,
maple, oak, bamboo, or birch) [152].

Hazardous Materials that Look Similar to Safe Ones: To represent cases where
some of the commonly found materials in workshops are hazardous (flammable,
toxic, or harmful to the machine) and cannot be safely laser cut [7, 73], we
included polyvinyl chloride (PVC), Lexan (polycarbonate), acrylonitrile
butadiene styrene (ABS), and carbon fiber sheets. PVC is often mistaken for
the common laser-cut material acrylic. However, it is highly toxic as it
releases hydrochloric acid fumes when heated, which also rapidly corrode the
laser system [58]. Lexan and ABS are also hazardous and easily
flammable111https://wiki.aalto.fi/display/AF/Laser+Cutter+Materials but look
similar to safe plastics. However, whether a material is considered safe for
laser cutting or not depends on the specific hardware setup (air filter type
and volume, power of laser) as well as local regulations [49]. For our setup,
materials in the ULS material database that comes with our laser cutter and
its UAC 2000 filter (MERV 14, HEPA, 2 Carbon filters) are marked as safe. For
instance, polystyrene is listed as safe for our setup but may not be safe for
others. Thus, we recommend that when deployed in a new workshop, SensiCut’s
database be updated by the workshop manager locally after checking material
safety data sheets (MSDS) for potential laser generated air contaminants
(LGAC). Workshop managers should also talk to their local occupational health
institution (e.g., NIOSH222https://www.cdc.gov/niosh/ in the US).

To make the material composition of our dataset representative of a real-world
workshop, where certain materials like acrylic and cardboard are much more
available in terms of quantity/color options, we included more than 1 sheet
for these as seen in Figure 3.10a. This also allows us to evaluate our system
for different colors and transparencies. In total, we used 59 material
samples, the majority of which were purchased from Ponoko [67], except for the
5 hazardous material sheets (PVC, Lexan, etc.), which we purchased from other
suppliers on Amazon.com. A list of these material samples and the associated
vendors can be found in the appendix.

####  3.5.2 Data Capture and Material Speckle Dataset

After purchasing the different materials, we captured images of each sample to
build a dataset for training our convolutional neural network.

Preliminary Experiment: Before capturing data for all materials, we ran a
preliminary experiment to determine two values: (1) the distance between the
image sensor and the material surface at which the speckle pattern is most
visible, and (2) the number of images necessary for training the classifier
with high accuracy. For the distance, we empirically found that 11cm between
our image sensor and the material surface led to the best results. For the
number of images, we placed material samples below the image sensor at the
recommended distance and took images, moving the sample in the xy plane
manually to simulate how the laser cutter would take images at various points
of the sheet. We found empirically that around 80-100 images are sufficient
for each material to train a CNN for classification.

Data Collection: After this manual exploration, we started the data collection
of all materials. For this, we wrote a script to automate the laser cutter’s
movement and image capture. For our material samples (6.3cm x 6.3cm), we chose
to capture a 9x9 grid of points leading to 81 images, which satisfies our
criteria from the preliminary experiment. For consistency, we kept the image
sensor settings, i.e., exposure time, digital/analog gains, and white balance
constant.

Additionally, we captured images at different heights (z-locations) to ensure
that the network can classify materials of different thicknesses. This is
necessary since the speckle pattern changes with the distance between the
material surface and the image sensor. We chose 8 different heights ranging
from 0mm (to support paper) to 7mm (thickest material sheet we were able to
buy) spaced at 1mm increments. However, not every sheet has a thickness of a
multiple of 1mm (e.g., some sheets are 2.5mm). We can generate this additional
data using data augmentation methods as explained in Section 3.5.3. Since our
model was trained for materials with a thickness of 0-7mm and the material
surface was 11cm away from the image sensor, this leads to an effective
detection range of 110-117mm. To integrate material identification into other
cutting-based methods like LaserOrigami [96] or FoldTronics [158], the model
can be trained for larger distances in the future.

Dataset: Our final data set contains 38,232 images from 59 material samples of
30 unique materials (14.93 GB, 800x800 pixels each). Each material sample
includes 648 images (81 images/height x 8 heights), which took about 40
minutes to capture with our automated setup. The majority of this time is
spent waiting for the laser head to stabilize after moving to a new location
to ensure that the captured image is not blurry. The dataset is used for
training the CNN and does not need to be stored on the user’s computer. The
trained CNN model that is used at detection time is 120MB. The dataset is
publicly
available333https://hcie.csail.mit.edu/research/sensicut/sensicut.html.

####  3.5.3 Training the Neural Network

To train the CNN and build a detection model using the captured images, we
used transfer learning with a ResNet-50 model [55] that was pre-trained on the
ImageNet dataset [115]. We used the Adam optimizer with a learning rate of
0.003 and a batch size of 64. We used 80% of images for the training set and
reserved 20% for the validation set.

Image Size Used for Training: For the input image size, we chose 256x256
pixels. Although we captured the images in 800x800 pixels, we found that the
higher resolution caused lower accuracy as the model overfit to irrelevant
details in the image. The lower resolution input also saves training time
because the model has fewer nodes to compute. Moreover, it speeds up the
detection during use (i.e., average prediction time: 0.21s for 256x256px vs.
0.51s for 400x400px on a 2GHz Intel Core i5). We still keep the full-size
images in our dataset to enable future research.

Data Augmentation: To make the model robust to different lighting conditions
and intermediate sheet thicknesses (e.g., 2.5mm), we generated additional
images during training using data augmentation. Every time the network starts
training on a new batch of images, a portion of the images is transformed by
changing the brightness and the contrast of all pixels (by up to $\pm$30%), as
well as zooming into the image to enlarge the speckles as would be the case
when the thickness of the sheet decreases (by up to $\pm$20%). This allows our
model to generalize better and also saves time by avoiding the capture of more
images with the physical setup.

In the future, new materials can be added to SensiCut by capturing more
speckle images and adding them to the dataset. For this, the neural network
needs to be retrained but the weights from this previous training can be used
(transfer learning), which significantly speeds up the process , i.e., takes
only 10-12 minutes vs. 6 hours training from scratch.

Figure 3.10: Material considerations and evaluation. (a) Most common material
types for laser cutting. (b) Visually similar materials. (c) Confusion matrix
from our trained classifier for 30 material subtypes.

###  3.6 Evaluation of Material Classification

We conducted a technical evaluation to determine our trained classifier’s
accuracy. We also carried out additional tests to understand how the model
generalizes to different physical conditions (rotation of sheets, illumination
variations) and material sheets purchased from different vendors.

####  3.6.1 Detection Accuracy Results

The results of the classification accuracy for the 30 different materials in
our dataset are shown as a confusion matrix in Figure 3.10c. Our average
identification accuracy is 98.01% (SD=0.20) across the different materials.
This is based on a 5-fold cross-validation, which we ran to ensure consistency
of the classification accuracy across different training and validation
splits. The small standard deviation shows that training our model leads to
similar results independent of how the dataset is split. For this reason and
the fact that cross-validation is a time-consuming procedure (30 hours for
5-fold), the remainder of the technical evaluation is based on a single run.

We further analyzed the results to understand which of the materials outlined
in Section 3.5.1 are confused for each other most. For instance, given a
specific color (either white, black, or red), we evaluated the identification
accuracy across cast acrylic, felt, paper, and laminated MDF. The accuracy was
100% for white and red, and 92% (SD=12.72) for black, on average. The latter
is likely due to the fact that black reflects less light. Since the image
sensor’s exposure is the same for all photos, this causes the reduced accuracy
for black materials. Enabling adaptive exposure when capturing images could
eliminate this difference in the future [159].

Figure 3.10c also shows materials that were mistaken for each other. For
instance, leather and silicone were confused with each other at a relatively
high rate compared to other pairs. We believe this is because our full-grain
leather piece is hot stuffed, i.e., conditioned with unrefined oils and
greases, which likely makes its surface structure closer to that of silicone.
One can also observe some confusion between walnut and paper-based materials,
such as cardstock. Thesimilarity in their surfaces may be due to the fact that
paper is produced using cellulose fibres derived from wood.

We also evaluated the accuracy of materials within the same material groups.
We got a mean accuracy of 98.92% across woods (SD=1.66), 98.84% across
plastics (SD=2.36), 97.25% across textiles (SD=2.50), 95.90% (SD=2.94) across
paper-based materials, and 97.00% (SD=2.16) across metals. The fact that
paper-based sheets had the lowest rate is expected as they share the most
similarities in their surface structures among different subtypes (e.g.,
cardstock vs. cardboard have a similar surface texture).

####  3.6.2 Effect of Illumination and Sheet Orientation on Detection

To understand how detection accuracy is affected under different illumination
(ambient light) conditions and sheet orientations, we ran additional tests.

Figure 3.11: Effect of (a) varying illumination and (b) sheet orientation on
detection.

Ambient Light: When we captured the images for our main dataset, we kept
illumination in the workshop low (i.e., all lights turned off). To evaluate if
the trained model can distinguish between materials even when the ambient
light varies, we created an additional test set of images under different
lighting conditions. For this, we used two lamps, one on the left and one on
the right corner of the room, resulting in three conditions (light1 on, light2
on, both lights on) that cover an illumination range of up to 80 lux, in
addition to the initial data with all lights off. We tested this on different
black and white sheets, representing the two ends of the light reflectance
spectrum, as well as clear (transparent) sheets. We compare the accuracy in
the following three scenarios: 8 white and 8 black sheets (for each color 2
sheets per type: plastic, paper, textile, wood), and 6 transparent sheets (all
plastic).

The results are shown in Figure 3.11a. We found that the increased brightness
did not have a major impact on the detection of white and black sheets. For
clear sheets, however, the mean accuracy was lower (66.67%). The reason for
this is that while opaque sheets benefit from data augmentation, this is not
the case for clear sheets. We found that the illumination increase in the room
was not realistically simulated in the digitally generated images of clear
sheets because such materials allow light to pass through in all directions.
This can be overcome by capturing additional images of clear materials under
the varying light conditions and then retraining the model. Indeed, such
retraining resulted in an increased accuracy of 88.10% (shown in the last
bar). We also found that retraining the model on this augmented dataset did
not have a major impact on other materials’ detection (only by 0.41% on
average).

Orientation of the Sheets: The images we captured for our main dataset were
all taken in a specific sheet orientation. We thus evaluated if the classifier
is still accurate when the sheets are arbitrarily rotated for materials with
uniform (e.g., acrylic) or non-uniform/irregular surface structure (e.g.,
wood). For this, we created an additional test set by capturing speckle images
while rotating the material sheets at 45∘ increments. For materials with
uniform surface (plastic, textile, paper, metal), we picked two subtypes each
(cast acrylic, Delrin, cardboard, matboard, felt, leather, aluminium, carbon
steel). For materials with non-uniform surface, we tested eight subtypes of
wood (oak, maple, walnut, birch, MDF, veneer, bamboo, laminated).

The results are shown in Figure 3.11b. The lowest average detection accuracy
was for wood sheets (70.31%), which also had a high standard deviation among
the wood subtypes (24.94%). This is due to the fact that wood sheets included
both artificial ones with regular surface structure (e.g., MDF), which
resulted in 100% detection accuracy, and natural woods with irregular surface
structure (e.g., oak), which resulted in lower accuracies. The misidentified
images for those materials were all captured at the odd degrees (45∘, 135∘,
etc.). We believe this is due to the cellular 3D microstructure of natural
wood that has a 90∘ rotational symmetry at the microscopic level [5]. We can
increase detection accuracy for natural woods by augmenting the training
dataset with more pictures taken at different angles, at the expense of longer
capture time.

####  3.6.3 Generalization to Different Material Batches and Manufacturers

In our main dataset, each set of samples came from one manufacturer. To ensure
that our trained model can work robustly for sheets from different batches of
the same manufacturer or different manufacturers, we conducted the following
two tests.

New sheets from the same manufacturer: Two months after we purchased our
samples, we ordered a second batch of sheets from Ponoko (two subtypes per
material: oak, maple, cast acrylic, Delrin, felt, leather, cardboard,
cardstock) and placed them inside the laser cutter to test if our trained
model can still identify them. We found that only the maple sheet was
incorrectly classified. As explained in Section 3.6.2, this is likely due to
unique microstructural orientation of natural wooden sheets.

New sheets from different manufacturers: To test if sheets from different
manufacturers can be reliably identified, we ordered 8 different sheets from
various vendors on Amazon.com (two subtypes per material: birch, cork, cast
acrylic, Lexan, leather, felt, cardboard, matboard). Only one of them,
leather, was incorrectly classified. We later found out that the new sample
was synthetic leather, whereas the classifier was trained on natural leather.

While the above evaluation demonstrated that our classification model can
detect material types across various conditions, more longitudinal tests are
needed to further verify its applicability across various workshop settings.

###  3.7 Software Implementation

Our user interface is Web-based and implemented using JavaScript and the
Paper.js library. When users request a material identification, our system
automatically moves the laser cutter head to the corresponding xy-coordinates
on the physical cutting bed. These coordinates are offset by the distance
between SensiCut’s laser pointer and the cutting laser. For the z-value, a
fixed distance to the sheet is used for capturing the speckles (Section
3.5.2). To input the coordinates and initiate the movement, our system uses
the PyAutoGUI library to interface with the ULS Universal Control Panel (UCP).
It then detects if the laser head stopped moving, i.e., is stable enough to
take a picture, by checking if the laser is in idle mode. This is indicated
via a color change in UCP, which our system can detect via PyAutoGUI’s
screenshot() and getcolors() functions.

Next, the captured image is wirelessly sent from the hardware add-on’s
Raspberry Pi board to the main server, which runs on an external computer.
This Python server uses the image as input to the trained CNN model, which was
implemented using PyTorch and fast.ai [61]. The CNN returns the classification
results, which are then displayed in the UI. The communication between the
JavaScript front-end and the Python back-end is handled by the Socket.IO
framework. The complete pipeline is shown in Figure 3.12.

Figure 3.12: Our detection pipeline takes the user’s drawing as input and
turns it into target points to capture speckle patterns. The captured images
at those points are passed to the CNN to retrieve the material label.

After the user confirms the results, the laser power, speed, and PPI (pulses
per inch) settings are retrieved from the ULS database based on the detected
material. Because the UCP interface does not have an API, we extracted these
values from the its back-end using a Firebird server and Database Workbench 5
Pro to create an interim datasheet from which we can look up values as needed.

To detect kerf-related issues, we first dilate the drawing with a kernel of
the size of the material-specific kerf. We check if two curves overlap or if
the dilation results in extra blobs, i.e., the cut may lead to an undesired
shape. The materials’ kerf values are based on the "Minimum feature size"
values listed on Ponoko.

For the multi-material object mode, our system samples points uniformly along
the cutting path and processes the captured images according to the pipeline
described previously. Our system then assigns the respective identified
material to each part of the users’ drawings.

###  3.8 Discussion

In the next section, we discuss insights gained from our work, acknowledge the
limitations of our approach, and propose future research.

Avoiding Dust on the Sensor: In conventional cameras, lenses help prevent dust
particles from landing on the sensor. Although we use a lensless image sensor
to capture the speckle patterns, over the course of our research we did not
observe the lack of the lens to interfere with classification results. We
hypothesize that this is the case because (1) the sensor is facing down, which
prevents dust particles from reaching the surface of the sensor due to
gravity, and (2) the ventilation in the laser cutter bed sucks away particles
from the image sensor.

Confidence Scores for Misidentified and/or Unknown Materials: The neural
network’s final layer outputs a vector for the confidence score of each
material type. If multiple types have similar scores, the material is either
misclassified or not included in the original training dataset. As part of our
future work, we will extend the user interface to show bars to visualize the
confidence scores and inform the user to act with caution when confidence
scores of multiple materials are similar. Additionally, a confidence threshold
for when a material is safely classified could be set by the workshop manager
for all users of the workshop.

Effect of Scratches on Sheets: Scratches on sheets are often local, i.e., they
occur when a sheet’s sharp corner abrades a spot on another sheet’s surface.
We picked 2 cast acrylic, 2 birch, 2 cardboard sheets with the most scratches
from the material pile in our workshop and captured speckle patterns at 30
uniformly distributed points across the surface of each material sample. We
found that the majority of points were correctly classified, i.e. 90% for
acrylic, 91.7% for birch, 86.7% for cardboard. In future work, to make
material detection robust to local scratches, SensiCut could take more than
one image and cross-check the classification result at the expense of longer
detection times.

Materials with Protective Cover: Some material sheets come with a protective
plastic/paper cover to avoid scratches during transportation and some users
may prefer to leave it on during laser cutting. Since SensiCut needs access to
the material’s surface, users can peel a small section from the corner and use
our interface’s Pinpoint function to detect the material type from that
corner.

Estimating the Sheet Thickness from Speckles: As the distance between the
image sensor and the material surface increases, the speckles appear larger in
the image [127]. If the pictures are taken at a fixed height (i.e., a fixed
distance of the laser head to the cutting bed), then the surface of thicker
sheets is closer to the laser head, which results in smaller speckles, while
thinner sheets are further away from the laser head, resulting in larger
speckles. We tested if this can be used to detect the sheet thickness by using
the same dataset and CNN structure as the material type classifier (ResNet).
However, as this is a regression problem, we used mean squared error instead
of cross-entropy as our loss function. An initial test across 14 material
sheets gave us a mean error of 0.55mm (SD=0.68mm). For the ULS laser lens we
have, the depth of focus (i.e., tolerance to deviations from the laser’s
focus) is 2.54 mm, which is larger than this detection rate. Thus, for future
versions of SensiCut, we can also include thickness detection.

Labeling Workflows: While some users may prefer to keep material sheets
unlabelled and launch SensiCut every time they use the laser cutter, SensiCut
can also support hybrid workflows, such as printing a sticker tag after
identifying a sheet, which can then subsequently be attached to the material
sheet. Similarly, the software interface could remind users to label the
material sheet with a pen after use as a courtesy to the next maker.

Material Identification for Other Fabrication Tools: For future work, we plan
to explore how SensiCut’s material identification method can be used for other
personal fabrication machines as well. For example, in 3D printing, some
manufacturers, such as Ultimaker, add NFC chips into filament spools to allow
the chip reader integrated in the 3D printer to automatically detect them.
However, not all spools come with such chips. To address this issue, we plan
to investigate how speckle sensing can be integrated into filament feeder
systems to detect the filament type when a new spool is loaded onto the 3D
printer.

###  3.9 Conclusion

In this chapter, we presented SensiCut, a material sensing platform that helps
laser cutter users to identify visually similar materials commonly found in
workshops. We demonstrated how this can be achieved with speckle sensing by
adding a compact and low-cost hardware add-on to existing laser cutters. We
showed how the material type detection can be used to create a user interface
that can warn users of hazardous materials, show material-relevant
information, and suggest kerf adjustments. Our applications demonstrated how
SensiCut can help users identify unlabeled sheets, test various materials at
once, and engrave onto multi-material objects. We discussed how we chose the
materials in our dataset and how we trained the convolutional neural network
for their classification. We reported on the detection accuracy for different
material types and evaluated the impact of varying the room illumination,
rotating the sheets, and using sheets purchased from different manufacturers.
We then highlighted how our system can be extended to also detect the
thickness of sheets. For future work, we plan to investigate how speckle
sensing can be used to detect materials in other fabrication tools.
Furthermore, we plan to collaborate with laser cutter manufacturers to
integrate our material sensing approach into future commercial products, which
only requires adding the lensless image sensor and adjusting the power of the
existing visible laser pointer.

##  Chapter 4 StructCodes: Leveraging Fabrication Artifacts to Store Data in
Laser-Cut Objects

###  4.1 Introduction

Figure 4.1: StructCode embeds data in the fabrication artifacts of laser-cut
objects, such as the patterns of (a) finger joints and (b) living hinges, to
augment objects with data. Here, the embedded StructCodes allow narration for
a painting and status updates for a potted plant, among others.

To achieve unobtrusive passive tags that do not require object modification,
researchers investigated ways to use the inherent characteristics of objects,
which may manifest in the form of either naturally occurring or engineered
features. As an example of natural unobtrusive tags, SensiCut [23] senses the
micron-scale surface structure of materials using speckle sensing to identify
the material’s type. Similarly, Verifiable Smart Packaging [146] uses radio-
frequency signals that penetrate through an object’s exterior to identify the
object by its internal structure. The second approach, engineered unobtrusive
tags, offers more flexibility to define how the tag should by embedded with
the object. For example, in agriculture, researchers developed tamper-evident
tags using biodegradable silk particles, allowing for seed traceability and
anti-counterfeiting [134].

In this project, the question that prompted our research was: Can we make use
of existing fabrication artifacts of objects to store metadata in them? Our
goal is two-fold: (1) not introducing additional materials, parts, or features
to the object, but simply make use of its existing visual structures that
arise as a byproduct of the fabrication process. (2) The added data should be
decoded from regular camera images so the meta-information can be easily used
to augment the objects, e.g., in augmented reality (AR) applications.

Our proposed solution, StructCode, slightly varies fabrication artifacts,
i.e., the patterns resulting from joints, which does not change the main
object geometry, but can still be detected from camera images. Thus,
StructCode does not add any new features to the object, but instead exploits
the structures that are inevitable artifacts of the fabrication process. While
this idea can be applied using many different manufacturing processes and
materials that allow for the customization of individual tangibles (to embed
individual tags), for the scope of this project we focus on laser cutting.
Laser cutting is particularly suitable because it allows the rapid fabrication
of sturdy, functional, and large-scale objects [9, 23]. These typically
contain many visual artifacts across their surfaces due to their finger joints
and living hinges. StructCode subtly changes the widths of these elements to
represent machine-readable information. The StructCodes can be read as part of
existing AR pipelines to augment the user’s view with information relevant to
the tangible objects that they are interacting with.

Figure 4.1 shows how laser-cut objects use their existing joints to store
meta-information, such as labels (e.g., used to retrieve watering updates for
the plant), context (e.g., to access the digital model of the chair),
disassembly instructions (e.g., to recycle the bookshelf), and narration
(e.g., explanatory video for the artwork on the wall). StructCode achieves
this by modifying the lengths of laser-cut finger joints and living hinges,
which are inherent artifacts of commonly laser-cut objects. The modified
lengths represent different bits while ensuring the codes are easy to capture
with a mobile camera. Because StructCodes are an inherent part of the objects,
they cannot be easily removed without causing damage to the object. Compared
to other unobtrusive tagging methods in HCI, which use additional equipment
such as infrared cameras [29, 153], StructCode only requires a conventional
RGB camera for detection, and thus can be used on off-the-shelf mobile devices
and headsets.

While this project demonstrates the idea of opportunistic data embedding for
the example of laser-cut objects, we discuss how it can be generalized to
other types of fabricated objects in the future in Section 4.8. We note that
this chapter was originally published at ACM SCF 2023 [22]. Our contributions
can be listed as follows:

  * •

An unobtrusive and integrated tagging method that embeds data in the existing
fabrication artifacts of objects, as demonstrated for laser cutting, and only
requires a standard camera for detection.

  * •

A tag decoding pipeline that is robust to various backgrounds, viewing angles,
and wood types.

  * •

A set of applications showing how this opportunistic embedding can enrich
tangible interactions with laser-cut objects.

  * •

A mechanical evaluation illustrating that the addition of StructCodes
maintains the overall integrity of objects.

###  4.2 Method: Embedding Data in StructCodes

In this section, we discuss how to physically embed the code into existing
laser-cut structures and which encoding scheme we use for StructCodes. Our
goal is to be able to store custom text (e.g., sentences or website URLs).

####  4.2.1 Identifying Features Suitable for StructCodes

To better understand how to embed information in laser-cut objects’ artifacts,
we surveyed what joints are used in their structure and which of these are
suitable to be leveraged as a StructCode. We considered the top 100 laser
cutting projects on the popular online repository
Instructables111https://www.instructables.com/ and classified the identified
joints in objects. If an object contained more than one joint type, we took
into account all of them.

In total, 58% of projects had at least one type of joint. We found four main
types of joints contained in the laser-cut objects: finger joints (26%),
mortise and tenon joints (16%), slot joints (8%), and living hinges (8%). We
count living hinges as a type of joint because under joints, we consider any
connection between two planes in the laser-cut model. These joint types can
also be commonly found in the research community, e.g., finger joints in
Enclosed [149], CutCAD [57], and Fresh Press Modeler [17], slot joints in
FlatFitFab [93], SketchChair [120], and Planar Pieces [122].

Out of these four joint types, we identify finger joints and living hinges as
most suitable for illustrating the concept of StructCodes. Both contain
repeating elements, i.e., finger joints have individual fingers repeated along
the edges of a plate and living hinges are made up of repeated individual
lines. They are both on the outside surface of the object, i.e., visible to
the camera. Living hinges form a well-defined shape that can be tightly
enclosed by a bounding rectangle. While finger joints can also be applied to
non-rectangular parts, they are most common on rectangular shapes (73.1% of
the joint projects had joints on rectangular plates). Both structures offer
large encoding capacity; particularly, living hinges have many line cuts that
could be used as bits.

####  4.2.2 Structural Embedding of Data Bits

As shown in Figure 4.2, the identified structures typically contain material
interleaved with cuts, i.e., gaps. For instance, in finger joints, the fingers
are interleaved with cut outs that match the fingers of the neighboring plate,
and in living hinges, the sheet is cut at specific distances to make the
material bendable.

To encode a message, we can either vary the length of the material (fingers of
joints or links of hinges) or the length of the gaps between the fingers or
between the links. Below we outline how we embed the codes for each of the two
structural element types.

Figure 4.2: General design of (a) finger joints and (b) living hinges. To
embed bits, we resize both the fingers and gaps in the finger joints, but only
the links in the living hinges.

##### Finger joints

To embed StructCodes into finger joints, we modify both the widths of the
individual fingers and the widths of the gaps between the fingers, as labeled
in Figure 4.2a (i.e., $d_{1},d_{2},d_{3},...$). We did not consider modifying
the depth of the fingers because the depth has to be the same as the sheet
thickness to ensure properly interlocking plates. To maintain the structural
integrity of the joints, modifications to the widths of the elements (i.e.,
fingers or gaps) should be minimized to be as close to the original width as
possible while being distinguishable by the camera and distributing evenly
over the plate length (see Section 4.5.1).

##### Living hinges

There are two ways to embed StructCodes into living hinges, either by
modifying the size of the individual links, the widths of the hinge gaps
(i.e., the vertical cuts between the links), or both as shown in Figure 4.2b.
To ensure that the hinge can be bent into the desired curved configuration
after laser cutting, the links have to be vertically aligned. Therefore, we
only modify the lengths of the links. Furthermore, since the links are
typically shorter than the gaps, they are less prone to warping when comparing
the lengths against each other and thus more suitable for our purposes. This
is because the links are situated in a smaller local region, which allows the
projection to the image plane to maintain the ratio of distances [89]. To
ensure the structural integrity, we chose lengths that are as close to the
original as possible while being distinguishable by the camera (see Section
4.5.2).

####  4.2.3 Encoding Scheme

StructCode employs a base 3 (ternary) encoding scheme222In ternary logic, the
digits are called trits (trinary digit). However, for familiarity, we use the
term bits in this chapter although it is normally used for the binary system.,
i.e., it uses the bits 0, 1, and 2. This is represented in the physical code
as a narrower element ("0"), a medium element ("1"), and a wider element
("2"). Four elements, i.e., four bits, represent one character. The different