# Replicating Human Anatomy with Vision Controlled Jetting –
A Pneumatic Musculoskeletal Hand and Forearm

Thomas Buchner1†, Stefan Weirich1†, Alexander M. Kübler1,
Wojciech Matusik2,3, and Robert K. Katzschmann1∗ 1Soft Robotics Lab, IRIS,
D-MAVT, ETH Zurich, Switzerland2Inkbit, 1 Cabot Rd, Suite 400, Medford 02155
MA, USA3CSAIL, MIT, 32 Vassar St, Cambridge, 02139 MA, USA$\dagger$ Thomas
Buchner and Stefan Weirich are co-first authors.$*$ Corresponding author:
<EMAIL_ADDRESS>
###### Abstract

The functional replication and actuation of complex structures inspired by
nature is a longstanding goal for humanity. Creating such complex structures
combining soft and rigid features and actuating them with artificial muscles
would further our understanding of natural kinematic structures. We printed a
biomimetic hand in a single print process comprised of a rigid skeleton, soft
joint capsules, tendons, and printed touch sensors. We showed it’s actuation
using electric motors. In this work, we expand on this work by adding a
forearm that is also closely modeled after the human anatomy and replacing the
hand’s motors with 22 independently controlled pneumatic artificial muscles
(PAMs). Our thin, high-strain (up to 30.1 %) PAMs match the performance of
state-of-the-art artificial muscles at a lower cost. The system showcases
human-like dexterity with independent finger movements, demonstrating
successful grasping of various objects, ranging from a small, lightweight coin
to a large can of 272 g in weight. The performance evaluation, based on
fingertip and grasping forces along with finger joint range of motion,
highlights the system’s potential.

##  I Introduction

###  I-A Motivation

When we engage in routine activities like sipping from a bottle, composing a
letter, or typing out a document, the intricate movements and manipulations
required come naturally to us. This seamless integration of thought and action
stems from our cognitive faculties and the exceptional dexterity of human
hands [1]. The human hand distinguishes itself through its combination
agility, strength, and speed. Current robotic hands fall behind in those
metrics. The design of robotic and prosthetic hands has been researched for
many decades but still has not reached the levels of dexterity, versatility,
and compactness of their human counterpart [2].

Figure 1: Pneumatic musculoskeletal hand and forearm. Inspired by the human
anatomy, we printed a musculoskeletal hand and forearm. The soft-rigid hybrid
hand is printed in a single process and does not need assembly apart from
attaching thin McKibben muscles for actuation. We replicated the anchor points
and muscle lengths from the biological inspiration.

Previously developed robotic hands are either biomimetic and very complex to
fabricate, or the designs use strong simplifications for ease of fabrication
and cost reduction. Biomimicry and dexterity versus the ease of fabrication
and cost efficiency is a critical trade-off decision in robotic and prosthetic
hand design to date.

Figure 2: Inspiration from human hand and forearm replicated in a 3D print. A
Front view of a 3D bone model of a human arm and hand with accurate muscle
placement. B Front view of a 3D model of the printed hand and forearm. The
bone shapes and positions were extracted from MRI data. The tendons and
actuators replacing the muscles are placed according to their biological
counterparts’ anchor points and positions. C Back view of model shown in sub-
figure A. D Back view of model shown in sub-figure B. The sperical volumes
shown around the joints are our modeled joint capsules/ligaments.

###  I-B Related Work

Biomimetic hand designs [3, 4] mimic human bones, joints, tendons, and
ligaments using tendon-driven mechanisms. While the hands themselves achieve
high levels of biomimicry, the biomimetic aspects in the forearm area are
neglected entirely by using rotating electric motors to actuate the tendons.
Fabrication and assembly of these hands are complex and time-consuming. So
far, no biomimetic robotic hand has been realized in combination with an
equally biomimetic forearm and wrist.

Many robotic hands use non-biomimetic, standard mechanical components, such as
cables, gears, belts, linkages, and parallel mechanisms, or directly integrate
electric motors into the fingers [5, 6, 7, 8]. The electric motors do not
resemble the features of human muscles and constrain the compactness and
compliance of the hand design. The motor’s rigid components can become unsafe
in dynamic, collaborative applications [9]. Recent hand designs place the
electric motors either in the forearm [3, 4, 10] or in the palm [11]. The
motor arrangements in the forearm enable accurate control of complex hands
with a large number of DoF but come with major drawbacks such as the weight,
size, and bulkiness of the design, often limiting possible applications. The
number of pneumatic artificial muscles (PAMs) that can be located in the palm
is limited due to spatial constraints. Commercially available robotic [12] and
prosthetic [13, 14] hands typically use electric motors for actuation and
implement varying design simplifications with conventional mechanical
components. Manipulation research also focuses on unteractuated designs
resulting in reduced dexterity. Catalano et al. [15] designed a five-fingered
robotic hand with only one motor using adaptive synergies.

In soft robotics, robotic hands typically integrate bending PAMs in the
fingers [16, 17, 18]. Functional soft prosthetic hands with bending PAMs [19]
and soft, tendon-driven hands made from foam [20] or 3D printed thermoplastic
polyurethane [21] have been realized. While this actuation method has
advantages for many applications requiring safety and compliance, it
inherently limits the dexterity, payload capacity, and level of biomimicry in
the hand design.

The biological design – consisting of rigid bones, soft muscles, tendons,
ligaments, and compliant skin – inspired the development of hybrid soft-rigid
robots (Figure 1). While the rigidity of such a robot provides a stiff
underlying structure for enhanced strength, the softness enables inherent
system compliance that is sought after in many areas of robotics. Such robots
have the great potential to be placed within human environments and provide
smoother interactions without having to fear harm caused by the robot.

A great option to drive tendons in hybrid soft-rigid robots are soft linear
artificial muscles that are compliant and can stretch and contract in one
direction. Soft linear artificial muscles can stretch and contract in one
direction, making them suitable for driving tendons in hybrid soft-rigid
robots. Air pressure is a common method for actuating these muscles. The
McKibben, a well-known type of pneumatic muscle, increases its diameter when
pressurized, leading to contraction due to its woven outer layer [22]. Other
pneumatic designs include high-strain inverse artificial muscles [23] or pouch
motors [24]. Other artificial muscles include using shape memory alloys [25],
dielectric elastomer actuators (DEAs) [26], and hydraulically amplified self-
healing electrostatic (HASEL) muscles [27]. Compared to pneumatic muscles,
these alternatives are directly actuated with electric energy, requiring less
equipment to operate. However, pneumatically driven muscles can respond faster
than shape memory alloys and generate higher forces than DEAs and achieve
larger contractions than HASEL actuators [28].

Recently, the integrated fabrication of soft and rigid structures was
simplified by advances in multi-material additive manufacturing. Vision
controlled jetting (VCJ), a contactless 3D inkjet deposition method, can print
slow-curing polymers like thiol-ene. These polymers are less viscous and more
stable to environmental influences than state-of-the-art acrylate materials
[29]. Using wax-based support material that easily melts at low temperatures
allows for thin-walled cavities and long hollow tubes following a complex path
to be printed in place. These features enable the printing of complete and
functional systems with minimal post-processing and assembly.

###  I-C Contributions

The contributions of this work are:

  * •

The design of an accurate biomimetic hand and forearm actuated with linear
PAMs.

  * •

An easy-to-manufacture, cost-effective, high-strain PAM.

  * •

The integration of vision controlled jetting parts into a functioning robotic
system.

  * •

The evaluation of the system.

##  II Methods

###  II-A Anatomy of the Human Hand and Forearm

In the human hand, the movements of the fingers are actuated by two separate
muscle groups [30, 31]: the extrinsic muscles located in the forearm and the
intrinsic muscles located in the palm. Extrinsic muscles are responsible for
larger movements of the fingers. They can be further categorized into two
functional groups: flexors, which bend the fingers, and extensors, which
straighten the fingers. Figure 2 shows the most important human muscles and
joints with their acronyms. These intrinsic muscles spread the fingers and
contribute to the flexion of the MCP joint, the base joint between the fingers
and the palm. The thumb is actuated independently from the other fingers by
four main extrinsic muscles, which are important for gripping and pinching
motions. For the opposing thumb motion used for many grasps, an important role
is played by the CMC joint of the thumb and intrinsic muscles OP and AP.
Partially shared muscle bellies for different fingers and interconnected
tendons add additional complexity. This is why full flexion or extension of an
individual finger is usually difficult without the muscle affecting the
neighboring fingers [4].

Human skeletal muscles exhibits a maximum static stress of 0.35, which can
only be held briefly due to muscle fatigue. The maximum sustainable stress is
around 0.1 [32] and the skeletal muscle strain is typically around 20 [33].
Muscles can only generate tensile forces and are typically found in antagonist
arrangements to enable flexion and extension motions.

TABLE I: Muscles imitated by PAMs. | Acronym
---
| Reference
---
muscle
| Actuator
---
amount x length
Flexor Digitorium Profundus | FDP | $4\times\qty{220}{\milli}$
Flexor Digitorium Superficialis | FDS | $4\times\qty{200}{\milli}$
|  | $4\times\qty{160}{\milli}$
Flexor Pollicis Longus | FPL | $2\times\qty{160}{\milli}$
Extensor Digitorum | ED | $4\times\qty{220}{\milli}$
Extensor Pollicis Longus | EPL | $1\times\qty{160}{\milli}$
Abductor Pollicis | AP | $2\times\qty{160}{\milli}$
Opponens Pollicis | OP | $1\times\qty{140}{\milli}$

###  II-B Forearm Design

We designed the hand based on a detailed 3D anatomical model of a human male
right forearm, derived from MRI scans from the BodyParts3D database [34]. And
characterized the hand with electric motors [29]. The anatomical bones model
established the rigid basis of our design, while muscle and tendon models were
used as inspiration for placing anchor points for attaching artificial muscles
and shaping the soft joints and material transitions. Figure 2 shows the
integrated PAMs and joints, and Table I lists the PAMs.

The main focus of this work was on recreating the functionality of the
extrinsic muscles ED, FDS, and FDP, which effectuate the larger finger
movements for flexion and extension motions. The resulting system has a total
of 22 PAMs. We neglected inter-dependencies between muscles, as found in
nature, to reduce mechanical and control complexity. It could, however, be
realized by appropriate control strategies.

Special consideration was given to the wrist and thumb CMC joints due to their
complexity and the necessity of seamless integration of soft and rigid
materials. In the wrist, we replicated seven individual tendons, navigating
the challenge of connecting five different bones.

For the thumb CMC joint, a saddle joint, we modified adjacent bones to create
a smooth surface for the joint capsule, also modeling soft joints and
reinforcing ligaments for the remaining two interphalangeal joints of the
thumb. To enable an opposing thumb motion, we mimic the FPL and EPL muscles,
as well as the intrinsic AP and OP muscles. The hand does not provide enough
space to place PAMs of significant length at the original location of the
intrinsic muscles in the palm. Therefore, the corresponding PAMs are placed in
the forearm area. The tendons are guided through tunnels in the metacarpal
bone and along the carpal base to recreate the direction of action of these
muscles.

To facilitate the fixation of the fishing line tendons at the fingers, we
designed attachment holes within the finger bones. To guide the air supply
tubes of the PAMs, we integrate tunnels through the bones. The forearm was
printed as a single rigid component, with the elbow joint stiffened and bones
fused at the elbow.

We use vision controlled jetting, a contactless 3D inkjet deposition method,
to fabricate our soft-rigid-hybrid hand. We were able to print the hand in one
process with sensor pads and pressure lines ready to use. We added tendons
made from fishing line to conduct experiments.

###  II-C Pneumatic Artifical Muscle

We fabricated thin PAMs based on the McKibben principle [22]. Figure 3 shows
the manufacturing process. The PAMs consist of Flexo PET 3.2 (Techflex Inc.)
braided sleeve made of polyethylene terephthalate fibers and an expandable
silicone tubing (Conta Elastomers GmbH) with an inner diameter of 1.5, wall
thickness of 0.25, elongation at break of 540, and hardness Shore A30. The
silicone tube is inserted into the braided sleeving. We use 5-braid
(polyethylene) fiber fishing line (Pure Fishing Inc.) to close the ends with a
knot. We close the knot with a Loctite 401 adhesive (Henkel & Cie. AGA) to
ensure airtight sealing and avoid knot slipping. The fishing line on the upper
end of the PAM is directly used as the tendon connected to the fingers. On the
other end of the PAM, we insert the air supply before closing the knot,
consisting of a PTFE tube with an inner diameter of 0.8 and an outer tube
diameter of 1.6. The insertion length is 5. The air supply tube is connected
to the pneumatic source. The material cost for this PAM amounts to 3.45USD/,
creating a cost-efficient and easily replicable alternative to commercially
available PAM.

We performed load-lifting experiments with incrementally increased pressure to
characterize the force-strain behavior of our PAM (Figure 4). The force
increases approximately linearly with increasing operating pressures. The
maximum force measured at 0 strain was 38.05 at 0.5. The maximum strain
measured was 30.1 at 0.5, which was just slightly lower than the highest
McKibben strain reported in literature [35].

Figure 3: Materials and step-by-step illustration of the manual fabrication
process for the PAM. A Overview of the used materials and the insertion of the
tube into the braided sleeving. B Manufacturing steps for the end terminal. C
Manufacturing steps for the end part with the air supply inlet. Figure 4:
Experimental results for the SRL PAM. A The SRL PAMs used in the printed hand
and forearm are tested in a force strain test setup. A weight is applied to
simulate forces. Displacements can be recorded for different pressures applied
to the PAM. B Force strain curves for different pressures applied to the PAMs.
The error indicators represent an approximated measurement inaccuracy
corresponding to 2 stroke.

###  II-D Pneumatic touch sensors

To provide the hand with a sense of touch, we integrated sensor cavities in
the fingertips and the palm (Figure 5). These cavities were printed from a
low-viscosity polymer and are used to relay a fingertips’ and palm’s contact
with an object. When touching an object with sufficient force, the pouch is
compressed, leading to a measurable pressure increase. The sensors and
connected tubes are integrated into the hand component and fully printed. A
printed soft tube connects the sensor cavity with a pressure
sensor111Honeywell SSCMANV015PG2A3. We used the pressure readings for valve
control by comparing them to a pre-defined threshold of 4.

This threshold can be adjusted depending on the desired tightness of the grip.
When the pressure of the touch sensor of one finger exceeds the threshold
during a flexion motion, the lead pressure increase for the flexor PAMs of
this finger is stopped, and the lead pressure is held constant at the last
value. This control strategy aims at adaptive grasping of objects, in which
each finger bends until a touch contact is measured so the finger positions
adjust individually to an unknown object to grasp it.

Figure 5: Pneumatic touch sensor. A Schematic of a pressure sensor pad with
tube connection. B Photo of the printed functional pressure sensor pad with
tubing. C Sensor pressure over time showing several contacts with the sensor
pad.

###  II-E Fingertip and Grasping Force Test Setup

We characterized the force the hand can exert by measuring the force an
individual finger can apply with a load cell (Figure 6A, B). We also tested
the hand’s grasping force by constructing a cylinder of two half-cylinders
with a load cell connecting the two halves (Figure 6C, D). The load cell
measures the force of grasping and thereby compressing the cylinder during a
medium wrap power grasp [36].

Figure 6: Setup for force measurements. A Fingertip force test, load cell
connected to a microcontroller for readout. B A fingertip applying force to
the load cell. C Load cell in between two cylinder halves used to measure the
grasping force compressing the cylinder. D Grasping force test of the robotic
hand with the test object shown in sub-figure C.

##  III Results

###  III-A Range of Motion

The range of motion of the finger joints is analyzed for the little finger
(Figure 7A-C and Table II) by measuring the finger joint angles in a
stretched, relaxed, and curled position. The flexor PAMs are operated at a
lead pressure of 0.2. The denoted range is not the physical maximum, but
further tendon displacement could lead to damage to the prototype.

Figure 7: Range of motion analysis of the printed hand. The images A, B, and C show the fingers of the hand in A flexed, B relaxed, and C extended position. The joint angles were visually approximated from still images. A Kapandji test, touching the fingertip of each finger with the thumb, is shown in sub-figures D through G. TABLE II: Range of Motion. | Stretched | Relaxed | Curled
---|---|---|---
$\alpha$ | $None$ | $None$ | $None$
$\beta$ | $None$ | $None$ | $None$
$\gamma$ | $None$ | $None$ | $None$

###  III-B Dexterity and grasping

We performed a Kapandji test [37] to assess the opposability of the thumb. In
this medical test, the thumb touches the fingertips or joints of the long
fingers and is attributed a score from 1 to 10. The system successfully
touched all fingertips of the long fingers with the thumb, corresponding to a
Kapandji score of 6 (see Figure 7D-G). The thumb could not reach the joints of
the little finger. While this score is lower than the ideal human performance,
the experiment demonstrates a good opposability of the thumb, which is on the
same level as highly anthropomorphic robotic hands [10].

Various finger motions, hand gestures, and grasps were tested with this system
(Figure 8). We realized power grasps, precision grasps, and intermediate
grasps [36] of a large variety of differently shaped objects with the hand.
The object set included a pen, a coin, a screw, cardboard boxes, plastic
bottles, a spray can, a sphere of 67 diameter (corresponding to a tennis
ball), a Rubik’s cube and other objects with weights of up to 272. A full
flexion and extension movement – closing and opening the hand – takes around
3s. Higher speeds are possible but have not been tested.

Figure 8: Grasping different objects. We demonstrate the ability to grasp a
variety of objects firmly. Pinch grasps for A a ballpoint pen, B a coin, C a
screw, and D a pack of cards. An enveloping grasp is shown for E a cardboard
box, F a ball the size of a tennis ball, G a spray can, and H a Rubik’s cube.

###  III-C Fingertip and Grasping Forces

We measured the fingertip and grasping forces with the test setup described in
Section II-E. Table III shows the results. The measured fingertip force was
[separate-uncertainty=true]1.95(0.15) at a PAM operating pressure of 0.2. We
measured a grasping force of [separate-uncertainty=true]2.97(0.25) at a lead
pressure of 0.18. We estimate a lower systematic error for the fingertip force
compared to the grasping force test. Because we used a 1D load cell in the
test object, slight deviations in its orientation when grasping can effect the
result. For our human participant, we measured 24 at subjectively moderate
effort and 43 at high effort.

TABLE III: Fingertip and Grasping Forces. Type | Force | Actuator Pressure
---|---|---
Fingertip |  [separate-uncertainty=true]1.95(0.15) |  0.2
Grasping |  [separate-uncertainty=true]2.97(0.25) |  0.18
Human moderate grasp |  24 | N/A
Human strong grasp |  43 | N/A

##  IV Discussion

In this work, we developed and integrated a cutting-edge design and
fabrication method of a highly biomimetic robotic hand and forearm system
featuring state-of-the-art soft pneumatic artificial muscles based on the
McKibben PAMs. Simplifying the fabrication process by integrating multi-
material 3D printing pushes the boundaries of biomimetic hand design. It
allows us to quickly and accurately manufacture and test entire designs made
of different soft and rigid materials. Furthermore, touch sensors based on
pressure readings can be directly printed and integrated into the design.

Our pneumatic artificial muscles achieved up to 30.1% strain and forces as
high as 38.05, matching state-of-the-art McKibben muscles at a fraction of
their material cost while decreasing the manufacturing effort. The 22 muscles
actuate the tendon-driven hand, fabricated as a single-part multi-material 3D
print. This innovative design facilitates flexion and extensions in all
fingers and enables a range of dexterous hand gestures and movements.

We showcased a high range of motion and the possibility of grasping very
different objects. The integration of an opposable thumb proved to be crucial.
In the Kapandji test, the hand reached a score of 6 out of 10. While this
score is high for robotic hands, it lacks behind the human inspiration. The
measured maximal fingertip force was 1.95, with material failure being the
primary limiting factor. This complete system weighs 431 and not only mirrors
the appearance of a human hand but also ensures ease of assembly.

The material strength of our printed components is significantly lower than
their biological counterparts. For instance, our printed rigid material has an
ultimate tensile strength of 45 MPa, starkly contrasting with the 133
exhibited by the cortical bone. Furthermore, we observed wear effects on joint
capsules and articulating bone surfaces after approximately 50 flexion
movements of the hand, indicating potential durability issues. The pneumatic
pumps and valves required for the PAMs also presented challenges, being bulky
and expensive, limiting the system’s mobility.

##  V Conclusion

We presented a highly dexterous and functional robotic hand and forearm system
based on human MRI scans. Vision controlled jetting allowed us to drastically
speed up the fabrication process and fully 3D print a soft-rigid robot. Our
easy-to-manufacture, cost-effective, and high-strain pneumatic artificial
muscle provides muscle-like actuation.

Future work will focus on refining the design for enhanced grasping, improving
the touch sensors, and addressing the issues of joint durability and finger
kinematics. Integrating additional sensors and implementing advanced feedback
control systems will be crucial in achieving more precise control and
functionality. Additionally, exploring untethered compressed air sources could
significantly enhance the system’s mobility and application range, bringing us
closer to realizing a fully functional robotic arm.

## ACKNOWLEDGMENT

The authors thank the team at Inkbit Inc. for printing the hand structure, for
their support, and for their advice when designing and operating the robotic
structures. We are also grateful for the valuable advice from Stefan Weirich’s
advisors at RWTH Aachen and the support by the IDEA League scholarship for
Stefan Weirich.

## References

  * [1] Aldo Faisal, Dietrich Stout, Jan Apel and Bruce Bradley  “The Manipulative Complexity of Lower Paleolithic Stone Toolmaking”  In _PLoS ONE_ 5.11, 2010, pp. e13718  DOI: 10.1371/journal.pone.0013718
  * [2] Suhas Kadalagere Sampath, Ning Wang, Hao Wu and Chenguang Yang  “Review on human‐like robot manipulation using dexterous hands”  In _Cognitive Computation and Systems_ 5.1, 2023, pp. 14–29  DOI: 10.1049/ccs2.12073
  * [3] Zhe Xu and Emanuel Todorov  “Design of a highly biomimetic anthropomorphic robotic hand towards artificial limb regeneration”  In _2016 IEEE International Conference on Robotics and Automation (ICRA)_ Stockholm, Sweden: IEEE, 2016, pp. 3485–3492  DOI: 10.1109/ICRA.2016.7487528
  * [4] Benedek József Tasi, Miklós Koller and György Cserey  “Design of the Anatomically Correct, Biomechatronic Hand”  In _arXiv preprint arXiv:1909.07966._ , 2019  DOI: arXiv:1909.07966
  * [5] C. Piazza, G. Grioli, M.G. Catalano and A. Bicchi  “A Century of Robotic Hands”  In _Annual Review of Control, Robotics, and Autonomous Systems_ 2.1, 2019, pp. 1–32  DOI: 10.1146/annurev-control-060117-105003
  * [6] Ehtesham Nazma and Suhaib Mohd  “TENDON DRIVEN ROBOTIC HANDS: A REVIEW”, 2012, pp. 10
  * [7] Seokwon Lee, Samyeul Noh, Yongkwun Lee and Jong Hyeon Park  “Development of bio-mimetic robot hand using parallel mechanisms”  In _2009 IEEE International Conference on Robotics and Biomimetics (ROBIO)_ Guilin, China: IEEE, 2009, pp. 550–555  DOI: 10.1109/ROBIO.2009.5420706
  * [8] S. Kashef, Samane Amini and Alireza Akbarzadeh  “Robotic hand: A review on linkage-driven finger mechanisms of prosthetic hands and evaluation of the performance criteria”  In _Mechanism and Machine Theory_ 145, 2020, pp. 103677  DOI: 10.1016/j.mechmachtheory.2019.103677
  * [9] Charlotte Firth, Kate Dunn, M. Haeusler and Yi Sun  “Anthropomorphic soft robotic end-effector for use with collaborative robots in the construction industry”  In _Automation in Construction_ 138, 2022, pp. 104218  DOI: 10.1016/j.autcon.2022.104218
  * [10] Markus Grebenstein et al.  “The hand of the DLR Hand Arm System: Designed for interaction”  In _The International Journal of Robotics Research_ 31.13, 2012, pp. 1531–1555  DOI: 10.1177/0278364912459209
  * [11] Lois Liow, Angus B. Clark and Nicolas Rojas  “OLYMPIC: A Modular, Tendon-Driven Prosthetic Hand With Novel Finger and Wrist Coupling Mechanisms”  In _IEEE Robotics and Automation Letters_ 5.2, 2020, pp. 299–306  DOI: 10.1109/LRA.2019.2956636
  * [12] Steffen W. Ruehl et al.  “Experimental evaluation of the schunk 5-Finger gripping hand for grasping tasks”  In _2014 IEEE International Conference on Robotics and Biomimetics (ROBIO 2014)_ , 2014, pp. 2465–2470  DOI: 10.1109/ROBIO.2014.7090710
  * [13] Liz Haverkate, Gerwin Smit and Dick H. Plettenburg  “Assessment of body-powered upper limb prostheses by able-bodied subjects, using the Box and Blocks Test and the Nine-Hole Peg Test”  In _Prosthetics and Orthotics International_ 40.1, 2016, pp. 109  DOI: 10.1177/0309364614554030
  * [14] Mohd Azeem and Aasiya Parveen  “Study on Design and Performance Specifications of the Prosthetic Hands”  In _Advancement in Materials, Manufacturing and Energy Engineering, Vol. I_ , Lecture Notes in Mechanical Engineering  Singapore: Springer, 2022, pp. 119–135  DOI: 10.1007/978-981-16-5371-1˙12
  * [15] M.G. Catalano et al.  “Adaptive synergies for the design and control of the Pisa/IIT SoftHand”  In _The International Journal of Robotics Research_ 33.5, 2014, pp. 768–782  DOI: 10.1177/0278364913518998
  * [16] Jan Fras and Kaspar Althoefer  “Soft Biomimetic Prosthetic Hand: Design, Manufacturing and Preliminary Examination”  In _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ Madrid: IEEE, 2018, pp. 1–6  DOI: 10.1109/IROS.2018.8593666
  * [17] Ryan L. Truby, Robert K. Katzschmann, Jennifer A. Lewis and Daniela Rus  “Soft Robotic Fingers with Embedded Ionogel Sensors and Discrete Actuation Modes for Somatosensitive Manipulation”  In _2019 2nd IEEE International Conference on Soft Robotics (RoboSoft)_ Seoul, Korea (South): IEEE, 2019, pp. 322–329  DOI: 10.1109/ROBOSOFT.2019.8722722
  * [18] Jianshu Zhou et al.  “BCL-13: A 13-DOF Soft Robotic Hand for Dexterous Grasping and In-Hand Manipulation”  In _IEEE Robotics and Automation Letters_ 3.4, 2018, pp. 3379–3386  DOI: 10.1109/LRA.2018.2851360
  * [19] Guoying Gu et al.  “A soft neuroprosthetic hand providing simultaneous myoelectric control and tactile feedback”  In _Nature Biomedical Engineering_ , 2021  DOI: 10.1038/s41551-021-00767-0
  * [20] Jonathan P. King et al.  “Design. Fabrication, and Evaluation of Tendon-Driven Multi-Fingered Foam Hands”  In _2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids)_ Beijing, China: IEEE, 2018, pp. 1–9  DOI: 10.1109/HUMANOIDS.2018.8624997
  * [21] Alireza Mohammadi et al.  “A practical 3D-printed soft robotic prosthetic hand with multi-articulating capabilities”  In _PLOS ONE_ 15.5, 2020, pp. e0232766  DOI: 10.1371/journal.pone.0232766
  * [22] Ching-Ping Chou and B. Hannaford  “Measurement and modeling of McKibben pneumatic artificial muscles”  In _IEEE Transactions on Robotics and Automation_ 12.1, 1996, pp. 90–102  DOI: 10.1109/70.481753
  * [23] Elliot W. Hawkes, David L. Christensen and Allison M. Okamura  “Design and implementation of a 300% strain soft artificial muscle”  In _2016 IEEE International Conference on Robotics and Automation (ICRA)_ Stockholm, Sweden: IEEE, 2016, pp. 4022–4029  DOI: 10.1109/ICRA.2016.7487592
  * [24] Ryuma Niiyama et al.  “Pouch Motors: Printable Soft Actuators Integrated with Computational Design”  In _Soft Robotics_ 2.2, 2015, pp. 59–70  DOI: 10.1089/soro.2014.0023
  * [25] Seong Jun Park, Uikyum Kim and Cheol Hoon Park  “A Novel Fabric Muscle Based on Shape Memory Alloy Springs”  In _Soft Robotics_ 7.3, 2020, pp. 321–331  DOI: 10.1089/soro.2018.0107
  * [26] Jung-Hwan Youn et al.  “Dielectric Elastomer Actuator for Soft Robotics Applications and Challenges”  In _Applied Sciences_ 10.2, 2020, pp. 640  DOI: 10.3390/app10020640
  * [27] Philipp Rothemund et al.  “HASEL Artificial Muscles for a New Generation of Lifelike Robots—Recent Progress and Future Opportunities”  In _Advanced Materials_ 33.19, 2021, pp. 2003375  DOI: 10.1002/adma.202003375
  * [28] Kunal Singh and Shilpa Gupta  “Controlled actuation, adhesion, and stiffness in soft robots: A review”  In _Journal of Intelligent & Robotic Systems_ 106.3, 2022, pp. 59  DOI: 10.1007/s10846-022-01754-6
  * [29] Thomas J.. Buchner et al.  “Vision-controlled jetting for composite systems and robots” Number: 7987 Publisher: Nature Publishing Group  In _Nature_ 623.7987, 2023, pp. 522–530  DOI: 10.1038/s41586-023-06684-3
  * [30] M. Schünke et al.  “Prometheus: Allgemeine Anatomie und Bewegungssystem”, LernAtlas der Anatomie  Stuttgart: Thieme, 2005
  * [31] Karl Zilles and Bernhard Tillmann  “Anatomie”, Springer-Lehrbuch  Heidelberg: Springer, 2010
  * [32] I.W. Hunter and S. Lafontaine  “A comparison of muscle with artificial actuators”  In _Technical Digest IEEE Solid-State Sensor and Actuator Workshop_ Hilton Head Island, SC, USA: IEEE, 1992, pp. 178–185  DOI: 10.1109/SOLSEN.1992.228297
  * [33] Seyed M. Mirvakili and Ian W. Hunter  “Artificial Muscles: Mechanisms, Applications, and Challenges”  In _Advanced Materials_ 30.6, 2018, pp. 1704407  DOI: 10.1002/adma.201704407
  * [34] N. Mitsuhashi et al.  “BodyParts3D: 3D structure database for anatomical concepts”  In _Nucleic Acids Research_ 37, 2009, pp. D782–D785  DOI: 10.1093/nar/gkn613
  * [35] Shunichi Kurumaya, Hiroyuki Nabae, Gen Endo and Koichi Suzumori  “Design of thin McKibben muscle and multifilament structure”  In _Sensors and Actuators A: Physical_ 261, 2017, pp. 66–74  DOI: 10.1016/j.sna.2017.04.047
  * [36] Thomas Feix et al.  “The GRASP Taxonomy of Human Grasp Types”  In _IEEE Transactions on Human-Machine Systems_ 46.1, 2016, pp. 66–77  DOI: 10.1109/THMS.2015.2470657
  * [37] A. Kapandji  “Cotation clinique de l’opposition et de la contre-opposition du pouce”  In _Annales de Chirurgie de la Main_ 5.1, 1986, pp. 67–73  DOI: 10.1016/S0753-9053(86)80053-9


# Activation Functions: Dive into an optimal activation function

Vipul Bansal
Department of Mechanical and Industrial Engineering
Indian Institute Of Technology–Roorkee
Roorkee, India
<EMAIL_ADDRESS>

###### Abstract

Activation functions have come up as one of the essential components of neural
networks. The choice of adequate activation function can impact the accuracy
of these methods. In this study, we experiment for finding an optimal
activation function by defining it as a weighted sum of existing activation
functions and then further optimizing these weights while training the
network. The study uses three activation functions, ReLU, tanh, and sin, over
three popular image datasets, MNIST, FashionMNIST, and KMNIST. We observe that
the ReLU activation function can easily overlook other activation functions.
Also, we see that initial layers prefer to have ReLU or LeakyReLU type of
activation functions, but deeper layers tend to prefer more convergent
activation functions.

_Keywords_ Activation Functions $\cdot$ Deep Learning $\cdot$ Neural Networks

##  1 Introduction

Deep Learning is currently one of the most booming fields(Alzubaidi et al.
(2021)). Deep Neural Networks have been a standard component for all sorts of
research done in the domain. These methods have acquired a wide variety of
success in transforming the modern world. The development of many standard
architectures has shown the effectiveness of these methods for various
classification(Sharma et al. (2018)) and segmentation(Minaee et al. (2020))
tasks.

Many models namely, AlexNet(Krizhevsky (2014)), VGG(Simonyan and Zisserman
(2014)), ResNet(He et al. (2016)), SqueezeNet(Iandola et al. (2016)),
DenseNet(Huang et al. (2018)), Inception (Szegedy et al. (2015)),
GoogLeNet(Szegedy et al. (2014)), etc., have gained popularity in recent
years. This study focuses on one of the critical components in all the
standard architectures, i.e., Activation Functions. Before going deep into the
activation function, we need to understand how a neural network works.

Lets consider a input $x_{i}$ such that we need to predict $y$ and estimate a
function $f(x_{i})$ with some error $\epsilon$ such that:

$y=f(x_{i})+\epsilon$ (1)

The simplest way we can find predicted value $\hat{y}$ is by writing
$\hat{y}=w_{i}*x_{i}+b$, where $w_{i}$ stands for weight and $b$ stands for
bais. now lets consider $X$ be a set of input features
$[x_{1}~{}x_{2}.....~{}x_{n}]^{T}$ of shape $n\times 1$. In this scenario
$\hat{y}$ can be written as:

$\hat{y}=[w_{1}~{}w_{2}~{}w_{3}....~{}w_{n}][x_{1}~{}x_{2}.....~{}x_{n}]^{T}+[b]$
(2)

Now, lets consider $y$ to be such that $[y_{1}~{}y_{2}.....~{}y_{m}]^{T}$ of
shape $m\times 1$. In such a case weight matrix $W_{(1)}$ has to be defined of
shape $n\times m$ and bias $B_{(1)}$ is defined of shape $m\times 1$.
Therefore we can write the regression equation as:

$\hat{y}=W_{(1)}^{T}X+B_{(1)}$ (3)

Now, to model at a higher level we can further multiply another matrix
$W_{(2)}$ with shape $n1\times m$ and bias $B_{(2)}$ of shape $m\times 1$.
Also we modify weight matrix $W_{(1)}$ has to be defined of shape $n\times n1$
and bias $B_{(1)}$ is defined of shape $n1\times 1$. Hence, we can find
$\hat{y}$ as:

$\hat{y}=W_{(2)}^{T}(W_{(1)}^{T}X+B_{(1)})+B_{(2)}$ (4)

But just using additional matrix $W_{(2)}$, would not solve the problem of
modelling difficult functions. Hence there is a need of a function that can
add non-linearity to the model. Hence activation function $\mathcal{A}_{1}(.)$
and $\mathcal{A}_{2}(.)$ are required. Hence we can write final equation 5 as:

$\hat{y}=\mathcal{A}_{2}(W_{(2)}^{T}\mathcal{A}_{1}(W_{(1)}^{T}X+B_{(1)})+B_{(2)})$
(5)

The choice of activation function can actually be sometime quite tricky. There
are wide variety of popular activation function namely, ReLU, Sigmoid, Tanh,
GeLU, GLU(Hendrycks and Gimpel (2020)), Swish(Ramachandran et al. (2017)),
Softplus, Mish (Misra (2020)), PRelU(He et al. (2015)), SeLU(Klambauer et al.
(2017)), Maxout(Goodfellow et al. (2013)), ReLU6(Howard et al. (2017)),
HardSwish (Howard et al. (2019)), ELU(Clevert et al. (2016)), Shifted
Softplus(schütt2017schnet), SiLU(Elfwing et al. (2017)), CReLU(Shang et al.
(2016)), modReLU(Arjovsky et al. (2016)), KAF(Scardapane et al. (2017)),
TanhExp(Liu and Di (2020)) etc.

##  2 Experimental Setup

In this work we are looking for a possibility of a better activation function
by expressing it as a weighted sum of multiple activation functions and then
optimizing those weights for them to find the dominant activation function.

To start with we take 3 common machine learning datasets, MNIST(Lecun et al.
(1998)), Fashion MNIST(Xiao et al. (2017)) and KMNIST(Clanuwat et al. (2018)).
Further, we need to choose a network for experimentation. The model used
consist of two convolutional layers followed by two fully connected layers.
There are activation functions between the layers which is weighted in format.
We take 3 functions as activation function, ReLU($f_{1}$), Tanh($f_{2}$) and
sin($f_{3}$) as given below:

$f_{1}(x)=ReLU(x)=max(o,x)$ (6)

$f_{2}(x)=tanh(x)=\frac{exp(x)-exp(-x)}{exp(x)+exp(-x)}$ (7)

$f_{3}(x)=Sin(x)$ (8)

Now further we define weights $w_{1}$, $w_{2}$, and $w_{3}$ such that
$w_{1},w_{2},w_{3}\geq 0$. Using these we find parameters as $P_{1}$, $P_{2}$
and $P_{3}$ such that:

$P_{i}=\frac{w_{i}}{\Sigma w_{i}},~{}for~{}i=1,2,3$ (9)

Finally we define activation function $A(.)$ as:

$A(x)=P_{1}f_{1}(x)+P_{2}f_{2}(x)+P_{3}f_{3}(x)$ (10)

In the above equation $0\leq P_{i}\leq 1$ always. For each layer we define a
set of unique parameters $w_{1},w_{2},w_{3}$ for each activation function.

For training this network we adopt a unique strategy using Adam optimizer and
adoption of three cycle of training. For the first cycle we start with
freezing the weights $w_{1},w_{2},w_{3}$ and training the parameters of all
other layers with a learning rate of $1e-3$ for 10 epochs. Then further
freezing the parameters of convolutional and fully connected layers and
allowing training of only $w_{1},w_{2},w_{3}$ with a learning rate of $1e-2$
for 10 epochs. At last, we again freeze the weights $w_{1},w_{2},w_{3}$ and
train the parameters of all other layers with a learning rate of $1e-3$ for 10
epochs. We look into the final activation functions and weights for analysis.

##  3 Results and Discussion

In this section we discuss the results from various experiments on the three
datasets. Table 1 shows the results obtained for all three datasets for all 3
layers of activation function for parameters $P_{1}$, $P_{2}$, and $P_{3}$.
The Figure 1 shows the activation functions for the corresponding layers for a
range of value given $-3\leq x\leq 3$.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

Figure 1: Activation Function for MNIST((a) Layer 1, (b) Layer 2, (c) Layer
3)), Fashion MNIST ((d) Layer 1, (e) Layer 2, (f) Layer 3))and KMNIST((g)
Layer 1, (h) Layer 2, (i) Layer 3))

We refer $A_{1}$ to be activation function of first layer of the network and
write the same as for MNIST, Fashion MNIST and KMNIST:

$A_{1}=0.4848f_{1}+0.4437f_{2}+0.0715f_{3}~{}for~{}MNIST$ (11)

$A_{1}=0.5178f_{1}+0.1470f_{2}+0.3352f_{3}~{}for~{}FashionMNIST$ (12)

$A_{1}=0.5590f_{1}+0.0101f_{2}+0.4309f_{3}~{}for~{}KMNIST$ (13)

Table 1: Weights $P_{1},P_{2},P_{3}$ for various activation layters for various datasets DATASETS | LAYERS | $P_{1}$ | $P_{2}$ | $P_{3}$
---|---|---|---|---
| LAYER 1 | 0.4848 | 0.4437 | 0.0715
MNIST | LAYER 2 | 0.0276 | 0.4923 | 0.48
| LAYER 3 | 0.284 | 0.0877 | 0.6283
| LAYER 1 | 0.5178 | 0.147 | 0.3352
FashionMNIST | LAYER 2 | 0.2907 | 0.7001 | 0.0091
| LAYER 3 | 0.1221 | 0.041 | 0.8369
| LAYER 1 | 0.559 | 0.0101 | 0.4309
KMNIST | LAYER 2 | 0.3754 | 0.1167 | 0.5079
| LAYER 3 | 0.0679 | 0.0156 | 0.9165

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

Figure 2: Activation Function for MNIST((a) $-1\leq x\leq 1$, (b) $-10\leq
x\leq 10$, (c)$-100\leq x\leq 100$)), Fashion MNIST ((a) $-1\leq x\leq 1$, (b)
$-10\leq x\leq 10$, (c)$-100\leq x\leq 100$))and KMNIST((a) $-1\leq x\leq 1$,
(b) $-10\leq x\leq 10$, (c)$-100\leq x\leq 100$))

From the equation 11, 12 and 13 we can observe that first activation layer is
more dominated by ReLU ($f_{1}$) activation function. This shows that for
first layer of a network it would always be prefered to choose an activation
function similar to ReLU activation function. When we further take a look into
activation layers $A_{2}$ and $A_{3}$, we observe that the cofficient of ReLU
activation function is smaller than that of other activation functions.

$A_{2}=0.0276f_{1}+0.4923f_{2}+0.4800f_{3}~{}for~{}MNIST$ (14)

$A_{2}=0.2907f_{1}+0.7001f_{2}+0.0091f_{3}~{}for~{}FashionMNIST$ (15)

$A_{2}=0.3754f_{1}+0.1167f_{2}+0.5079f_{3}~{}for~{}KMNIST$ (16)

$A_{3}=0.2840f_{1}+0.0877f_{2}+0.6283f_{3}~{}for~{}MNIST$ (17)

$A_{3}=0.1221f_{1}+0.0410f_{2}+0.8369f_{3}~{}for~{}FashionMNIST$ (18)

$A_{3}=0.0679f_{1}+0.0156f_{2}+0.9165f_{3}~{}for~{}KMNIST$ (19)

After we observe the activation function for individual layer we look for
observing them in various ranges of input. When we take $-1\leq x\leq 1$ than
we observe the plot in figure 2 (a), (d), and (e). We find that activation
function can be approximately written as:

$A=\begin{cases}h_{1}*x,&if~{}x\geq 0\\\ h_{2}*x,&\text{otherwise}\end{cases}$
(20)

In the above activation function $h_{1}$ and $h_{2}$ are arbitrary small
constants. The above function can be transfered and written as LeakyReLU
activation function by taking $h_{1}=1$ and $h_{2}$ be a small value. hence it
can be written as:

$A=max(h_{2}*x,x)=LeakyReLU(X)$ (21)

If we take a broader look these approximations are good for activation
function in initial layer of the network, but as we go deeper we observe that
more convergent activation functions start becoming dominant. But, when $x$
starts to be larger we observe that ReLU can easily dominate other activation
functions because of it larger value as shown in Figure 2 (c), (f), and (i).

##  4 Conclusion

This work shows an experimental study of utilizing a weighted linear
combination of activation functions, looking for a better activation function
needed by the model. We observe that the ReLU activation function is a
powerful activation function that can overlook other activation functions when
used in conjunction. We also follow that initial activation layers prefer a
ReLU type or LeakyReLU activation function. We observe that networks prefer
more convergent activation functions like tanh, sigmoid or sin for deeper
layers. We can further explore the possibility of utilizing more activation
functions and experimenting with more datasets. Additionally, it is possible
to use multiplicative combinations of activation functions that would help us
find better and more robust activation functions for neural networks.

## References

  * Alzubaidi et al. [2021] Laith Alzubaidi, Jinglan Zhang, Amjad J Humaidi, Ayad Al-Dujaili, Ye Duan, Omran Al-Shamma, J Santamaría, Mohammed A Fadhel, Muthana Al-Amidie, and Laith Farhan.  Review of deep learning: Concepts, cnn architectures, challenges, applications, future directions.  _Journal of big Data_ , 8(1):1–74, 2021.
  * Sharma et al. [2018] Neha Sharma, Vibhor Jain, and Anju Mishra.  An analysis of convolutional neural networks for image classification.  _Procedia Computer Science_ , 132:377–384, 2018.  ISSN 1877-0509.  doi:https://doi.org/10.1016/j.procs.2018.05.198.  URL https://www.sciencedirect.com/science/article/pii/S1877050918309335.  International Conference on Computational Intelligence and Data Science.
  * Minaee et al. [2020] Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos.  Image segmentation using deep learning: A survey.  _CoRR_ , abs/2001.05566, 2020.  URL https://arxiv.org/abs/2001.05566.
  * Krizhevsky [2014] Alex Krizhevsky.  One weird trick for parallelizing convolutional neural networks.  _CoRR_ , abs/1404.5997, 2014.  URL http://arxiv.org/abs/1404.5997.
  * Simonyan and Zisserman [2014] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Iandola et al. [2016] Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer.  Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size.  _arXiv preprint arXiv:1602.07360_ , 2016.
  * Huang et al. [2018] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger.  Densely connected convolutional networks, 2018.
  * Szegedy et al. [2015] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision, 2015.
  * Szegedy et al. [2014] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions, 2014.
  * Hendrycks and Gimpel [2020] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus), 2020.
  * Ramachandran et al. [2017] Prajit Ramachandran, Barret Zoph, and Quoc V. Le.  Searching for activation functions, 2017.
  * Misra [2020] Diganta Misra.  Mish: A self regularized non-monotonic activation function, 2020.
  * He et al. [2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, 2015.
  * Klambauer et al. [2017] Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter.  Self-normalizing neural networks, 2017.
  * Goodfellow et al. [2013] Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio.  Maxout networks, 2013.
  * Howard et al. [2017] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications, 2017.
  * Howard et al. [2019] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, and Hartwig Adam.  Searching for mobilenetv3, 2019.
  * Clevert et al. [2016] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter.  Fast and accurate deep network learning by exponential linear units (elus), 2016.
  * Elfwing et al. [2017] Stefan Elfwing, Eiji Uchibe, and Kenji Doya.  Sigmoid-weighted linear units for neural network function approximation in reinforcement learning, 2017.
  * Shang et al. [2016] Wenling Shang, Kihyuk Sohn, Diogo Almeida, and Honglak Lee.  Understanding and improving convolutional neural networks via concatenated rectified linear units, 2016.
  * Arjovsky et al. [2016] Martin Arjovsky, Amar Shah, and Yoshua Bengio.  Unitary evolution recurrent neural networks, 2016.
  * Scardapane et al. [2017] Simone Scardapane, Steven Van Vaerenbergh, Simone Totaro, and Aurelio Uncini.  Kafnets: kernel-based non-parametric activation functions for neural networks, 2017.
  * Liu and Di [2020] Xinyu Liu and Xiaoguang Di.  Tanhexp: A smooth activation function with high convergence speed for lightweight neural networks, 2020.
  * Lecun et al. [1998] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, 1998.  doi:10.1109/5.726791.
  * Xiao et al. [2017] Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.
  * Clanuwat et al. [2018] Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha.  Deep learning for classical japanese literature.  _CoRR_ , abs/1812.01718, 2018.  URL http://arxiv.org/abs/1812.01718.


# Knuth’s non-associative “group” on $\mathcal{P}(\mathbb{N})$

Dominic van der Zypen

###### Abstract.

Donald Knuth introduced in [1] a fast approximation to the addition of
integers (given in binary) in terms of bit-wise operations by

$a+b\;\approx\;a\oplus b\oplus((a\land b)\ll 1).$

Generalizing this to infinite bit-strings we get a binary operation on
$\mathcal{P}(\mathbb{N})$, the power-set of $\mathbb{N}$ (which we identify
with the collection of infinite bit-strings). We show that this operation is
“group-like” in that it has a neutral element, inverses, but it is not
associative. There are a lot of questions left, which the author has not been
able to answer.

##  1\. Introduction

Addition of integers is an important operation in computer science (and in
daily life). Knuth [1] noted that for integers $a,b$ given in binary, we have

$a+b=(a\oplus b)+((a\land b)\ll 1),$

where $\oplus$ denotes bit-wise XOR, $\land$ is bit-wise ${\tt AND}$ and $\ll
1$ means shifting to left by $1$ position.

This identity can be used for an approximation of $+$ using exclusively bit-
wise operations111These operations are very fast operations in computers,
often using only $1$ or a very low number of CPU-cycles:

$a+b\;\approx\;a\oplus b\oplus((a\land b)\ll 1).$

Note that $((a\land b)\ll 1)$ is used to simulate the carry-bit propagation.

This approximation is not only of academic interest; it is used in the
cryptographic scheme NORX [2], for instance.

##  2\. The binary operation $\oplus$ on $\mathcal{P}(\mathbb{N})$

Let $\mathbb{N}=\\{0,1,2,3,\ldots\\}$ be the collection of non-negative
integers and $\mathcal{P}(\mathbb{N})$ be the power-set of $\mathbb{N}$, that
is the collection of all subsets of $\mathbb{N}$. By slight abuse of notation,
we are going to define an operation
$\oplus:\mathcal{P}(\mathbb{N})\times\mathcal{P}(\mathbb{N})\to\mathcal{P}(\mathbb{N})$
and will not use $\oplus$ any more as bit-wise XOR on finite bit-strings.

For any set $A\in\mathcal{P}(\mathbb{N})$, let $A+1=\\{a+1:a\in A\\}$, so
$A+1$ simulates the left-shift. Moreover, given
$A,B\in\mathcal{P}(\mathbb{N})$, we let

$A\,\triangle\,B=(A\setminus B)\cup(B\setminus A)$

be the symmetric difference of $A,B$. Note that $A\,\triangle\,B$ plays the
role of bit-wise XOR.

Finally, we define for all $A,B\in\mathcal{P}(\mathbb{N})$:

$A\oplus B:=(A\,\triangle\,B)\,\triangle\,((A\cap B)+1).$

##  3\. Basic properties of $\oplus$

###  3.1. Commutativity

The definition is clearly symmetric on the two variables, so $\oplus$ is
commutative.

###  3.2. Neutral element

It is easy to see that the empty set $\emptyset\in\mathcal{P}(\mathbb{N})$ is
the neutral element with respect to $\oplus$.

###  3.3. Non-associativity

Let $A=B=\\{0\\}$ and $C=\\{1\\}$. Then $(A\oplus A)\oplus C=\\{2\\}$, but
$A\oplus(A\oplus C)=\emptyset$.

##  4\. Inverse elements in $\oplus$

The goal of this section is to show that for every
$A\in\mathcal{P}(\mathbb{N})$ there is $A^{\prime}\in\mathcal{P}(\mathbb{N})$
such that $A\oplus A^{\prime}=(A\oplus A^{\prime})\oplus((A\cap
A^{\prime})+1)=\emptyset$.

First, a basic observation will be useful later:

######  Fact 4.1.

For any sets $X,Y$ we have $X\,\triangle\,Y=\emptyset$ if and only if $X=Y$,
so $A\oplus A^{\prime}=\emptyset$ amounts to saying
$A\,\triangle\,A^{\prime}=(A\cap A^{\prime})+1$.

Let us first consider a few examples:

  * •

Let $A=\\{0\\}\in\mathcal{P}(\mathbb{N})$. Then let $A^{\prime}=\\{0,1\\}$.

  * •

More generally, let $A=\\{n\\}$ for some $n\in\mathbb{N}$. Then
$A^{\prime}=\\{n,n+1\\}$.

  * •

Let $A=\\{3,4,5\\}$. Then $A^{\prime}=\\{3,5,6\\}$.

Note that always we need $\min(A)\in A^{\prime}$ for $A\neq\emptyset$. Now we
are ready to construct $A^{\prime}$ for general $A\in\mathcal{P}(\mathbb{N})$.

We assume that $A\in\mathcal{P}(\mathbb{N})\setminus\\{\emptyset\\}$ for the
remainder of this section.

First, for $a\leq b\in\mathbb{N}$ we let $[a,b]$ denote the finite set of
integers $x$ with $a\leq x\leq b$. For $n\in\mathbb{N}$ we define the
(backward) stretch of $A$ with respect to $n$ by

$\text{stretch}(A,n)=0\text{ if }n\notin A,$

and

$\text{stretch}(A)=\max\\{k\leq n:[n-k,n]\subseteq A\\}+1\text{ if }n\in A.$

We first illustrate and motivate graphically the notion of
$\text{stretch}(A,n)$, as well as the construction of $A^{\prime}$, for the
example $A=\\{3,4,5,10,12\\}\in\mathcal{P}(\mathbb{N})$.

$A:$$A^{\prime}:$01234567891011121314$\text{stretch}(A,5)=3$$\text{stretch}(A^{\prime},11)=2$$\text{stretch}(A,12)=1$$\text{stretch}(A^{\prime},4)=0$

We can quickly verify that for $A^{\prime}=\\{3,5,6,10,11,12,13\\}$ we have
$A\oplus A^{\prime}=\emptyset$.

######  Proposition 4.2.

Let $A\in\mathcal{P}(\mathbb{N})$ be non-empty, and let

$A^{\prime}=\\{x\in A:\text{stretch}(A,x)\text{ is odd}\\}\;\cup$
$\hskip 28.45274pt\\{y\in\mathbb{N}\setminus A:y>0\text{ and
stretch}(A,y-1)\text{ is odd}\\}.$

Then $A\oplus A^{\prime}=\emptyset$.

(Note that we call $n\in\mathbb{N}$ odd if $n=2k+1$ for some
$k\in\mathbb{N}$.)

Proof of 4.2. By fact 4.1 we need to show that

$A\,\triangle\,A^{\prime}=(A\cap A^{\prime})+1.$

In the following we show that either set is a subset of the other set.

$\subseteq$: Suppose that $x\in A\,\triangle\,A^{\prime}$.

Case 1.1 : $x\in A\setminus A^{\prime}$. This means that
$\text{stretch}(A,x)>0$. From $x\notin A^{\prime}$ and the definition of
$A^{\prime}$ we get that $\text{stretch}(A,x)$ is even. So in particular
$\text{stretch}(A,x)\geq 2$, implying $x-1\in A$ and $x\geq 1$. Therefore
$\text{stretch}(A,x-1)$ is odd, implying $x-1\in A^{\prime}$ by definition of
$A^{\prime}$. So $x-1\in(A\cap A^{\prime})$, whence $x\in(A\cap
A^{\prime})+1$.

Case 1.2 : $x\in A^{\prime}\setminus A$. By definition of $A^{\prime}$, this
means that $x>0$ and $\text{stretch}(A,x-1)$ is odd. So $x-1\in A$, and the
definition of $A^{\prime}$ implies $x-1\in A^{\prime}$, yielding $x-1\in(A\cap
A^{\prime})$ and $x\in(A\cap A^{\prime})+1$.

$\supseteq$: Suppose that $x\in(A\cap A^{\prime})+1$. In particular, $x>0$ and
$x-1\in(A\cap A^{\prime})$. The statements $x-1\in A^{\prime}$ and $x-1\in A$
and the definition of $A^{\prime}$ collectively give us:

> $(\star)$ $\text{stretch}(A,x-1)$ is odd.

Case 2.1 : $x\in A$. Statement $(\star)$ and the definition of
$\text{stretch}(\cdot,\cdot)$ imply $\text{stretch}(A,x)$ is even, and by the
definition of $A^{\prime}$ we get $x\notin A^{\prime}$. So we get $x\in
A\setminus A^{\prime}$.

Case 2.2 : $x\notin A$. The definition of $A^{\prime}$ and $(\star)$ jointly
imply $x\in A^{\prime}$, therefore $x\in A^{\prime}\setminus A$.

So we established that $A\,\triangle\,A^{\prime}=(A\cap A^{\prime})+1$, which
is equivalent to $A\oplus A^{\prime}=\emptyset$. $\Box$

##  5\. Further inquiries

###  5.1. Uniquess of solutions to $A\oplus X=B$

I think that the inverses constructed in proposition 4.2 are unique, and there
could be an inductive argument showing this. Moreover, it seems that the
following more general statement holds:

> For all $A,B\in\mathcal{P}(\mathbb{N})$ there is a unique
> $X\in\mathcal{P}(\mathbb{N})$ such that $A\oplus X=B$.

###  5.2. Associative substructures of $(\mathcal{P}(\mathbb{N}),\oplus)$

One interesting direction in the analysis of $\oplus$ is the search for “sub-
groups”, that is, associative subsets of $\mathcal{P}(\mathbb{N})$ closed
under $\oplus$ and inverses. Which finite or infinite Abelian groups are
isomorphic to a sub-group of $(\mathcal{P}(\mathbb{N}),\oplus)$? Moreover,
Zorn’s Lemma implies that every sub-group of
$(\mathcal{P}(\mathbb{N}),\oplus)$ is contained in a maximal sub-group with
respect to set inclusion $\subseteq$. Does every maximal subgroup of
$(\mathcal{P}(\mathbb{N}),\oplus)$ have the same cardinality?

## References

  * [1] Donald E. Knuth, The Art of Computer Programming, Volume 4A, Addison-Wesley, Upper Saddle River, New Jersey (2011).
  * [2] Jean-Philippe Aumasson, Philipp Jovanovic, Samuel Neves, Analysis of NORX: Investigating differential and rotational properties, https://www.aumasson.jp/data/papers/AJ14a.pdf