* Suppose that deleting two circles from a maximal cut system creates a complementary component of type (0,5). Then in this component there is a cycle of five A-moves involving the circles $\beta_i$ shown in figure <ref>(b): $\{\beta_1,\beta_3\}\to \{\beta_1,\beta_4\}\to \{\beta_2,\beta_4\}\to \{\beta_2,\beta_5\}\to\{\beta_3,\beta_1\}.$

* Suppose that deleting one circle from a maximal cut system creates a component of type (0,4). Then in this component there are circles $\beta_1,\beta_2$ and $\beta_3$, as shown in figure <ref>(c), which yield a circle of three A-moves: $\beta_1\to \beta_2 \to \beta_3\to \beta_1$. No other loops in the given pants decomposition change.

* Suppose that deleting two circles from a maximal cut system creates a complementary component of type (1,2). Then in this component there is a cycle of four A-moves and two S-moves as shown in figure <ref>: $\{\alpha_1,\alpha_3\}\to\{\alpha_1,\epsilon_3\}\to\{\alpha_2,\epsilon_3\}\to\{\alpha_2,\epsilon_2\}\to\{\alpha_2,\epsilon_1\}\to\{\alpha_3,\epsilon_1\}\to\{\alpha_3,\alpha_1\}$.

* If two moves which are either A-moves or S-moves are supported in disjoint subsurfaces of $\mathcal{R}$, then they commute, and their commutator is a cycle of four moves.

It is even possible to construct a two dimensional cell complex $\mathcal{P}(\mathcal{R})$ whose vertices are isotopy classes of pants decompositions of $\mathcal{R}$,  edges are elementary moves, and  2-cells are attached via the basic relations.
Such a  pants decomposition complex $\mathcal{P}(\mathcal{R})$ is simply connected [34].

§ THE MATRICES FOR THE FENCHEL-NIELSEN COORDINATES

Consider a hyperbolic Riemann surface $\mathcal{R}$ with genus $g$ and $n$ punctures, corresponding to the Fuchsian group $\Gamma$, so that $\mathcal{R}=\mathbb{H}/\Gamma$.   In this section we shall review the construction of the matrices that generates the Fuchsian group as a function of the  Fenchel-Nielsen coordinates for the Teichmüller space $\Gamma$ [32] .

§.§ The Fenchel-Nielsen System

There are at most $p=3g-3+n$ distinct simple disjoint geodesics $L_1,\cdots, L_p$ on the genus $g$ Riemann surface $\mathcal{R}$ with $n$ punctures.  We shall call these $p$ simple geodesics the coordinate geodesics. The coordinate geodesics divide the Riemann surface $\mathcal{R}$ into $q=2g-2+n$ pairs of pants, $P_1,\cdots,P_q$.  Each coordinate geodesic corresponds to  either a boundary element of two distinct pairs of pants, or corresponds to two boundary element of the same pair of pants.  Let us order the coordinate geodesics  and  the pairs of pants with the following set of rules, although there is no canonical way to do this.

* If there is a  coordinate geodesic, then $L_1$ is dividing and lies between $P_1$ and $P_2$.

* If $q\geq 3$, then $L_2$ lies between $P_1$ and $P_3$, see figure. <ref>.

* The first $q-1$ coordinate geodesics, and the $q$ pairs of pants, $P_1,\cdots,P_q$, are ordered so that, for every $j=3,\cdots,q-1$, there is an $i=i(j)$, with $1\leq i(j)<j$, so that $L_j$ lies on the common boundary of $P_i$ and $P_j$.

* The coordinate geodesics $L_1,\cdots, L_{q-1}$ are called the attaching geodesics and coordinate geodesics $L_q,\cdots,L_p$ are called the handle geodesics.

[line width=1pt] (1,1) .. controls (1.75,1) and (2.25,.75)  ..(3,0);
[line width=1pt] (1,-2) .. controls(1.75,-2) and (2.25,-1.75)  ..(3,-1);
[line width=1pt] (1,.4) .. controls(2.75,.2)  and (2.75,-1.2) ..(1,-1.4);
[gray,line width=1pt] (1,.7) ellipse (.115 and .315);
[gray,line width=1pt] (1,-1.7) ellipse (.115 and .315);
[line width=1pt] (3,0) .. controls(3.3,-.25) and (3.75,-.25) ..(4,0);
[line width=1pt] (3,-1) .. controls(3.3,-.75) and (3.75,-.75) ..(4,-1);
[line width=1pt] (4,0) .. controls(5,1) and (6,1) ..(7,0);
[line width=1pt] (4,-1) .. controls(5,-2) and (6,-2) ..(7,-1);
[line width=1pt] (7,0) .. controls(7.2,-.15)  ..(8,-.15);
[line width=1pt] (7,-1) .. controls(7.2,-.85)  ..(8,-.85);
[gray,line width=1pt] (8,-.5) ellipse (.15 and .35);
[line width=1pt] (4.75,-.75) .. controls(5.25,-1.15) and (5.75,-1.15) ..(6.25,-.75);
[line width=1pt] (4.75,-.35) .. controls(5.25,0.1) and (5.75,0.1) ..(6.25,-.35);
[line width=1pt] (4.75,-.35) .. controls(4.65,-.475) and (4.65,-.625) ..(4.75,-.75);
[line width=1pt] (6.25,-.35) .. controls(6.35,-.475) and (6.35,-.625) ..(6.25,-.75);
[line width =1.5pt, color=gray] (3.5,-.5) ellipse (1.2 and .15);
[line width =1.5pt, color=gray] (5.5,.375) ellipse (.15 and .375);
[line width =1.5pt, color=gray] (5.5,-1.39) ellipse (.15 and .35);
(2,.5) node[below right] $P_2$  (2,-1.5) node[above right] $P_1$ (6.5,-.5)node [ right ] $P_3$  (3.5,-.25) node  [below right ] $L_1$ (5.55,-1.25) node  [ right ] $L_2$ (5.55,0.5) node  [below right ] $L_3$;

Ordered coordinate geodesics and pair of pants. Thick gray curves are the coordinates geodesics. $L_1$ and $L_2$ are attaching  geodesics and $L_3$ is handle geodesic.

We shall call this ordered set as  Fenchel-Nielsen system (F-N system). Denote  the Fenchel-Nielsen coordinates  defined with respect to F-N system as

\begin{equation}\label{FNcord}
\Phi=(l_1,\cdots,l_{p},\tau_1,\cdots,\tau_p)\in (\mathbb{R^+})^{p}\times\mathbb{R}^p
\end{equation}

where $l_i,~i=1,\cdots,p$ are the geodesic lengths of curves $L_i,~i=1,\cdots,p+$ and $\tau_i,~i=1,\cdots,p$ are the twist about $L_i,~i=1,\cdots,p$. The length of geodesic is an absolute concept, but the twist about a geodesic is not an absolute quantity. It requires a reference surface. Let us call this reference surface $\mathcal{R}_0$. For convenience, let us introduce the parameters:

\begin{equation}
s_i=\mathrm{sinh}~\frac{l_i}{2},  ~ t_i=\mathrm{sinh}~\frac{\tau_i}{2},~ \mathrm{for}~ i=1,\cdots,p.
\end{equation}

§.§.§ The Ordering for the Boundary Components of the Pairs of Pants

The Fenchel-Nielsen system contains an ordered set of geodesics on the surface $\mathcal{R}$. Let us assign an ordering for the boundary components of the pairs of pants obtained from $\mathcal{R}_0$ as a result of  a pair of pants decomposition of $\mathcal{R}$ along the coordinate geodesics of the Fenchel-Nielsen system.   In the most cases, the ordering of the coordinate geodesics and boundary elements of $\mathcal{R}_0$ induces an ordering of the boundary elements of the pairs of pants. There are two exceptional cases in which there are two boundary elements of a pair of pants $P$ corresponding to just one coordinate geodesic of $\mathcal{R}_0$. They are the following.

* $\mathcal{R}_0$ is torus with one  puncture. In this case, two of the boundary elements of the pair of pants $P$ are hyperbolic, necessarily of the same size, and the third one is parabolic. It is not possible to differentiate between the two boundary elements of $P$ corresponding to one coordinate geodesic on $\mathcal{R}_0$.

* The pair of pants $P\subset \mathcal{R}_0$ having one boundary element, $ b_1$, corresponds to an attaching geodesic on $\mathcal{R}_0$. While the other two boundary elements $b_2, b_3$ both correspond to the same handle geodesic. Which boundary element of $P$ we call $b_2$ or $b_3$ is arbitrary.

If the surface is not an example of exceptional case then we shall label the  boundary elements of the pair of pants, $P_i$, as $b_{i,1},b_{i,2},b_{i,3}$, in the following order: hyperbolic boundary elements come before the parabolic boundary elements.

§.§.§ Directing the Coordinate and the Boundary Geodesics

Since the twist parameter around a curve depends on the orientation, we need to assign a direction for all coordinate geodesics on $\mathcal{R}_0$. We say that  two geodesics on the boundary of some pair of pants $P$ are consistently oriented with respect to $P$, if $P$ lies on the right as we traverse either geodesics in the positive direction. Assume that we already directed geodesics $L_1,\cdots,L_j,~j\geq1$. If $L_{j+1}$ lies on the boundary of two distinct pairs of pants, or is a boundary geodesic, then there is a lowest index $i$, so that $L_{j+1}$ lies on the boundary of $P_i$. Since $j+1>1,~L_{j+1}$ corresponds to either $b_{i,2}$ or $b_{i,3}$, for $b_{i,1}$ must correspond to some attaching geodesic, $L_{j'},~j'\leq j$. We direct $L_{j+1}$ so that $L_{j+1}$ and $L_{j'}$ are consistently oriented as boundary elements of $L_i$. If $L_{j+1}$ is a handle geodesic, with same pair of pants, $P_i$, on both sides of $L_{j+1}$, then its direction is more complicated. We shall explain this case when we encounter such a situation later.

§.§ From the F-N System to the Generators of the Fuchsian Group

Assume that we are given an F-N system on  $\mathcal{R}_0$, where the coordinate geodesic $L_1,\cdots,L_p$, the pairs of pants $P_1,\cdots,P_q$  are ordered and the coordinate geodesics are directed as already discussed. We also assume that we are given a point

\begin{equation}\label{pointfn}
\Phi=(l_1,\cdots,l_{p},\tau_1,\cdots,\tau_p)\in (\mathbb{R}^+)^{p}\times \mathbb{R}^p
\end{equation}

in the corresponding space of F-N coordinates. Now we shall describe a canonical procedure for writing down a set of generators for the corresponding Fuchsian group.  We write the generators in the following order. The first $2p-q+1$ generators are hyperbolic, $p$ coordinate geodesics followed by $p-q+1$ handle closers. The remaining $n$ generators are  parabolic boundary elements. In general the total number of generators, $d=2q+1=4g-3+2n$, is  far from minimal, which is $2g+n$.

§.§.§ The Normalization

Let $a_1,\cdots,a_d$ be a set of generators for a Fuchsian group. Assume that $a_1$ and $a_2$ are as follows.

* The generators $a_1$ and $a_2$ are both hyperbolic with disjoint axes.

* The axis of $a_1$ denoted by $A_1$ lie on the imaginary axis, pointing towards $\infty$; the axis of $a_2$ denoted by $A_2$ lies in the right half-plane, with the attracting fixed point smaller than the repelling  one.

* The common orthogonal between $A_1$ and $A_2$ lies in the unit circle.

§.§.§ The Basic Building Blocks

Let us label  the three distinct boundary elements of each pair of pants $P_i$  as $b_{i,1},b_{i,2},b_{i,3}$. The order of the boundary elements is determined by the order of the coordinates geodesics and boundary elements of the F-N system, except for the cases where we have not yet distinguished between $b_{i,2}$ and $b_{i,3}$. The size of each hyperbolic boundary element is specified by the Fenchel-Nielsen coordinate $\Phi$. Let $P_i'$ be the Nielsen extension of $P_i$. For each $i=1,\cdots,q$ there is a unique normalized pants group $\Gamma_i$ representing $P_i'$, i.e., $\mathbb{H}/\Gamma_i=P_i'$. $\Gamma_i$ has three distinguished generators $a_{i,1},a_{i,2},a_{i,3}$, where $a_{i,1}a_{i,2}a_{i,3}=1$. The axis of $a_{i,k}$, $A_{i,k}$, projects onto $b_{i,k},~k=1,2,3$.

§.§.§ Sphere with Holes

Consider subspace $Q_j$ of  $\mathcal{R}_0$, defined as the interior of the closure of the union of the $P_i,~i\leq j$ for $j=1,\cdots,q-1$. Let $Q_j'$ be the Nielsen extension of $Q_j$. Each subspace $Q_j'$ has a naturally defined F-N system, where the coordinate geodesics are $L_1,\cdots,L_{j-1}$ and the pairs of pants are $P_1,\cdots,P_j$. Say, $\Gamma_1=J_1$ be the normalized pants group representing $P_1$. Assume that $a_{1,1}$ and $a_{1,2}$ generate  the group $\Gamma_1$.  We can choose the normalizations for different cases as follows.

* If $a_{1,1}$ is hyperbolic, then $A_1$, the axis of $a_{1,1}$, is the imaginary axis, pointing towards $\infty$.  $A_2$, the axis of $a_{1,2}$, lies in the right-half plane and $M_3$, the common orthogonal between $A_1$ and $A_2$, lies on the unit circle.

* If $a_{1,1}$ is parabolic, then $A_1$ is the point at infinity, and $M_3$ lies on the imaginary axis. If $a_{1,2}$ is also parabolic, then the axis  $A_2$ is necessarily at 0.

The imaginary axis, which is the axis of $a_{1,1}$, projects to $L_1$ and the positive direction of axis projects to the positive direction $L_{1}$. Consider $\Gamma_2$, the normalized pants group representing $P_2$. Let $c_2$ be the hyperbolic isometry which maps the right half-plane onto the left half-plane, while introducing a twist of $\tau_1$ in the positive direction on $L_1$; i.e.,

\begin{equation}
c_2(0)=\infty,~c_2(\infty)=0,~c_2(\mathrm{i})=e^{\tau_1}\mathrm{i}. \nonumber
\end{equation}

\begin{equation}
\hat \Gamma_2=c_2\Gamma_2c_2^{-1}
\end{equation}

\begin{equation}
a_{1,1}=\hat a_{1,1}=c_2a_{2,1}c_2^{-1}=\hat a_{2,1}
\end{equation}

It follows that one can use the amalgamated free product (AFP) combination theorem to amalgamate $\hat \Gamma_2$ to $\Gamma_1$. The AFP combination theorem  asserts that

* $J_2=\langle \Gamma_1,\hat \Gamma_2\rangle$ is Fuchsian.

* $J_2$ is generated by $a_{1,1},a_{1,2},a_{1,3},\hat a_{2,2},\hat a_{2,3}$. In addition to the defining relations of $\Gamma_1$ and $\Gamma_2$, these satisfy the additional relation: $a_{1,1}=\hat a_{2,2}\hat a_{2,3}$.

* $\mathbb{H}/J_2$ is sphere with 4 borders; the corresponding boundary subgroups are generated by the above four generators.

Now we can identify $\mathbb{H}/J_2$ with $Q_2'$. This imposes a new order on the boundary elements of $\mathbb{H}/J_2$ as follows.

* If $b$ and $b'$ are boundary elements of $Q_2$, where $b$ precede $b'$ as coordinate geodesic or as boundary elements of $\mathcal{R}_0$, then $b$ precede $b'$ as a boundary element of $Q_2$.

* If $b$ and $b'$ both corresponds to the same coordinate geodesic on $\mathcal{R}_0$, and $b$ corresponds to a boundary geodesic on $P_1$, while $b'$ corresponds to a boundary geodesic on $P_2$, then as boundary elements of $Q_2$, $b$ precede $b'$.

* If $b$ and $b'$ both correspond to boundary elements of either $P_1$ or $P_2$, then $b$ precede $b'$ if $b$ corresponds to $A_{i,2}$ and $b'$ corresponds to $A_{i,3}$.

* In the case that $P_i$ has two boundary elements corresponding to the same handle geodesic, $L$, we direct $L$ so that positive direction of $A_{i,2}$ projects onto the positive direction of $L$.

This way we can order the boundary elements of $Q_2$ and direct them. Introduce a new ordered set of generators for $J_2$ as $a_1^2,\cdots,a_5^2$, where $a_1^2=a_{1,1}=\hat a_{2,1}^{-1}$ and $a_2^2,\cdots,a_5^2$ are the generators $a_{1,2},a_{1,3},\hat a_{2,3}$. $A_j^2$ projects onto $b_{j-1}^2$. Those $A_j^2$ that are hyperbolic are all directed so that the attracting fixed point of $a_j^2$ is smaller than the repelling fixed point.  Each $A_j^2$ has a canonical base point on it. For $i=1,\cdots, q$ the point $i$ is the canonical base point on $A_{i,1}$. For $i=1,\cdots,q$ and for $k=2,3$ the base point on $A_{i,k}$ is the point of intersection of $A_{i,k}$ with the common orthogonal between $A_{i,1}$ and $A_{i,k}$. In the case that $a_j^2=a_{1,i}$, the base point is the canonical base point for $a_{1,i}$. In the case that $a_j^2=c_2a_{2,i}c_2^{-1}$ then the canonical base point is the $c_2$ image of the canonical base point for $a_{2,i}$.

Now iterate the above process. Assume that we have found $J_k$ representing $Q'_k$, where $k<q$, with distinguished subgroups, $\hat \Gamma_1,\cdots,\hat \Gamma_k$. Assume that we have ordered boundary elements of $Q_k$, and that we have  found the distinguished generators $a_1^k,\cdots,a_{2k+1}^k$ for $J_k$, where each distinguished generator lies in some distinguished subgroup, so that, for $i=1,\cdots,k$, $A_i^k$ projects onto the boundary element $b^k_{k-i}$ of $Q_k$. Assume that we have assigned a canonical base point for on each geodesic $A_i^k,~i=k+1,\cdots,2k+1$. Also assume that all the hyperbolic boundary generators of $J_k$ are directed so that the attracting fixed point is smaller than the repelling point. Then there is some $j$ so that $A^k_{k+j}$  projects onto $L_{k+1}$. We need to renormalize $\Gamma_{k+1}$ so that the renormalized $A_{k+1,1}$ agrees with $A^k_{k+j}$, but with opposite direction and with an appropriate twist.

Let us define the conjugator $c_{k+1}$ to be the unique orientation preserving hyperbolic isometry mapping the left half-plane onto the action half-plane of $A^k_{k+j}$, while mapping the base point $i$ onto the point whose distance from the base point on $A^k_{k+j}$, measured in the positive direction along $A^k_{k+j}$, is exactly $\tau_k$. The AFP combination theorem assures that,

* $J_{k+1}=\langle J_k,c_{k+1}\Gamma_{k+1}c^{-1}_{k+1}\rangle$ is Fuchsian.

* $J_{k+1}=\langle a^k_1,\cdots, a^k_{2k+1},c_{k+1}a_{k+1}c^{-1}_{k+1},c_{k+1}a_{k+1,2}c_{k+1}^{-1},c_{k+1}a_{k+1,3}c_{k+1}^{-1}\rangle$. These satisfy the dining relations of $J_k$ and $\Gamma_{k+1}$ together with the additional defining relation: $a^k_{k+j}=(c_{k+1}a_{k+1,1}c^{-1}_{k+1})^{-1}$.

* $\mathbb{H}/J_{k+1}$ is a sphere with $2k+1$ holes .

As above, rewrite the generators of $J_{k+1}$ as $a_1^{k+1},\cdots,a_{2k+3}^{k+1}$, where the first $k$ generators correspond to the $k$ coordinate geodesics of $Q_{k+1}$, and the remaining generators correspond to the boundary elements of $Q_{k+1}$ in the following order.  Each boundary generator $a_i$ of $J_{k+1}$ corresponds to either a coordinate geodesic $L_j$ on $\mathcal{R}_0$ or it corresponds to a boundary element $b_j$ of $\mathcal{R}_0$. Pulling back the order of the coordinate geodesic and boundary elements for $\mathcal{R}_0$ imposes a partial order on the boundary generators of $J_{k+1}$. The only ambiguities occur  when the generators $a_i$ and $a_i'$ both corresponds to the same coordinate geodesic. If $a_i$, respectively , $a_i'$, lies in the distinguished subgroups $\hat \Gamma_j$, respectively, $\hat \Gamma_{j'}$, where $j<j'$, then $a_i$ precedes $a_i'$. If $a_i$ and $a_i'$ both lie in the same distinguished subgroup, $\hat \Gamma_j$, then $\hat a_{j,2}$ precedes $\hat a_{j,3}$.

The hyperbolic boundary generators of $J_k$ all have distinguished base points. We assign the $c_k$ image of the distinguished point on $a_{k_1,2}$ and $a_{k+1,3}$ as the distinguished base point on the new boundaryngunerators of $J_{k+1}$. We also observe that the hyperbolic boundary generators of $J_{k+1}$ are all directed so that their attracting fixed points are smaller than  their repelling fixed points. When $k=q-1$, we reach the group $J_q$, representing $Q_q'$.

§.§.§ Closing the Handles

Rename the group $J_q$ and call it $K_0$. Also  rename its ordered set of generators and call them, in order, $a_1^0,\cdots,a^0_{2q+1}$. The first $q-1$ of these generators corresponds to the attaching coordinate geodesics of $\mathcal{R}_0$; the next $2(p-q+1)$ generators correspond to the boundary elements of $Q_q$ that are handle geodesics on $\mathcal{R}_0$.  Define the first handle-closer $d_1$ as the orientation preserving hyperbolic isometry mapping the action half plane of $a_q^0$ onto the boundary half-plane of $a^0_{q+1}$, while mapping the point at distance $-\tau_q$ from the base point on $A^0_q$ to the base point on $A^0_{q+1}$. Note that $d_1$ conjugates $a^0_q$ onto $(a^0_{q+1})^{-1}$. Then the HNN combination theorem asserts that,

* $K_1=\langle K_0,d_1\rangle$ is Fuchsian.

* $K_1$ is generated by $a^0_1,\cdots,a^0_{2q+1},d_1$. These satisfy the defining relations of $K_1$ together with the additional relation: $a^0_{q+1}=d_1(a^0_q)^{-1}d_1^{-1}$.

* $\mathbb{H}/K_1$ is hyperbolic Riemann surface with genus 1 and $q$ boundary elements.

The distinguished subgroups of $K_1$ are the distinguished subgroups of $K_0$. The generators of $K_1$ are generators of $K_0$, in the same order, but with $a^0_{q+1}$ deleted, and $d_1$ added to the list. In the list of generators for $K_1$, $d_1$ appears after all the generators corresponding to coordinate geodesics on $\mathcal{R}_0$, and before the first generator corresponding to a boundary element of $\mathcal{R}_0$. After $g$ iterations, we reach the discrete group $\Gamma=K_g$, where $\mathbb{H}/\Gamma=\mathcal{R}_0$. Further, $\Gamma$ has $q$ distinguished subgroups, representing in order the $q$ pairs of pants, $P_1,\cdots,P_q$. $\Gamma$ has $2q+1$ distinguished generators, $a_1,\cdots, a_{2p-q+1}$. For $i=1,\cdots,p$ the axis $A_i$ projects onto the coordinate geodesic $L_i$; the generators $a_{p+1},\cdots, a_{2p-q+1}$ are handle closers and the axes of the remaining generators project, in order, onto the boundary elements of $\mathcal{R}_0$.

§.§ The Explicit Matrices

In this section we shall describe an algorithm for writing the generators of the Fuchsian group in terms of the Fenchel-Nielsen coordinates.

§.§.§ Reduction to Primitive Conjugators

Let $\Gamma_i$ and $\Gamma_{j+1}$ be the normalized pants groups representing the pairs of pants, $P_i$ and $P_{j+1}$, respectively. We assume that $i<j+1$, and that there is a $k,~1\leq k\leq 3$, so that $a_{j+1,1}$ and $a_{i,k}$ represent the same attaching geodesic, $L_j$ on $\mathcal{R}_0$, but with reverse orientations. Then $|\mathrm{tr}(\tilde a_{j+1,1})|=|\mathrm{tr}(\tilde a_{i,k})|$. The elementary conjugator $e_{i,k}$ maps the boundary half-plane of $a'=a_{j+1,1}$ onto the action half-plane of $a=a_{i,k}$ while introducing a twist of $\tau_j$.  The untwisted elementary conjugator, $e^0_{i,k}$, which is independent of the index $j$, also maps the left half-plane onto the action half-plane of $a$, but maps the base point on $A_{j+1,1}$ to the base point on $A_{i,k}$. If a generates $a$ boundary subgroup of $G$, then the corresponding axis $A$ separates upper-half plane  into two half-planes. The boundary half-plane is precisely invariant under $\langle a\rangle$ in $G$. The other half-plane, which is not precisely invariant (unless $G$ is elementary), is called the action half-plane.

Once we have found matrices for the elementary conjugators, then we can inductively find matrices for all conjugators. If $P_i$ and $P_{j+1}$ are adjacent pairs of pants in the F-N system on $\mathcal{R}_0$, with $i<j+1$, and as above, $b_{j+1,1}$ attached to $b_{i,k}$, once we have found the matrix for the elementary conjugator, $e_{i,k}$, the matrix for the conjugator $c_{j+1}$ is given by

\begin{equation}\label{econj}
\tilde c_{j+1}=\tilde c_i\tilde e_{i,k}
\end{equation}

For each $j=q,\cdots, p$ the handle closer $d_j$ conjugates one distinguished boundary generator of $K_0$ onto the inverse of another distinguished boundary generator. These two boundary generators either lie in the same distinguished subgroup or they lie in different distinguished subgroups. If the two boundary generator lie in the same distinguished subgroup, $\hat \Gamma_i$, then there is a conjugator $c_i$, and there is normalized pants subgroup $\Gamma_i$, so that $\hat \Gamma_i=c_i\Gamma_ic^{-1}_i$. In this case, we can choose the matrix for the handle closer as

\begin{equation}\label{hc}
\tilde d_j=\tilde c_i\tilde d\tilde c_i^{-1}
\end{equation}

where $\tilde d$ is the matrix for the handle closer.

If these two boundary generators lie in distinct distinguished subgroups, say $\hat \Gamma_i$ and $\hat \Gamma_{i'}$, where $i<i'$. Assume that the first boundary generator corresponds to $\hat a_{i,k}$ and the second corresponds to $\hat a_{i',k'}$. We can write $d_j$ as a product of four transformations: first twist along the axis $\hat A_{i,k}$, then map the boundary half-plane of $a_{i,k}$ onto the right half-plane; then interchange left and right half-planes, and last map the right half-plane onto the boundary half-plane of $\hat a_{i',k'}$.

* Twist transformation preserve both sides of $\hat A_{i,k}$ and maps the point on $\hat A_{i,k}$
, whose distance from the base point is $-\tau_j$, to the base point on $\hat A_{i,k}$. We can write the matrix for this transformation as

\begin{equation}\label{twist}
(\tilde c_i\tilde e^0_{i,k})\tilde f_{\tau_j}(\tilde c_i\tilde e^0_{i,k})^{-1}
\end{equation}

where $f_{\tau}$ is the universal twist map, $f_{\tau}(z)=e^{-\tau}z$.

* The second transformation maps the boundary half-plane of $\hat a_{i,k}$ onto the right half-plane, and maps the base point on $\hat A_{i,k}$ to the base point on the imaginary axis.  The matrix for this transformation is

\begin{equation}\label{brmap}
(\tilde c_i\tilde e^0_{i,k})^{-1}
\end{equation}

* The interchange transformation $g$ interchanges left and right half-planes and preserves the base point on the imaginary axis;  that is

\begin{equation}\label{in}
\end{equation}

* The final transformation transformation maps the left half-plane onto the boundary half-plane of $\hat a_{i',k'}$. The matrix representation of this transformation can be written as

\begin{equation}\label{final}
\tilde c_{i'}\tilde e^0_{i',k'}
\end{equation}

Combining all we obtain

\begin{equation}\label{comb}
\tilde d_j=\tilde c_{i'}\tilde e^0_{i',k'}\tilde g \tilde f_{\tau_j}(\tilde c_i\tilde e^0_{i,k})^{-1}
\end{equation}

§.§.§ The Normalized Pants Groups

Given three numbers $\lambda_1,\lambda_2$ and $\lambda_3$, it is possible to  write down matrices $\tilde a_1$ and $\tilde a_2$, corresponding to the geometric generators for pants group, such that  $|\mathrm{tr}(\tilde a_1)|=2~\mathrm{cosh}~\lambda_1, |\mathrm{tr}(\tilde a_2)|=2~\mathrm{cosh}~\lambda_2$ and $|\mathrm{tr}(\tilde a_3)|=|\mathrm{tr}(\tilde a_1\tilde a_2)^{-1}|=2~\mathrm{cosh}~\lambda_3$. We can assume that the $\lambda_i$ are given so that $a_i$ are in non-increasing order and  $\mathrm{tr}(\tilde a_1)\geq 0$ and $\mathrm{tr}(\tilde a_2)\geq 0$,  without any loss of generality.

Let us choose normalizations for different cases as follows.

* $a_1$ is hyperbolic: then $A_1$ is the imaginary axis, pointing towards $\infty$.  $A_2$ lies in the right-half plane and $M_3$, the common orthogonal between $A_1$ and $A_2$, lies on the unit circle.

* $a_1$ is parabolic: then $A_1$ is the point at infinity, and $M_3$ lies on the imaginary axis. If $a_2$ is also parabolic, then $A_2$ is necessarily at 0.

Now we shall list the generators of all possible pants groups satisfying above normalization.

Three hyperbolic elements

In this case $\lambda_i>0,~i=1,2,3$. We need to find matrices $\tilde a_1$ and $\tilde a_2$ with $\mathrm{tr}(\tilde a_1)=2~\mathrm{cosh}~(\lambda_1);~\mathrm{tr}(\tilde a_2)=2~\mathrm{cosh}~(\lambda_2)$ and $\mathrm{tr}(\tilde a_1\tilde a_2)=-2~\mathrm{cosh}~(\lambda_3)$. We need  $a_1$ and $a_2$ to be appropriately oriented; hence we write our matrices so that the repelling fixed points of $a_2$ is greater than 1, while the attracting fixed point is less than 1. Define $\mu$ and $\nu$ by:

\begin{align}\label{mu}
\mathrm{coth}~\mu&=\frac{\mathrm{cosh}~\lambda_1\mathrm{cosh}~\lambda_2+\mathrm{cosh}~\lambda_3}{\mathrm{sinh}~\lambda_1\mathrm{sinh}~\lambda_2},~\mu>0,\nonumber\\~\nonumber\\
\mathrm{coth}~\nu&=\frac{\mathrm{cosh}~\lambda_1\mathrm{cosh}~\lambda_3+\mathrm{cosh}~\lambda_2}{\mathrm{sinh}~\lambda_1\mathrm{sinh}~\lambda_3},~\nu>0
\end{align}

Then we can write the generators $\tilde a_1, \tilde a_2$ and $\tilde a_3=-\tilde a_2^{-1}\tilde a_2^{-1}$ as follows

\begin{align}\label{3hypg}
\tilde a_1&=\left(\begin{array}{cc}e^{\lambda_1} & 0 \\0 & e^{-\lambda_1}\end{array}\right),\nonumber\\~\nonumber\\
\tilde a_2&=\frac{1}{\mathrm{sinh}~\mu}\left(\begin{array}{cc}\mathrm{sinh}~(\mu-\lambda_2)& \mathrm{sinh}~\lambda_2 \\-\mathrm{sinh}~\lambda_2 & \mathrm{sinh}~(\mu+\lambda_2)\end{array}\right),\nonumber\\~\nonumber\\
\tilde a_3&=\frac{1}{\mathrm{sinh}~\nu}\left(\begin{array}{cc}\mathrm{sinh}~(\nu-\lambda_3)& e^{\lambda_1}\mathrm{sinh}~\lambda_3 \\-e^{\lambda_1}\mathrm{sinh}~\lambda_3 & \mathrm{sinh}~(\nu+\lambda_3)\end{array}\right).
\end{align}

$\mu$ is related to $\delta$, the distance between $A_1$ and $A_2$, by $\mathrm{coth}~\mu=\mathrm{cosh}~\delta$.

Exactly one hyperbolic and at least one parabolic

In this case $\lambda_1>0,~\lambda_2=0$. The matrices are

\begin{align}\label{ohop}
\tilde a_1&=\left(\begin{array}{cc}e^{\lambda_1} & 0 \\0 & e^{-\lambda_1}\end{array}\right), ~\tilde a_2=\left(\begin{array}{cc}1+\beta & -\beta\\\beta & 1-\beta\end{array}\right)\nonumber\\
\text{where}&\nonumber\\
\beta&=-\frac{\text{cosh}~\lambda_1+\text{cosh}~\lambda_3}{\text{sinh}~\lambda_1}
\end{align}

§.§.§ Closing a Handle on a Pair of Pants

Consider three hyperbolic elements $\tilde a_1,\tilde a_2$ and $\tilde a_3$ with $\lambda_2=\lambda_3$. We need to find the matrix corresponds to the handle-closer $d$ which identifies the action of $a_2$ and $a_3$, while twisting by $\tau$ in the positive direction of $A_2$. Matrix corresponding to $d$ is given as

d   =r̃r̃_2ẽ_τ,     where    r=

0     e^λ_1/2

e^-λ_1/2     0

r_2   =1/sinh

coshμ    1

-1    coshμ,

e_τ   =1/sinhμ

sinh(μ-τ/2)     sinhτ/2

-sinh τ/2    sinh(μ+τ/2)

Here $e_{\tau}$ is the hyperbolic motion with the same fixed point as $a_2$ and with trace equal to $2~\mathrm{cosh} ~\frac{\tau}{2}$. $a_2$ and $e_{\tau}$ have the same attracting fixed point if $\tau>0$, and have opposite attracting fixed points if $\tau>0$. $r_2$ is the reflection in $A_2$ and $r$ is the reflection in the line half way between $A_2$ and $A_3$.

Torus with One Puncture

We require $a_1$ to be parabolic.  Then $\lambda_2=\lambda_3>0$. We normalize so that $A_1$ lies on the positive imaginary axis so that the unit circle is the common orthogonal between $A_2$ and $A_3$. Here the fixed points of $a_3$ are positive, with the repelling fixed point larger than the attracting one and the fixed point of $a_2$ are negative. Then $\tilde a_2,\tilde a_3$ is given as follows

\begin{align}\label{toop}
\tilde a_2&=\frac{1}{\mathrm{sinh}~\mu}\left(\begin{array}{cc}\mathrm{sinh}~(\mu+\lambda_2) & \mathrm{sinh}~\lambda_2 \\-\mathrm{sinh}~\lambda_2 & \mathrm{sinh}~(\mu-\lambda_2)\end{array}\right),\nonumber\\~\nonumber\\~ \tilde a_3&=\frac{1}{\mathrm{sinh}~\mu}\left(\begin{array}{cc}\mathrm{sinh}~(\mu-\lambda_2) & \mathrm{sinh}~\lambda_2 \\-\mathrm{sinh}~\lambda_2 & \mathrm{sinh}~(\mu+\lambda_2)\end{array}\right),\nonumber\\
\mathrm{where}&\nonumber\\
\mathrm{sinh}^2~\mu&=\frac{2\mathrm{sinh}^2~\lambda_2}{\mathrm{cosh}~\lambda_1+1},~\mu>0.
\end{align}

The handle-closer $d$ which conjugates $a_2$ onto $a_3^{-1}$ while introducing a twist of $\tau$ along $A_2$ is given by

\begin{equation}
\end{equation}

Here $r_2$ is the reflection in $A_2$, $f_{\tau}$ is the twist by $\tau$ along the imaginary axis  and $r_0$ is the reflection in the imaginary axis. The corresponding matrices are given by

\begin{align}\label{toop1}
\tilde r_0= \left(\begin{array}{cc}1 & 0 \\0 & -1\end{array}\right),~ \tilde r_2= \frac{1}{\mathrm{sinh}~\mu}\left(\begin{array}{cc}\mathrm{cosh}~\mu & 1 \\-1 & -\mathrm{cosh}~\mu\end{array}\right),~ \tilde f_{\tau}= \left(\begin{array}{cc}e^{\frac{\tau}{2}} & 0 \\0 & e^{-\frac{\tau}{2}}\end{array}\right).
\end{align}

§.§.§ Matrices for the Universal Twist Map, Interchange and Elementary Conjugators

Now we shall write down the explicit matrices for universal twist map, interchange and elementary conjugators.

Universal Twist Map and Interchange

The explicit matrices generating the twist and interchange are given by

\begin{equation}\label{utmi}
\tilde f_{\tau}=\left(\begin{array}{cc}e^{-\frac{\tau}{2}} & 0 \\0 & e^{\frac{\tau}{2}}\end{array}\right); ~ \tilde g=\left(\begin{array}{cc}0 & -1 \\1 & 0\end{array}\right).
\end{equation}

Elementary Conjugator for $a_{i,1}$

This case occurs only for $i=j=1$. We can choose $\tilde g$ as the matrix for untwisted elementary conjugator $\tilde e_{2,1}^0$. Matrix for twisted elementary conjugator is given by

\begin{equation}\label{ecai1}
\tilde e_{2,1}=\tilde e^0_{2,1}\tilde f_{\tau}=\left(\begin{array}{cc}0 & -e^{\frac{\tau_1}{2}} \\e^{-\frac{\tau_1}{2}} & 0\end{array}\right).
\end{equation}

Elementary Conjugator for $a_{i,2}$

\begin{equation}\label{tecai2}
\tilde e^0_{i,2}=\frac{1}{\sqrt{2\mathrm{sinh}~\mu_i}}\left(\begin{array}{cc}e^{\frac{1}{2}\mu_i} & e^{-\frac{1}{2}\mu_i} \\e^{-\frac{1}{2}\mu_i} & e^{\frac{1}{2}\mu_i}\end{array}\right)
\end{equation}

where $\mu_i$ defined by the equation <ref>  or <ref> depending on the type of $a_{i,3}$.

\begin{equation}\label{ecai2}
\tilde e_{i,2}=\frac{1}{\sqrt{2\mathrm{sinh}~\mu_i}}\left(\begin{array}{cc}e^{\frac{1}{2}(\mu_i-\tau_j)} & e^{-\frac{1}{2}(\mu_i-\tau_j)} \\e^{-\frac{1}{2}(\mu_i+\tau_j)} & e^{\frac{1}{2}(\mu_i+\tau_j)}\end{array}\right)
\end{equation}

Primitive conjugator for $a_{i,3}$

\begin{equation}\label{tecai3}
\tilde e^0_{i,3}=\frac{1}{\sqrt{2\mathrm{sinh}~\nu_i}}\left(\begin{array}{cc}e^{\frac{1}{2}\nu_i+\lambda_i} & e^{-\frac{1}{2}\mu_i} \\e^{-\frac{1}{2}\mu_i}& e^{\frac{1}{2}\mu_i-\lambda_i}\end{array}\right)
\end{equation}

where $\nu_i$ defined by the equation <ref> and $\lambda_i$ is the size of $a_{i,1}$, i.e., $|\mathrm{tr}(a_i,1)|=\mathrm{2 ~cosh}~\lambda_i$.

\begin{equation}\label{ecai3}
\tilde e_{i,3}=\frac{1}{\sqrt{2\mathrm{sinh}~\mu_i}}\left(\begin{array}{cc}e^{\frac{1}{2}(\mu_i-\tau_j)+\lambda_i} & e^{-\frac{1}{2}(\mu_i-\tau_j)} \\e^{-\frac{1}{2}(\mu_i+\tau_j)} & e^{\frac{1}{2}(\mu_i+\tau_j)-\lambda_i}\end{array}\right)
\end{equation}

§.§ The Algorithm

In this subsection, we shall summarize the algorithm for constructing the generators of the Fuchsian group as a function of the Fenchel-Nielsen coordinates. An explicit F-N system on a hyperbolic Riemann surface can be represented by a table. We shall call this  the pairing table for the  F-N system. Pairing table has $q$ row, one for each pair of pants $P_i$. The entry in the $i^{\mathrm{th}}$ row ant $k^{\mathrm{th}}$ column identifies the boundary element $b_{i,k}$ as corresponding to either a coordinate geodesic $L_j$ or boundary element $b_j$ (cusp)  of $\mathcal{R}_0$. We also assume that we are given the appropriate point $\Phi$ in F-N coordinate space. Then the following algorithm gives the explicit matrices.

* For each $i=1,\cdots,q$ and for $k=1,2,3$ we read off from the pairing table whether $b_{i,k}$ corresponds to a coordinate geodesic or to a boundary element of $\mathcal{R}_0$. If $b_{i,k}$ corresponds to coordinate geodesic $L_j$, then we read off the size of $a_{i,k}$ from $j^{\mathrm{th}}$ entry in $\Phi$. Due to the ordering, each $b_{i,1}$ corresponds to an attaching geodesic; we write the size of this geodesic as $\lambda_i$.  Use the constructions of section <ref> to write down the matrices $\tilde a_{i,k}$.

* The first conjugator $c_1$ is the identity. Second conjugator is the elementary conjugator $\tilde e_{2,1}$. Continuing inductively, assume that we have found matrices for the conjugators $c_1,\cdots,c_{j-1},~j\leq q$. The attaching geodesic $L_j$ appears in the pairing table once as $b_{j+1,1}$ and once as $b_{i,k},~i<j+1,~k>1$. Equation <ref> gives the formula for $\tilde c_j$.

* Write the matrices for the first  $q-1$generators. These are the generators corresponding to the attaching geodesics. They are given as,

\begin{equation}\label{q-1g}
\tilde a_1=\tilde a_{1,1},~\tilde a_2=\tilde c_3\tilde a_{3,1}^{-1},~\cdots,~\tilde a_{q-1}=\tilde c_q\tilde a_{q,1}^{-1}\tilde c_q^{-1}.
\end{equation}

* Find  matrices for generators corresponding to the handle geodesics; these are the generators $a_q,\cdots,a_p$. Each $L_j,~ q\leq j \leq p$, appears twice in the pairing table. Either there is some $i$ so that $L_j$ appears as both $b_{i,2}$ and $b_{i,3}$ or there are two distinct rows $i<i'$, so that $L_j$ appears as $b_{i,k}$ and $b_{i',k'}$. In the first case, we write the matrix for the handle generator as $\tilde a_j=\tilde c_i\tilde a_{i,2}\tilde c_i^{-1}$. In the second case, we write the matrix for the handle generator as $\tilde a_j=\tilde c_i \tilde a_{i,k}\tilde c_i^{-1}$.

* Find matrices for handle closing generators $a_{p+1},\cdots, a_{2p-q+1}$. If there is some $j,~q\leq j\leq p,$ so that $L_j$ appears as both $b_{i,2}$ and $b_{i,3}$ in the pairing table, then we write $\tilde a_{p+j}=\tilde d_j$, where $\tilde d_j$ is given by <ref>. If the two entries of $L_j$ in the pairing table appear as both $b_{i,k}$ and $b_{i',k'}$, where $i<i'$, then we write  $\tilde a_{p+j}=\tilde d_j$, where $\tilde d_j$ is given by <ref>.

* Write down the matrices corresponding to the boundary elements of $\mathcal{R}_0$. Each $b_j,~j=1,\cdots,m+n$, appears exactly once in the pairing table. If $b_j$ corresponds to $b_{i,k}$, the n the matrix for the corresponding generator is given by $\tilde a_{2p+q+j}=\tilde c_i \tilde a_{i,k}\tilde c_i^{-1}$.

[1]
A. M. Polyakov,
“Quantum Geometry of Bosonic Strings,”
Phys. Lett. B 103, 207 (1981)
[Phys. Lett.  103B, 207 (1981)].

[2]
A. M. Polyakov,
“Quantum Geometry of Fermionic Strings,”
Phys. Lett. B 103, 211 (1981)
[Phys. Lett.  103B, 211 (1981)].

[3]
D. Friedan, E. J. Martinec and S. H. Shenker,
“Conformal Invariance, Supersymmetry and String Theory,”
Nucl. Phys. B 271, 93 (1986).
doi:10.1016/0550-3213(86)90356-1, 10.1016/S0550-3213(86)80006-2

[4]
M. Dine, N. Seiberg and E. Witten,

Nucl. Phys. B 289, 589 (1987).

[4]
M. Dine, N. Seiberg and E. Witten,

Nucl. Phys. B 289, 589 (1987).

[5]
J. J. Atick, L. J. Dixon and A. Sen,
“String Calculation of Fayet-Iliopoulos d Terms in Arbitrary Supersymmetric Compactifications,”
Nucl. Phys. B 292, 109 (1987).

[6]
M. Dine, I. Ichinose and N. Seiberg,
“F Terms and d Terms in String Theory,”
Nucl. Phys. B 293, 253 (1987).

[7]
M. B. Green and N. Seiberg,
“Contact Interactions in Superstring Theory,”
Nucl. Phys. B 299, 559 (1988).

[8]
A. Belopolsky,
“De Rham cohomology of the supermanifolds and superstring BRST cohomology,”
Phys. Lett. B 403, 47 (1997)

“New geometrical approach to superstrings,”

“Picture changing operators in supergeometry and superstring theory,”

[9]
E.D'Hoker and D.H.Phong; Two loop superstrings. I. Main formulas; Phys. Lett. B 529, 241 (2002)
[hep-th/0110247]; II. The Chiral measure on moduli space;
Nucl. Phys. B 636, 3 (2002) [hep-th/0110283]; III. Slice independence and absence of ambiguities;
Nucl. Phys. B 636, 61 (2002) [hep-th/0111016]; IV: The Cosmological constant and modular forms; Nucl. Phys. B 639, 129 (2002)
[hep-th/0111040]; V. Gauge slice independence of the N-point function;
Nucl. Phys. B 715, 91 (2005)
[hep-th/0501196]; VI: Non-renormalization theorems and the 4-point function;
Nucl. Phys. B 715, 3 (2005)
[hep-th/0501197]; VII. Cohomology of Chiral Amplitudes; Nucl. Phys. B 804, 421 (2008) [arXiv:0711.4314 [hep-th]]

[10]
A. Berera,

Nucl. Phys. B 411, 157 (1994).

[11]
E. Witten,
“Notes On Supermanifolds and Integration,”
arXiv:1209.2199 [hep-th];
“Notes On Super Riemann Surfaces And Their Moduli,”
arXiv:1209.2459 [hep-th];
“More On Superstring Perturbation Theory,”
arXiv:1304.2832 [hep-th];
“Notes On Holomorphic String And Superstring Theory Measures Of Low Genus,”
arXiv:1306.3621 [hep-th];
“The Feynman $i \epsilon$ in String Theory,”
arXiv:1307.5124 [hep-th].

[12]
R. Donagi and E. Witten,
“Supermoduli Space Is Not Projected,”
arXiv:1304.7798 [hep-th];
R. Donagi and E. Witten, “Super Atiyah classes and obstructions to splitting of supermoduli space,”
arXiv:1404.6257 [hep-th].

[13]
R. Pius, A. Rudra and A. Sen,
“Mass Renormalization in String Theory: Special States,”
JHEP 1407, 058 (2014)
[arXiv:1311.1257 [hep-th]].

[14]
R. Pius, A. Rudra and A. Sen,
“Mass Renormalization in String Theory: General States,”
JHEP 1407, 062 (2014)
[arXiv:1401.7014 [hep-th]].

[15]
A. Sen,
“Off-shell Amplitudes in Superstring Theory,”
Fortsch. Phys.  63, 149 (2015)
[arXiv:1408.0571 [hep-th]].
[16]
A. Sen and E. Witten,
“Filling the gaps with PCO?s,”
JHEP 1509, 004 (2015)
[arXiv:1504.00609 [hep-th]].

[17]
R. Pius, A. Rudra and A. Sen,
“String Perturbation Theory Around Dynamically Shifted Vacuum,”
JHEP 1410, 70 (2014)
[arXiv:1404.6254 [hep-th]].

[18]
A. Sen,
“Gauge Invariant 1PI Effective Action for Superstring Field Theory,”
JHEP 1506, 022 (2015)
[arXiv:1411.7478 [hep-th]].

[19]
A. Sen,
“Gauge Invariant 1PI Effective Superstring Field Theory: Inclusion of the Ramond Sector,”
JHEP 1508, 025 (2015)
[arXiv:1501.00988 [hep-th]].

[20]
A. Sen,
“Supersymmetry Restoration in Superstring Perturbation Theory,”
JHEP 1512, 075 (2015)
[arXiv:1508.02481 [hep-th]].

[21]
A. Sen,
“BV Master Action for Heterotic and Type II String Field Theories,”
JHEP 1602, 087 (2016)
[arXiv:1508.05387 [hep-th]].

[22]
R. Pius and A. Sen,
“Cutkosky Rules for Superstring Field Theory,”
JHEP 1610, 024 (2016)
[arXiv:1604.01783 [hep-th]].

[23]
A. Sen,
“Equivalence of Two Contour Prescriptions in Superstring Perturbation Theory,”
arXiv:1610.00443 [hep-th].

[24]
A. Sen,
“Reality of Superstring Field Theory Action,”
JHEP 1611, 014 (2016)
[arXiv:1606.03455 [hep-th]].

[25]
A. Sen,
“Unitarity of Superstring Field Theory,”
arXiv:1607.08244 [hep-th].

[26]
A. Sen,
“Wilsonian Effective Action of Superstring Theory,”
arXiv:1609.00459 [hep-th].

[27]
C. de Lacroix, H. Erbin, S. P. Kashyap, A. Sen and M. Verma,
“Closed Superstring Field Theory and its Applications,”
arXiv:1703.06410 [hep-th].

[28]
Verlinde, Erik, and Herman Verlinde. "Multiloop calculations in covariant superstring theory." Physics Letters B 192.1-2 (1987): 95-102.

[29]
Zwiebach, Barton. “Closed string field theory: Quantum action and the Batalin-Vilkovisky master equation." Nuclear Physics B 390.1 (1993): 33-152.

[30]
Polchinski, Joseph. “String theory. Vol. 1: An introduction to the bosonic string." Cambridge, UK: Univ. Pr 402 (1998).

[31]
D. J. Gross, J. A. Harvey, E. J. Martinec and R. Rohm,
“The Heterotic String,”
Phys. Rev. Lett.  54, 502 (1985).

[32]
Maskit, Bernard. “Matrices for Fenchel-Nielsen coordinates." RECON no. 20010088230.; Annales Academiae Scientiarum Fennicae: Mathmatica 26.2 (2001): 267-304.

[33]
Hatcher, Allen, and William Thurston. “A presentation for the mapping class group of a closed orientable surface." Topology 19.3 (1980): 221-237.

[34]
Hatcher, Allen. “Pants decompositions of surfaces." arXiv preprint math/9906084 (1999).

[35]
McShane, Greg. “Simple geodesics and a series constant over Teichmuller space." Inventiones mathematicae 132.3 (1998): 607-632.

[36]
M. Mirzakhani, Simple Geodesics and Weil-Petersson Volumes of Moduli Spaces of Bordered Riemann Surfaces, Inventiones mathematicae 167.1 (2007): 179-222.

[37]
Schiffer, Menahem, and Donald C. Spencer. “Functionals of finite Riemann surfaces”. Courier Corporation, 2014.

[38]
Gardiner, Frederick P. “Schiffer's interior variation and quasiconformal mapping." Duke Mathematical Journal 42.2 (1975): 371-380.

[39]
Wolpert, Scott A. “Asymptotics of the spectrum and the Selberg zeta function on the space of Riemann surfaces." Communications in Mathematical Physics 112.2 (1987): 283-315.

[40]
Deligne, Pierre, and David Mumford. “The irreducibility of the space of curves of given genus." Publications Mathématiques de l'IHES 36 (1969): 75-109.

[41]
Wolpert, Scott A. “The hyperbolic metric and the geometry of the universal curve." J. Differential Geom 31.2 (1990): 417-472.

[42]
Obitsu, Kunio, and Scott A. Wolpert. “Grafting hyperbolic metrics and Eisenstein series." Mathematische Annalen 341.3 (2008): 685-706.

[43]
R. Melrose, and X. Zhu. "Boundary Behaviour of Weil-Petersson and Fiber Metrics for Riemann Moduli Spaces." arXiv preprint arXiv:1606.01158 (2016).

[44]
Wolpert, Scott A. “Families of Riemann surfaces and Weil-Petersson geometry”. No. 113. American Mathematical Soc., 2010.

[45]
Wolpert, Scott. “The Fenchel-Nielsen deformation." Annals of Mathematics 115.3 (1982): 501-528.

[46]
Wolpert, Scott. “On the symplectic geometry of deformations of a hyperbolic surface." Annals of Mathematics (1983): 207-234.

[47]
Wolpert, Scott. “On the Weil-Petersson geometry of the moduli space of curves." American Journal of Mathematics 107.4 (1985): 969-997.

[48]
Randol, Burton. “Cylinders in Riemann surfaces." Commentarii Mathematici Helvetici 54.1 (1979): 1-5.

[49]
Matelski, J. Peter. “A compactness theorem for Fuchsian groups of the second kind." Duke Math. J 43.4 (1976): 829-840.

[50]
Fenchel, Werner, and Jakob Nielsen. “Discontinuous groups of non-Euclidean motions". 1948.

[51]
Imayoshi, Yoichi, and Masahiko Taniguchi. “An Introduction to Teichmüller spaces." Springer Science & Business Media, 2012.

[52]
Maskit, Bernard. “Kleinian groups”. Vol. 287. Springer Science & Business Media, 2012.

[53]
Lechtenfeld, Olaf. “Superconformal ghost correlations on Riemann surfaces." Physics Letters B 232.2 (1989): 193-198.

[54]
Morozov, A. “A straightforward proof of Lechtenfeld's formula for the $\beta\gamma$-correlator." Physics Letters B 234.1-2 (1990): 15-17.

[55]
E. Witten. “Two-dimensional gravity and intersection theory on moduli space." Surveys in Diff. Geom 1.243 (1991): 74.

[56]
Kontsevich, Maxim. "Intersection theory on the moduli space of curves and the matrix Airy function." Communications in Mathematical Physics 147.1 (1992): 1-23.

[57]
Mirzakhani, Maryam. “Weil-Petersson volumes and intersection theory on the moduli space of curves." Journal of the American Mathematical Society 20.1 (2007): 1-23.

[58]
R.Dijkgraaf, E.Verlinde, and H.Verlinde. “Notes on topological string theory and 2D quantum gravity." Proc. of the Trieste Spring School. 1990.

[59]
W.Abikoff; “The Uniformization theorem" Amer. Math. Monthly 88 (1981), no. 8,

[60]
S. F. Moosavian and R. Pius,
“Hyperbolic Geometry and Closed Bosonic String Field Theory I: The String Vertices Via Hyperbolic Riemann Surfaces,”
arXiv:1706.07366 [hep-th].

[61]
S. F. Moosavian and R. Pius,
“Hyperbolic Geometry and Closed Bosonic String Field Theory II: The Rules for Evaluating the Quantum BV Master Action,”
arXiv:1708.04977 [hep-th].

[62]
R. Pius,
“Quantum Closed Superstring Field Theory and Hyperbolic Geometry I: Construction of String Vertices,”
arXiv:1808.09441 [hep-th].

[63]
S. F.  Moosavian, “Construction of the Effective Actions of Closed Superstring Field Theories Using Hyperbolic Geometry,” to be published.

[64]
T. Erler, S. Konopka and I. Sachs,
“One Loop Tadpole in Heterotic String Field Theory,”
JHEP 1711, 056 (2017)
[arXiv:1704.01210 [hep-th]].

[65]
P. C. Nelson,
“Covariant Insertion of General Vertex Operators,”
Phys. Rev. Lett.  62, 993 (1989).


# Independent Ethical Assessment of Text Classification Models: A Hate Speech
Detection Case Study

Amitoj Singh  Emerging Technology PwCPwC Acceleration CenterMumbaiIndia ,
Jingshu Chen  PwC Acceleration Center ShanghaiPudong New DistrictShanghaiChina
,  Lihao Zhang  PwC Acceleration Center ShanghaiPudong New
DistrictShanghaiChina ,  Amin Rasekh  Product Innovation PwCWashingtonDCUSA ,
Ilana Golbin  Emerging Technology PwCLos AngelesCAUSA and  Anand S. Rao
Global AI Leader PwCBostonMAUSA

(2021)

###### Abstract.

An independent ethical assessment of an artificial intelligence system is an
impartial examination of the system’s development, deployment, and use in
alignment with ethical values. System-level qualitative frameworks that
describe high-level requirements and component-level quantitative metrics that
measure individual ethical dimensions have been developed over the past few
years. However, there exists a gap between the two, which hinders the
execution of independent ethical assessments in practice. This study bridges
this gap and designs a holistic independent ethical assessment process for a
text classification model with a special focus on the task of hate speech
detection. The assessment is further augmented with protected attributes
mining and counterfactual-based analysis to enhance bias assessment. It covers
assessments of technical performance, data bias, embedding bias,
classification bias, and interpretability. The proposed process is
demonstrated through an assessment of a deep hate speech detection model.

natural language processing, classification, responsible AI, ethical
assessment, fairness

††copyright: acmcopyright††journalyear: 2021††doi:
10.1145/1122445.1122456††conference: KDD ’21: SIGKDD International Conference
on Knowledge Discovery and Data Mining; August 14–18, 2021;
Singapore††booktitle: KDD ’21: SIGKDD International Conference on Knowledge
Discovery and Data Mining, August 14–18, 2021, Singapore††price: 15.00††isbn:
978-1-4503-XXXX-X/18/06††ccs: Computing methodologies Supervised learning by
classification††ccs: Computing methodologies Natural language processing

##  1\. Introduction

Internal and external independent assessments of artificial intelligence (AI)
systems are conducted to examine quality, responsibility and accountability
for their development, deployment, and use (on Artificial Intelligence, 2020;
Madiega, 2019). Major ethical concerns featuring language models have recently
attracted a lot of attention for the need of accountability in this domain for
example: Microsoft’s AI Chatbot Tay tweeted racists comments when trained on
Twitter Data and GPT-3 AI generated blog hit no. 1 on Hacker News which was
completely fake, hinting a mass misinformation threat. Independence in this
context is defined as the freedom from conditions that may jeopardize the
ability of the assessment activity to fulfill assessment responsibilities in
an impartial manner (of Internal Auditors, 2021). A three-level classification
of independent assessments is provided in Appendix A.

An independent ethical assessment requires both system-level qualitative
guidelines that describe the process and requirements as well as component-
level quantitative tools that enable the calculation of assessment metrics
related to individual ethical dimensions. The Assessment List for Trustworthy
Artificial Intelligence (on Artificial Intelligence, 2020) and the Artificial
Intelligence Auditing Framework (of Internal Auditors, 2017) are examples of
such qualitative guidelines, amongst others (Audit and (ISACA), 2018; Raji et
al., 2020; Office, 2020). Quantitative models that focus on different ethical
components have also been developed, such as the techniques for bias detection
in embedding models by (Papakyriakopoulos et al., 2020; Garg et al., 2018;
Rozado, 2020) and the metrics for evaluating bias in text classification
datasets and models by (Dixon et al., 2018; Blodgett et al., 2020; Hutchinson
et al., 2020; Park et al., 2018; Sap et al., 2019). Quantitative measures
require desired metrics, which must be defined by the context, not by the
framework.

There exists a gap between system-level qualitative guidelines and component-
level quantitative models that hinders execution of independent ethical
assessments in practice (Brown et al., 2021). This study bridges this gap for
text classification systems in the context of hate speech detection models.
While hate speech detection aims to serve the positive cause of reducing
hateful content, it also poses the risk of silencing harmless content, or even
worse, doing so with a false positive bias and without an ability to provide
an explanation of its reasoning.

We summarize our main contributions as follows:

  * •

We develop a process for independent ethical assessments of text
classification systems by bridging high-level qualitative guidelines and low-
level technical models.

  * •

We propose methods to mine protected attributes from unstructured data.

  * •

We demonstrate approaches and metrics for quantifying bias in data, word
embeddings, and classification models.

The paper is organized as follows. Section 2 presents the proposed assessment
process. Section 3 describes the text classification system under
consideration and its technical assessment. Section 4 presents the data bias
assessment using two different approaches. Section 5 describes the embedding
bias metrics and assessment. Section 6 presents the classification bias
assessment using the original, swapped, and synthetic datasets. Section 7
describes the local and global interpretability assessment. Section 8
concludes the study.

##  2\. Process Description

This process performs quantitative assessments to address the qualitative
requirements set by the Assessment List for Trustworthy Artificial
Intelligence (ALTAI) by the European Commission (on Artificial Intelligence,
2020). The classification accuracy assessment addresses Requirement #2
Technical Robustness and Safety. Bias assessments are conducted for the data,
word embeddings, and classification model, which correspond to Requirement #5
Diversity, Non-discrimination and Fairness. ALTAI defines bias as “systematic
and repeatable errors in a computer system that create unfair outcomes, such
as favoring one arbitrary group of users over others.” Data and embedding bias
are possible causes of unfairness, and may influence classification bias,
which is an actual measure of fairness. Protected attributes are defined as
those qualities, traits, or characteristics, such as gender, that cannot be
discriminated against. Model interpretability assessments are performed to
address Requirement #4 Transparency. Since the AI system under consideration
is not computationally intensive and was trained on a laptop, its
environmental impact is deemed to be negligible. Nevertheless, Requirement #6
Societal and Environmental Well-being is briefly addressed in Appendix I.
Indeed, this process does not cover all requirements and future work is
necessary to address the remainder.

##  3\. The AI System and Technical Assessment

The system under assessment is a pre-trained natural language processing (NLP)
system that identifies hateful comments on a social media platform. The system
has been trained on a dataset of Twitter posts in English and classifies
comments/posts as hateful or non-hateful. It is a proof of concept model that
was developed by a separate team within the same organization. Information
about the dataset and model are provided in Appendix B.

Requirement #2 in ALTAI states that reliable technical performance of the
system is one of the critical requirements for trustworthy AI. We assessed the
performance of the model on the test dataset using various metrics. The
results are presented in Table 1 and the accuracy of the model is 88%.

Table 1. Technical performance assessment of the AI system. Macro and weighted averages are calculated through averaging the unweighted mean and the support-weighted mean per label, respectively. | Precision | Recall | F1-Score | Support
---|---|---|---|---
Not-hateful | 0.78 | 0.92 | 0.84 | 1,031
Hateful | 0.95 | 0.87 | 0.91 | 2,502
Macro Avg. | 0.87 | 0.89 | 0.87 | 3,803
Weighted Avg. | 0.89 | 0.88 | 0.88 | 3,803

##  4\. Data Bias Assessment

Requirement #5 in ALTAI outlines the need for an assessment of the input data
for bias. This data bias assessment step is motivated by the possible impact
that data bias may have on the fairness of the classification model. In the
case of structured datasets, correlations between protected attribute features
and decision variables can be analyzed to assess data bias. For unstructured
data like text, however, this approach may not be possible because protected
attributes are not represented as separate, standalone features. This study
uses two approaches to tackle this problem and assess bias in the training
data.

In the first approach, which is inspired by (Dixon et al., 2018), a curated
set of identity terms are first identified and their frequencies across
hateful and all the comments in the training dataset are then found and
compared. The results for all the identity terms are provided in Appendix C.
They indicate that some of identity terms such as “gay” and “white” are
disproportionately used in hateful comments and this could be an indicative of
false positive bias.

The second approach is based on the frequency of protected attribute
references across hateful comments and overall. Protected attribute mining is
first conducted to identify references made to each subgroup of each attribute
in the comments (as elaborated upon in Section 6.2 below). Their frequencies
across the hateful comments and all the comments is then calculated and
compared. The results are shown in Table 2 for the protected attributes of
gender and religion. The data shows some bias is present in the data such as
for the “Islam” subgroup and the ”female” subgroup. The word Islam, in our
dataset, has a larger presence in the hateful comments as compared word
Christianity. Same holds true for the word female compared to the word male.

Table 2. Frequency of protected attribute references in hateful and not-hateful comments and overall for the protected attributes of gender and religion Protected Attribute | Subgroup |  Hateful | Not-hateful |  Overall
---|---|---|---|---
Gender | Female |  12.28% | 11.53% |  12.02%
| Male |  11.51% | 17.37% |  13.52%
Religion | Islam |  0.10% | 0.15% |  0.12%
| Christianity |  0.07% | 0.24% |  0.13%

##  5\. Embedding Bias Assessment

The embedding model should be assessed for bias as it influences the
classification model’s decisions, which relates to Requirement #5 in ALTAI.
This embedding bias assessment is motivated by the potential influence that
embedding bias has on the fairness of the classification model. To assess
embedding bias, we analyze similarity between a diverse set of 501 neutral
words (e.g., “admirable” and “miserable”) (Garg et al., 2018) and word groups
associated with three protected attributes of gender, religion, and ethnicity
(Appendix D). Neutral words mean they are expected to be similarly correlated
across protected attribute words (Garg et al., 2018). The similarity metric
used is cosine similarity between the individual terms’ embeddings and is
aggregated using average mean absolute error (AMAE) and average root mean
squared error (ARMSE).

Given a protected attribute (e.g., gender) and a subgroup within (e.g.,
female), the similarity between an individual neutral word and that subgroup
is calculated by averaging distances for the individual neutral word and
subgroup word, such as, for example, “nice” and “she”. This is done for all
the neutral words and subgroups. Then the AMAE and ARMSE of the average cosine
similarity across the subgroups is calculated (the equations are provided in
Appendix E). This is performed for all the protected attributes and the
results are illustrated in Figure 1. The magnitudes of the AMAE and ARMSE show
how strong the biases are with that particular category. In principal, the 2
measures are similar to the statistical measure of ”variance” of cosine-
distances across the various sub-groups in a category. Therefore, higher the
measure, greater is the disparity in association and more biased are the
embeddings for that category.

Figure 1. Embedding bias magnitude across different protected attributes

The AMAE and ARMSE scores indicate that the embedding model is more biased as
it relates to religion terms but less for gender terms. Appendix E provides a
comparison of the bias magnitude for this system’s embedding model against
other mainstream embedding models.

##  6\. Classification Bias Assessment

While classification accuracy at an aggregate level is related to ALTAI
Requirement #2 Technical Robustness and Safety, an accuracy report broken down
by various protected attributes would address Requirement #5 Diversity, Non-
discrimination and Fairness. In contrast to data and embedding bias,
classification bias may directly impact the system’s stakeholders.

To assess this bias, subgroups based on gender, ethnicity, religion, etc. are
first identified in different comments. Bias assessment is then performed to
understand how the model’s prediction accuracy differs across these protected
attributes. This assessment can be enhanced by measuring changes in the
model’s accuracy as the subgroups in the comments are swapped (for example,
does the model’s true positive prediction for ”men are universally terrible”
changes to a false negative if ”men” is swapped with ”women”?). Synthetic
sentences containing protected attribute references are also generated and
analyzed for a better understanding of how the model learns biases.

###  6.1. Protected Attributes Mining

To extract protected attributes from text, two approaches are applied in this
study: 1) a look-up based approach and 2) a named-entity recognition (NER)
based approach. These approaches have been validated using datasets with
human-annotated comments (Dixon et al., 2018). The look-up approach is based
on the word counts extracted from the text examples and the words are looked
up in a list of common identity terms such as pronouns and professions. The
NER based approach uses spaCy’s pre-trained NER model to extract entities
(i.e. protected attributes). A named entity is a “real-world object” with a
type associated with it, which might be a country, a product, a religion, etc.
In this study, entities tagged with the “NORP” type are extracted. “NORP”
refers to the “nationalities or religious or political groups”. spaCy’s NER
model takes into account the context and part of speech to determine the tag
for a given word. Gender and religion are the protected attributes addressed
in this subsection. Several examples of protected attribute extractions from
comments are included in Appendix G.

###  6.2. Bias Assessment based on Protected Attributes Mining

The PwC Responsible AI (RAI) Toolkit performs a bias assessment using
protected attribute references, ground-truth labels, and prediction
probabilities (Rao et al., 2019). It checks whether the model discriminates
against various groups or individuals using a variety of bias metrics. Table 3
and Figure 2 show the results for gender. There were 501 and 442 references to
male and female subgroups, respectively. Appendix F provides a description of
the seven bias metrics used.

Table 3. Average prediction probabilities for comments with gender references Actual Comment Type | Subgroup | Avg. Predicted Probability
---|---|---
Not-hateful | Female | 14.0% hateful
Not-hateful | Male | 15.4% hateful
Hateful | Female | 85.6% hateful
Hateful | Male | 80.4% hateful

Figure 2. Bias assessment based on protected attribute extraction for gender

###  6.3. Assessment Based on Swapped Protected Attributes

Versions of the comments where identity terms are swapped help address the
limitations of relying solely on the ground-truth data such as in the case of
disproportionate representation of one subgroup over another. Examples are
provided in G.

A bias assessment can be performed using the actual label and the predicted
probabilities before and after identity swapping. We define the term ’favor’
as the model’s confidence in its prediction one way or the other. For not-
hateful comments, a decrease in the prediction probability after swapping
identities (e.g. male to female) means the model favors the new identity. For
hateful comments, an increase in the prediction probability after swapping
identities (e.g. male to female) means the model favors the new identity. This
process of swapping is repeated for all the comments with the protected
attribute references. This bias assessment indicated that for 55.9% of the
comments the model favored the female subgroup, while in 43.4% of the comments
it favored the male subgroup. The model prediction did not change for 0.7% of
the comments with probabilities rounded to four decimal places.

###  6.4. Bias Assessment Based on Synthetic Counterfactual Dataset

Bias assessment can also be performed using a synthetic counterfactual
dataset. This method offers greater flexibility but is labor-intensive and the
resulting dataset may not be representative of the actual texts the model
encounters in practice. This idea is related to counterfactual fairness, which
as stated by (Kusner et al., 2017), “captures the intuition that a decision is
fair towards an individual if it is the same in (a) the actual world and (b) a
counterfactual world where the individual belonged to a different demographic
group”. Examples of such counterfactuals for religion are provided in Appendix
G.

Once the synthetic counterfactual dataset is generated, a bias assessment,
similar to the above swapped dataset assessment, can be conducted. We used
several templates (some inspired from (Dixon et al., 2018)) for gender and
religion. The average prediction probabilities (Appendix G), indicate that
there appears to be a bias towards treating all talk of Islam as more hateful
compared to Christianity.

We use the following threshold-insensitive counterfactual bias metric for each
reference subgroup:

(1)
$CB(\mathbf{X},O,\mathbf{y})=\sum_{i=1}^{|\mathbf{X}|}(\hat{p}_{\mathbf{x_{i}}}-\frac{1}{|O|}\sum^{|O|}_{j=1}\hat{p}_{O_{ij}})\times
g(y_{i})$

where $\mathbf{X}$ is a series of reference group (e.g. female) examples (e.g.
(”she is kind”, ”women are universally terrible”, $\ldots$). $O$ is a series
of sets of counterfactual examples of the form ((”he is kind”, $\ldots$),
(”men are universally terrible”, $\ldots$)).
$\hat{p}_{\mathbf{x}_{i}}=f(\mathbf{x}_{i})$ (i.e. the model’s predicted
probability). $\mathbf{y}$ is a series of labels for the reference group
examples, where 0s correspond to not-hateful comments and 1s corresponds to
hateful comments. $g(y_{i})$ is -1 when $y_{i}=0$ and 1 when $y_{i}=1$ and is
a term that corrects for the direction of the bias. For the model under
consideration, the preferred label is not-hateful and therefore $g(y)$ is -1
for not-hateful comments. A positive value of $CB$ means the model favors the
reference subgroup.

Setting “Islam” as the reference subgroup returns a bias value of -0.0067,
which denotes the model slightly favors the “Christianity” subgroup. For
gender, setting “Male” as the reference subgroup would return a bias value of
0.0737. A comprehensive assessment may also include other bias metrics such as
equalized odds (Hardt et al., 2016).

##  7\. Classification Interpretability Assessment

Requirement #4 in ALTAI focuses on transparency requirements and describes
explainability as one of the elements to consider alongside traceability and
open communication. The question is simple - why are certain tweets being
classified as hate-speech? Understanding hate in speech is a function of the
vocabulary and the context of the language used and we as humans have an
innate ability to understand hate. We seek to understand how an AI model
infers it. Does it consider the vocabulary? If yes, how does it do so since we
have not explicitly fed the model a list of abusive words or profane phrases.
Does it understand the context? If yes, how does it distinguish between cases
where strong words can be used as a way of free-speech or expression (e.g. _“I
am Gay”_) versus cases where the words are used disparagingly as hateful
comments directed at someone? We explored two methods to address these
questions. Overall, the two explainability packages explored offer end-users a
unique perspective to tie model decisions with the vocabulary used in a
particular tweet. This may help automate, at least partially, some of the
content modulation tasks Social Media players are trying to do and at the same
time, provide powerful measurable evidence for driving hate.

The first method focuses on local interpretability and uses LIME, which is a
model agnostic method that helps explain a given prediction through the
features used in the model (Ribeiro et al., 2016). In the case of a text
classification model, LIME assigns each token in a given sentence an
importance score which can either be positive or negative, depending on the
word’s effect on the prediction. For example, in the tweet shown in Figure 3,
LIME shows that words like “vote” and “liberals” contribute to the model
predicting non-hate, while words like “filthy” and “sick” push the model to
classify as a hate-comment. LIME can be used to look at incidents of hate
tweets as shown in Fig 3. to understand how the AI model determines hate. This
can be of important use for the en-user to better analyze the classification
and what kind of vocabulary is too strong for the model to predict the comment
as non-hateful. Here the use of strong-language was the key for the model to
predict hate.

Figure 3. Local interpretability of an example prediction using LIME

The second method concerns global interpretability and uses SHAP values. SHAP
is a model-agnostic explainability tool that generates feature importances
using the overall datasets. It produces various permutations over feature
selection to get an estimate of the importance of a particular feature on the
overall model (Lundberg and Lee, 2017). The global interpretability results
are provided in Appendix H. The results show at an overall level that strong-
language is what drives model in predicting hate. The keywords listed here had
the largest impact on model’s detection of hate-speech.

##  8\. Conclusions

This study proposed a process for independent ethical assessments of text
classification systems and demonstrated it through a first-party assessment of
a hate speech detection model. While the ALTAI recommendations are largely
applicable to text classification systems, they are underspecified, leaving a
looming implementation gap. This study addresses that gap through a series of
component-wise quantitative assessments. A discussion of the other
requirements with ideas for addressing them are provided in the Appendix I.
Risk mitigation is also a logical next step to any risk assessment process.
This work serves as a concrete blueprint for the design and implementation of
independent ethical assessment processes for text classification systems. More
work is needed to apply these ideas to other domains.

###### Acknowledgements.

We wish to thank Todd Morrill for thorough feedback on this article.

## References

  * (1)
  * Audit and (ISACA) (2018) Information Systems Audit and Control Association (ISACA). 2018\.  _Auditing Artificial Intelligence_.  https://ec.europa.eu/futurium/en/system/files/ged/auditing-artificial-intelligence.pdf
  * Blodgett et al. (2020) Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020.  Language (technology) is power: A critical survey of” bias” in nlp.  _arXiv preprint arXiv:2005.14050_ (2020).
  * Brown et al. (2021) Shea Brown, Jovana Davidovic, and Ali Hasan. 2021\.  The algorithm audit: Scoring the algorithms that score us.  _Big Data & Society_ 8, 1 (2021), 2053951720983865\.
  * Davidson et al. (2017) Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017.  Automated hate speech detection and the problem of offensive language. In _Proceedings of the International AAAI Conference on Web and Social Media_ , Vol. 11.
  * Dixon et al. (2018) Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018.  Measuring and mitigating unintended bias in text classification. In _Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society_. 67–73.
  * eurostat Statistics Explained (2017) eurostat Statistics Explained. 2017\.  _Glossary:Carbon dioxide equivalent_.  Retrieved May 19, 2021 from https://ec.europa.eu/eurostat/statistics-explained/index.php?title=Glossary:Carbon_dioxide_equivalent#:~:text=A%20carbon%20dioxide%20equivalent%20or,with%20the%20same%20global%20warming
  * Garg et al. (2018) Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2018.  Word embeddings quantify 100 years of gender and ethnic stereotypes.  _Proceedings of the National Academy of Sciences_ 115, 16 (2018), E3635–E3644.
  * Hardt et al. (2016) Moritz Hardt, Eric Price, and Nathan Srebro. 2016.  Equality of opportunity in supervised learning.  _arXiv preprint arXiv:1610.02413_ (2016).
  * Hutchinson et al. (2020) Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020.  Social biases in NLP models as barriers for persons with disabilities.  _arXiv preprint arXiv:2005.00813_ (2020).
  * Kazim et al. (2021) Emre Kazim, Danielle Mendes Thame Denny, and Adriano Koshiyama. 2021\.  AI auditing and impact assessment: according to the UK information commissioner’s office.  _AI and Ethics_ (2021), 1–10.
  * Kusner et al. (2017) Matt J Kusner, Joshua R Loftus, Chris Russell, and Ricardo Silva. 2017.  Counterfactual fairness.  _arXiv preprint arXiv:1703.06856_ (2017).
  * Lacoste et al. (2019) Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019.  Quantifying the carbon emissions of machine learning.  _arXiv preprint arXiv:1910.09700_ (2019).
  * Lundberg and Lee (2017) Scott Lundberg and Su-In Lee. 2017.  A unified approach to interpreting model predictions.  _arXiv preprint arXiv:1705.07874_ (2017).
  * Madiega (2019) T Madiega. 2019\.  EU guidelines on ethics in artificial intelligence: Context and implementation.  _European Parliamentary Research Service, PE_ 640 (2019).
  * of Internal Auditors (2021) Institute of Internal Auditors. 2021\.  _International Professional Practices Framework (IPPF)_.  Institute of Internal Auditors.
  * of Internal Auditors (2017) The Institute of Internal Auditors. 2017\.  _Artificial Intelligence Auditing Framework_.  https://na.theiia.org/periodicals/Public%20Documents/GPI-Artificial-Intelligence-Part-II.pdf
  * Office (2020) Information Commissioner’s Office. 2020\.  _Guidance on the AI auditing framework_.  https://ico.org.uk/media/about-the-ico/consultations/2617219/guidance-on-the-ai-auditing-framework-draft-for-consultation.pdf
  * on Artificial Intelligence (2020) High-Level Expert Group on Artificial Intelligence. 2020.  _Assessment List for Trustworthy Artificial Intelligence (ALTAI)_.  https://doi.org/10.2759/791819
  * Papakyriakopoulos et al. (2020) Orestis Papakyriakopoulos, Simon Hegelich, Juan Carlos Medina Serrano, and Fabienne Marco. 2020\.  Bias in word embeddings. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_. 446–457.
  * Park et al. (2018) Ji Ho Park, Jamin Shin, and Pascale Fung. 2018.  Reducing gender bias in abusive language detection.  _arXiv preprint arXiv:1808.07231_ (2018).
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation. In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_. 1532–1543.
  * Raji et al. (2020) Inioluwa Deborah Raji, Andrew Smart, Rebecca N White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. 2020\.  Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_. 33–44.
  * Rao et al. (2019) A Rao, F Palaci, and W Chow. 2019.  _A practical guide to Responsible Artificial Intelligence (AI)_.  Technical Report. PwC.
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016\.  ” Why should i trust you?” Explaining the predictions of any classifier. In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_. 1135–1144.
  * Rozado (2020) David Rozado. 2020\.  Wide range screening of algorithmic bias in word embedding models using large sentiment lexicons reveals underreported bias types.  _PloS one_ 15, 4 (2020), e0231189.
  * Russell (2019) JP Russell. 2019\.  _The ASQ auditing handbook_.  ASQ Quality Press.
  * Sap et al. (2019) Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A Smith. 2019.  The risk of racial bias in hate speech detection. In _Proceedings of the 57th annual meeting of the association for computational linguistics_. 1668–1678.
  * Wachter et al. (2021) Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2021\.  Why fairness cannot be automated: Bridging the gap between EU non-discrimination law and AI.  _Computer Law & Security Review_ 41 (2021), 105567.
  * Zampieri et al. (2019) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019\.  Predicting the type and target of offensive posts in social media.  _arXiv preprint arXiv:1902.09666_ (2019).
  * Zhao et al. (2018) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018.  Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , Vol. 2.

##  Appendix A A three-level independent assessment classification

The three-level classification that is common in the audit and assurance
landscape (Russell, 2019) can be adapted.

A first-party ethical assessment of an AI system is an internal assessment
conducted by a team within the same organization that owns the AI system but
who had no role in its development nor has any vested interest in its
assessment results.

A second-party assessment is an external assessment conducted by a contracted
party outside the organization on behalf of a customer.

A third-party assessment is an external assessment by a party independent of
the organization-customer relationship and assesses the level of conformity of
the AI system to certain ethical criteria and standards (Russell, 2019).

The AI system development team and the AI system assessment team are the two
primary groups involved in the process for a first-party ethical assessment.
The two teams are independent of each other, although they are still within
the same organization. The AI system development team is analogous to the
first and second lines in the IIA’s Three Line Model (of Internal Auditors,
2021), which are the creators, executors, operators, managers, and supervisors
who are in charge of designing, building, and deploying the AI system as well
as risk assessment and strategy. The AI system assessment team is related
closely to the third line, which are the auditors and ethicists who assess the
system against the ethical requirements.

##  Appendix B Dataset and model description

_The text dataset:_ The dataset has been created by combining two public
datasets from (Davidson et al., 2017; Zampieri et al., 2019) and used for the
purpose of training the hate speech classifier. The model was trained on
34,220 Tweets and an unseen test dataset of 3803 Tweets was also provided to
us.

_The word embedding model:_ The classifier uses the pretrained Stanford Glove
Common Crawl word embeddings with 300 dimensions (Pennington et al., 2014) for
training and predictions.

_The classification model:_ The classifier is a convolutional neural network
model that takes word embeddings as input. The network architecture is a 1D
CNN (32 filters with a kernel size of 17), followed by a max pooling (pool
size =4) layer and then two fully connected dense Layers (25 unit output
followed by a 1 unit output). The assessment here uses the pretrained model as
is (i.e., only to make predictions where needed) and the whole process is
repeatable regardless of knowing the architecture.

##  Appendix C Frequency of identity terms in data

The statistics are based on the training dataset and are provided in Table 4.
It contains three columns: (1) the % of comments labelled as “Hateful” that
contain an identity term, (2) the % of comments labelled as “Not-hateful” that
contain an identity term, and (3) the % of comments Overall (“Hateful” and
“Not-hateful”) that contain an identity term.

Table 4. Frequency of identity terms in hateful and not-hateful comments and overall Term | Hateful % | Not-hateful % | Overall %
---|---|---|---
atheist | 0.0044 | 0.0085 | 0.0058
queer | 0.2220 | 0.0427 | 0.1607
gay | 0.3864 | 0.1282 | 0.2981
transgender | 0.0044 | 0.0085 | 0.0058
lesbian | 0.0266 | 0.0256 | 0.0263
homosexual | 0.0133 | 0.0085 | 0.0117
feminist | 0.0222 | 0.0000 | 0.0146
black | 0.6706 | 0.5042 | 0.6137
white | 1.3367 | 0.5640 | 1.0725
heterosexual | 0.0044 | 0.0000 | 0.0029
islam | 0.0044 | 0.0000 | 0.0029
muslim | 0.0178 | 0.0171 | 0.0175

##  Appendix D Protected attribute word groups

###  D.1. Word groups for three protected attributes (based on (Garg et al.,
2018; Zhao et al., 2018))

Religion:
_Islam:_ ’allah’, ’ramadan’, ’turban’, ’emir’, ’salaam’, ’sunni’, ’koran’,
’imam’, ’sultan’, ’prophet’, ’veil’, ’ayatollah’, ’shiite’, ’mosque’, ’islam’,
’sheik’, ’muslim’, ’muhammad’;
_Christianity:_ ’baptism’, ’messiah’, ’catholicism’, ’resurrection’,
’christianity’, ’salvation’, ’protestant’, ’gospel’, ’trinity’, ’jesus’,
’christ’, ’christian’, ’cross’, ’catholic’, ’church’, ’christians’,
’catholics’

Gender:
_Male:_ ’cowboy’, ’cowboys’, ’cameramen’, ’cameraman’, ’busboy’, ’busboys’,
’bellboy’, ’bellboys’, ’barman’, ’barmen’, ’tailor’, ’tailors’, ’prince’,
’princes’, ’governor’, ’governors’, ’adultor’, ’adultors’, ’god’, ’gods’,
’host’, ’hosts’, ’abbot’, ’abbots’, ’actor’, ’actors’, ’bachelor’,
’bachelors’, ’baron’, ’barons’, ’beau’, ’beaus’, ’bridegroom’, ’bridegrooms’,
’brother’, ’brothers’, ’duke’, ’dukes’, ’emperor’, ’emperors’, ’enchanter’,
’father’, ’fathers’, ’fiance’, ’fiances’, ’priest’, ’priests’, ’gentleman’,
’gentlemen’, ’grandfather’, ’grandfathers’, ’headmaster’, ’headmasters’,
’hero’, ’heros’, ’lad’, ’lads’, ’landlord’, ’landlords’, ’male’, ’males’,
’man’, ’men’, ’manservant’, ’manservants’, ’marquis’, ’masseur’, ’masseurs’,
’master’, ’masters’, ’monk’, ’monks’, ’nephew’, ’nephews’, ’priest’,
’priests’, ’sorcerer’, ’sorcerers’, ’stepfather’, ’stepfathers’, ’stepson’,
’stepsons’, ’steward’, ’stewards’, ’uncle’, ’uncles’, ’waiter’, ’waiters’,
’widower’, ’widowers’, ’wizard’, ’wizards’, ’airman’, ’airmen’, ’boy’, ’boys’,
’groom’, ’grooms’, ’businessman’, ’businessmen’, ’chairman’, ’chairmen’,
’dude’, ’dudes’, ’dad’, ’dads’, ’daddy’, ’daddies’, ’son’, ’sons’, ’guy’,
’guys’, ’grandson’, ’grandsons’, ’guy’, ’guys’, ’he’, ’himself’, ’him’, ’his’,
’husband’, ’husbands’, ’king’, ’kings’, ’lord’, ’lords’, ’sir’, ’sir’, ’mr.’,
’mr.’, ’policeman’, ’spokesman’, ’spokesmen’;
_Female:_ ’cowgirl’, ’cowgirls’, ’camerawomen’, ’camerawoman’, ’busgirl’,
’busgirls’, ’bellgirl’, ’bellgirls’, ’barwoman’, ’barwomen’, ’seamstress’,
’seamstress’, ’princess’, ’princesses’, ’governess’, ’governesses’,
’adultress’, ’adultresses’, ’godess’, ’godesses’, ’hostess’, ’hostesses’,
’abbess’, ’abbesses’, ’actress’, ’actresses’, ’spinster’, ’spinsters’,
’baroness’, ’barnoesses’, ’belle’, ’belles’, ’bride’, ’brides’, ’sister’,
’sisters’, ’duchess’, ’duchesses’, ’empress’, ’empresses’, ’enchantress’,
’mother’, ’mothers’, ’fiancee’, ’fiancees’, ’nun’, ’nuns’, ’lady’, ’ladies’,
’grandmother’, ’grandmothers’, ’headmistress’, ’headmistresses’, ’heroine’,
’heroines’, ’lass’, ’lasses’, ’landlady’, ’landladies’, ’female’, ’females’,
’woman’, ’women’, ’maidservant’, ’maidservants’, ’marchioness’, ’masseuse’,
’masseuses’, ’mistress’, ’mistresses’, ’nun’, ’nuns’, ’niece’, ’nieces’,
’priestess’, ’priestesses’, ’sorceress’, ’sorceresses’, ’stepmother’,
’stepmothers’, ’stepdaughter’, ’stepdaughters’, ’stewardess’, ’stewardesses’,
’aunt’, ’aunts’, ’waitress’, ’waitresses’, ’widow’, ’widows’, ’witch’,
’witches’, ’airwoman’, ’airwomen’, ’girl’, ’girls’, ’bride’, ’brides’,
’businesswoman’, ’businesswomen’, ’chairwoman’, ’chairwomen’, ’chick’,
’chicks’, ’mom’, ’moms’, ’mommy’, ’mommies’, ’daughter’, ’daughters’, ’gal’,
’gals’, ’granddaughter’, ’granddaughters’, ’girl’, ’girls’, ’she’, ’herself’,
’her’, ’her’, ’wife’, ’wives’, ’queen’, ’queens’, ’lady’, ’ladies’, ”ma’am”,
’miss’, ’mrs.’, ’ms.’, ’policewoman’, ’spokeswoman’, ’spokeswomen’

Ethnicity:
_Chinese:_ ’chung’, ’liu’, ’wong’, ’huang’, ’ng’, ’hu’, ’chu’, ’chen’, ’lin’,
’liang’, ’wang’, ’wu’, ’yang’, ’tang’, ’chang’, ’hong’, ’li’;
_Hispanic:_ ’ruiz’, ’alvarez’, ’vargas’, ’castillo’, ’gomez’, ’soto’,
’gonzalez’, ’sanchez’, ’rivera’, ’mendoza’, ’martinez’, ’torres’, ’rodriguez’,
’perez’, ’lopez’, ’medina’, ’diaz’, ’garcia’, ’castro’, ’cruz’;
_White:_ ’harris’, ’nelson’, ’robinson’, ’thompson’, ’moore’, ’wright’,
’anderson’, ’clark’, ’jackson’, ’taylor’, ’scott’, ’davis’, ’allen’, ’adams’,
’lewis’, ’williams’, ’jones’, ’wilson’, ’martin’, ’johnson’

##  Appendix E Embedding bias metrics and values

Let $\mathbf{M}\in\mathbb{R}^{m\times d}$ be a matrix representing $d$
dimensional word embeddings for the $m$ neutral terms (e.g. crazy, nice,
etc.). Let $\mathbf{T}_{i}\in\mathbb{R}^{k_{i}\times d}$ be a matrix that
corresponds to the $d$ dimensional word embeddings for the $k_{i}$ terms that
represent subgroup $i$ (e.g. $k_{1}$ = 2, where 1 represents the male
subgroup, which has terms {he, boy}). Let $\mathbf{x}_{i}\in\mathbb{R}^{m}$ be
the vector representing the averaged cosine similarity scores between the $m$
neutral terms and the $k_{i}$ subgroup terms for subgroup $i$. We compute the
$j^{th}$ entry of $\mathbf{x}_{i}$ as follows.

(2)
$x_{ij}=\frac{1}{k_{i}}\sum^{k_{i}}_{\ell=1}COSINESIM(m_{j},\mathbf{T}_{i\ell})$

We measure deviations between subgroups (e.g. male vs female) as proxies for
word embedding bias. These measures include mean absolute error (MAE), root
mean squared error (RMSE), and several variations. When there are two
subgroups being compared we compute

(3) $MAE=\frac{1}{m}\sum^{m}_{j=1}|x_{1j}-x_{2j}|$

In the event that there are more than two subgroups being compared, we compute
the average MAE (AMAE) score across all pairwise comparisons between
subgroups.

(4) $AMAE=\frac{1}{{s\choose 2}}\sum^{s}_{k=1}\sum^{s}_{n=k+1}MAE$

where $s$ is the number of subgroups. Similarly, we compute the RMSE and
averaged RMSE (ARMSE) as follows.

(5) $RMSE=\sqrt{\frac{\sum^{m}_{j=1}(x_{1j}-x_{2j})^{2}}{m}}$

(6) $ARMSE=\frac{1}{{s\choose 2}}\sum^{s}_{k=1}\sum^{s}_{n=k+1}RMSE$

Figure 4. AMAE magnitude across different protected attributes and word
embedding models Figure 5. ARMSE magnitude across different protected
attributes and word embedding models

##  Appendix F Classification bias metric definitions

_Equal Opportunity:_ It measures the difference between true positive rates
(proportion of samples correctly classified into positive class) for reference
and protected groups.

_Gini Equality:_ It measures the difference between the Gini Inequality of
benefits, which are defined as: Benefits = Prediction - Target + 1

_Normalized Treatment Equality:_ It measures the difference between False
Negative - False Positive ratio in reference and protected groups.

_Overall Accuracy Equality:_ It measures the difference between accuracy in
reference and protected groups.

_Positive Predictive Value:_ It measures the difference between positive
predictive values (proportion of positive samples among all samples classified
as positive) for reference and protected groups.

_Positive Class Balance:_ It measures the difference between average predicted
probability, given predicted class is positive, for reference and protected
groups.

_Statistical Parity:_ It measures the difference between positive rates
(proportion of samples classified into the positive class) for reference and
protected groups.

The choice of metrics depends on which type of error the user assigns more
penalty to, and therefore wants to equalize across different groups. For
example: Equal Opportunity means equalizing the True Positive Rate (TPR)
across the different groups. If TPR is a criterion the user really cares
about, then Equal Opportunity is a suitable fairness metric to consider. If
the user cares more about the probability of getting the favourable result,
the user should consider the Statistical Parity. Therefore, the selection of
the metric depends on the user’s needs and preferences. Automation of this
process remains a challenge (Wachter et al., 2021).

##  Appendix G Classification bias supplementary data and results

Tables 5-9 provide supplementary results for classification bias assessment.

Table 5. Examples of the structured dataset entries generated through protected attribute mining. Gender and religion are extracted using the look-up and NER based methods, respectively Comment |  Gender Extracted |  Religion Extracted
---|---|---
A visit to the DC Holocaust Museum revealed Hitler won by 43% of the popular vote and 32% of the seats. He also used the Schutzstaffel (SS) to intimidate his opponents, reminding one of the Antifa thugs we have today. Hitler also got the Muslims on his side. Uncanny. |  Male |  Islam
”Seton Catholic where their own students talk trash about how low of a division there football team is in.” |  - |  Christianity
How to develop their competitive skills and social skills in Home Schooling…as my child too complain abt bullying and unjust behaviour of teachers since she is super active… |  Female |  -

.

Table 6. Swapped comments examples Version |  Comment | Gender | Label | Prediction
---|---|---|---|---
Original |  This Jerry Lewis ripoff needs to just go away already. The guy is so good at acting like a fool because he is a fool. | Male | Hateful | 0.9811
Swapped |  This Jerry Lewis ripoff needs to just go away already. The gal is so good at acting like a fool because she is a fool. | Female | Hateful | 0.9891
Table 7. Synthetic counterfactual dataset examples for religion Templates |  Subgroup: Christianity |  Subgroup: Islam |  Label
---|---|---|---
I am a [Identity] person, ask me anything |  I am a Christian person, ask me anything |  I am a Muslim person, ask me anything |  Not-hateful
[Identity] people are gross and universally terrible |  Christian people are gross and universally terrible |  Muslim people are gross and universally terrible |  Hateful
Table 8. Prediction probability statistics for counterfactual comments with religion references Actual Comment Type |  Subgroup |  Avg. Predicted Probability
---|---|---
Not-hateful |  Islam |  15.6% hateful
Not-hateful |  Christianity |  11.2% hateful
Hateful |  Islam |  67.9% hateful
Hateful |  Christianity |  63.5% hateful
Table 9. Prediction probability statistics for counterfactual comments with gender references Actual Comment Type |  Subgroup |  Avg. Predicted Probability
---|---|---
Not-hateful |  Male |  12.9% hateful
Not-hateful |  Female |  13.4% hateful
Hateful |  Male |  57.4% hateful
Hateful |  Female |  57.2% hateful

##  Appendix H Global interpretability results

The global interpretability of predictions using SHAP provides a holistic view
of how a given token impacts the overall working of the model. The results are
illustrated in Figure 6. The various dots in this plot signify the presence of
a token across different sets of comments. The vertical location shows what
token is being depicted, the color shows whether that token’s importance was
high or low for that observation, and the horizontal location shows whether
the effect of that value caused a higher or lower prediction (model’s output
probability). We generated this view for the hate-comments in particular, and
discovered that profane language or cuss words turned out to be significant
and positively impacting the model - thereby driving it to classify these
comments as “hateful”.

Figure 6. Global interpretability of predictions using SHAP

##  Appendix I Other ALTAI Requirements

ALTAI Requirement #6 focuses on societal and environmental well-being and
considers the environmental impact of AI systems. There exist different
methods and mechanisms that can be used to evaluate the associated
environmental impacts. Our process uses the Machine Learning Emissions
Calculator tool (Lacoste et al., 2019) to estimate the amount of carbon
emissions produced by training the classifier under assessment here. This can
be measured in Carbon dioxide equivalent (CO2eq) units, which is “a metric
measure used to compare the emissions from various greenhouse gases on the
basis of their global-warming potential, by converting amounts of other gases
to the equivalent amount of carbon dioxide with the same global warming
potential.” (eurostat Statistics Explained, 2017). The model assessed in this
study has been trained on CPUs locally and its carbon footprint was considered
to be negligible.

There are several additional ethical AI requirements that have not been been
applicable or covered in this paper. The model under assessment here has been
trained offline on reliable datasets. But resilience of the model to cyber-
attacks, such as data poisoning and model evasion, is a critical dimension if
the model is trained online and using comments labels through crowdsourcing
(ALTAI Requirement #2). The model here used public datasets. But in the event
of use of or application to private data, the assessment should also evaluate
privacy dimensions (ALTAI Requirement #3). This also includes data governance
on both a macro and a micro level, with the focus areas of data availability,
sharing, usability, consistency, integrity.

The assessment may also account for the interests of stakeholders and how much
each requirement is important to each. The idea of relevance matrix proposed
by (Brown et al., 2021) can be used to connect them with the models ethical
performance. The assessment output can also inform AI impact assessment (Kazim
et al., 2021), which are usually first party created, and not necessarily
covered in second/third party reviews.