# Inverse Rational Control with Partially Observable Continuous Nonlinear
Dynamics

Saurabh Daptardar
Department of ECE
Rice University
Houston, TX 77005
<EMAIL_ADDRESS>
&Paul Schrater
Department of Computer Science
University of Minnesota
Minnesota, IN 12345
<EMAIL_ADDRESS>
&Xaq Pitkow
Department of Neuroscience
Baylor College of Medicine
Houston, TX 77005
<EMAIL_ADDRESS>

###### Abstract

Continuous control and planning remains a major challenge in robotics and
machine learning. Neuroscience offers the possibility of learning from animal
brains that implement highly successful controllers, but it is unclear how to
relate an animal’s behavior to control principles. Animals may not always act
optimally from the perspective of an external observer, but may still act
rationally: we hypothesize that animals choose actions with highest expected
future subjective value according to their own internal model of the world.
Their actions thus result from solving a different optimal control problem
from those on which they are evaluated in neuroscience experiments. With this
assumption, we propose a novel framework of model-based inverse rational
control that learns the agent’s internal model that best explains their
actions in a task described as a partially observable Markov decision process
(POMDP). In this approach we first learn optimal policies generalized over the
entire model space of dynamics and subjective rewards, using an extended
Kalman filter to represent the belief space, a neural network in the actor-
critic framework to optimize the policy, and a simplified basis for the
parameter space. We then compute the model that maximizes the likelihood of
the experimentally observable data comprising the agent’s sensory observations
and chosen actions. Our proposed method is able to recover the true model of
simulated agents within theoretical error bounds given by limited data. We
illustrate this method by applying it to a complex naturalistic task currently
used in neuroscience experiments. This approach provides a foundation for
interpreting the behavioral and neural dynamics of highly adapted controllers
in animal brains.

##  1 Introduction

Brains evolved to understand, interpret, and act upon the physical world. To
thrive and reproduce in a harsh and dynamic natural environment, animals
therefore evolved flexible, robust controllers. Machine learning and
neuroscience both aim to emulate or understand how these successful
controllers operate.

Traditional imitation learning or inverse control methods extract reusable
policies to help generate autonomous behavior or help predict future actions
from an agent’s past behavior. In contrast, here we do not aim to extract the
policy of a well-trained expert. Instead we want to identify the internal
model and preferences of a real agent that may make interesting mistakes [13].
Unlike the convention in robotics and artificial intelligence, animals are not
optimized for stable, narrowly-defined tasks, but instead survive by
performing well enough in competitive, changing ecological niches. Comparing
actual control policies across conditions, animals, or species may ultimately
guide us to broad control principles that generalize beyond specific tasks.
Meanwhile, our approach estimates latent assumptions and dynamic beliefs for
real biological controllers, thereby providing targets for understanding
neural network representations and implementations of perception and action.

Our approach does not assume that agents perform optimally at a given task.
Rather, we assume that agents are rational — by which we mean that agents act
optimally according to their own internal model of the task, which may differ
from the actual task.

We solve this problem by formulating the agent’s policy as an optimal solution
of a Partially Observable Markov Decision Process (POMDP) [20] that the agent
assumes that it faces. Whereas Reinforcement Learning (RL) tries to find the
optimal policy given the dynamics and the reward function, Inverse
Reinforcement Learning (IRL, [18, 6, 2]) tries to find a reward function which
best accounts for observed actions from an agent. Similarly, inverse optimal
control (IOC) [8, 19] tries to find the assumed model dynamics that explains
observed actions with a given cost function yields the similar optimal policy.
Trying to find both is, in general, an ill-posed problem, but with a
sufficiently constrained model or proper conditioning it can be solved. Others
have solved this problem in the fully observable setting [10, 17].

Some of the present authors recently solved the partially observable case for
a discrete state space with discretized beliefs [22]. However, while that
solution was useful for their particular application, in general its
computational expense grows rapidly with the problem complexity and size, and
that solution assumes that actions are easy to select from the discretized
value function. These issues makes that solution infeasible for continuous
state spaces and continuous controls. In the present work, we provide methods
that address these challenges.

Two major questions are how to construct representations of the agent’s
beliefs, and how to choose a policy based on those beliefs. We solve the first
by assuming that agents know the basic task structure but not the task
parameters, so we can use model-based inference to identify the agent’s
beliefs. We solve the second by using flexible function approximation to
estimate values and policies over the parameter space. We demonstrate the
utility of our approach by applying it to simulated agents in a continuous
control task with latent variables that has been used in neuroscience:
catching fireflies in virtual reality [13].

##  2 Prior work on Inverse Reinforcement Learning

Inverse Reinforcement Learning (IRL) and Inverse Optimal Control (IOC) solve
aspects of the general problem of inferring internal models of an observed
agent. Ng and Russell [15] formulate the IRL problem as a linear programming
problem which takes agents policy as input and learn reward parameters with an
$\ell_{2}$ regularization which makes the agent’s policy optimal and also
maximizes difference between the value at optimal action and value at next
best action for each state (maximizing the margin of optimal policy). Abbeel
and Ng [1] extend the idea to perform maximum margin IRL. Max margin IRL
introduces certain biases to solve the ambiguity of multiple reward functions.
Some methods [23] resolve the ambiguity by adopting a principle of maximum
entropy [11] to obtain a distribution over reward functions without bias. Max
Entropy IRL [23] tries to estimate a distribution over trajectories which has
maximum entropy under the constraints of matching expectations of certain
features. This approach is feasible only for finite discrete states and
actions. and becomes intractable as the length of trajectories increases.

For IOC, the Relative Entropy IRL framework [5] allows for unknown dynamics,
and minimizes the KL divergence between two distributions over trajectories.
The analytical solution needs a transition dynamics function which is
estimated by importance sampling. It is important to note here that they do
not assume the agent behaves optimally in any sense. This method was extended
in Guided Cost Learning by [9] to model a free maximum entropy formulation
where the reward function is represented by a neural network. They use
reinforcement learning to move the baseline policy towards policies with
higher rewards imposing weak optimality.

A more principled way to think about this problem class is to view the state
action trajectories as observations about a reward and dynamics with latent
parameters [22]. [16] define the likelihood of trajectories as a exponential
Boltzmann distribution in state action value function $Q(s,a;r)$. Their
inference method performs a random walk over the reward parameter space using
the posterior, and compute the posterior over the parameters by computing the
optimal action value function $Q^{*}$. This is a nested loop approach that is
feasible only for small spaces, as computing the optimal policy and value
functions in the inner loop is costly. Other notable approaches include
maximum likelihood IRL [21, 2], Path Integral IRL [12], simultaneous
estimation of rewards and fully-observed dynamics [10]. [12] proposes learning
a continuous state, continuous action reward function by sampling locally
around the optimal agent trajectories. These methods are local searches that
do not consider trajectories with unexpected control input.

Across all of these methods, there is not a complete inverse solution that can
learn how an agent models rewards, dynamics, and uncertainty in a partially
observable task with continuous nonlinear dynamics and continuous controls.

##  3 Inverse Rational Control

To define the Inverse Rational Control problem, we first formalize our tasks
as Partially Observable Markov Decision Processes (POMDPs). A POMDP $M$ is a
tuple, $M=(S,A,\Omega,R,T,O,\gamma)$ that includes states $s\in S$, actions
$a\in A$, observations $o\in\Omega$, reward or loss function $R$, as well as
transition probabilities $T(s^{\prime}|s,a)$, observation probabilities
$O(o|s)$, and a temporal discount factor $\gamma$. Since the states $s$ are
only partially observable, the POMDP determines the agent’s time-dependent
‘belief’ about the world, namely a posterior over states given the history of
observations and actions, $b_{t}=p(s_{t}|o_{1:t},a_{1:t})$. An optimal
solution of the POMDP determines a value function $Q(b,a)$ through the Bellman
equation [4], which in turn defines a policy $\pi$ that selects the action
that maximizes the value. We parameterize $R$, $T$ and $O$, and therefore the
value function and policy, by $\theta=(\theta_{r},\theta_{t},\theta_{o})$.

We define the Inverse Rational Control problem as identifying, using only data
measurable from an agent’s behavior, the most probable internal model
parameters $\theta\in\Theta$ for an agent that solves the POMDP described
above. Specifically, we allow measurements of the states and action trajectory
taken by the agent, but we have no access to the beliefs that motivate those
actions. The agent’s sensory observations may be fully observed, fully
unobserved, or partially observed, for instance when we cannot access the
particular observation noise that corrupts the agent’s senses. A solution to
IRC provides both parameters and an estimate of the agent’s beliefs over time.

A core idea in our approach is first to learn policies and value functions
over the parameterized manifold of models, reflecting an optimized ensemble of
agents, rather than a single optimized agent. This then allows us to maximize
the likelihood of the observed data generated by an unknown agent, by finding
which parameters from the ensemble best explain the agent’s actions. Because
our overarching goal is scientific, a major benefit of this approach is to
provide the best interpretable explanation of an agent’s behaviors as rational
for a task as defined by some parameters $\theta$.

###  3.1 General formulation of optimal control ensembles

There are two difficulties in solving the POMDP model ensemble and computing a
likelihood function over trajectories. First, policies and value functions are
complex functions of the model parameters. We address this by using flexible,
trainable networks to learn the value function and policies, as described
below. Second, the belief updating process of the agent involves a difficult
integral and requires correct handling of the agent’s observations.

The agent does not observe the world states $s$, but gets partial observations
$o$ about them that induce a belief $b_{t}=p(s_{t}|o_{1:t},a_{1:t},\theta)$.
When the agent plans into the future, it does not know the future
observations, so it must marginalize over them to predict the consequences of
its actions $a$. If an agent were given the observations, future beliefs would
be known, but when observations are unknown the agent has only a distribution
over beliefs arising from an average transition probability
$\bar{T}_{\theta_{d}}$ between belief states,
$\bar{T}_{\theta_{d}}(b^{\prime}|b,a)=\int
do\,T_{\theta_{t}}(b^{\prime}|b,a,o)O_{\theta_{o}}(o|a,b)$. The parameters of
this average transition probability subsume the parameters from the
transitions and observation functions, $\theta_{d}=(\theta_{t},\theta_{o})$.
Once we have formulated the problem completely in terms of belief states, and
identified a tractable finite representation of them, then we can use many of
the tools developed for fully observed Markov Decision Processes (MDP).

An agent chooses its actions based on these beliefs according to a policy
$\pi$ as $a\sim\pi(a|b)$. An optimal policy is the one that produces the
maximal (temporally discounted) expected future reward, which can be computed
using the Bellman equation,

$\displaystyle Q^{\pi}_{\theta}(b,a)$
$\displaystyle=R_{\theta_{r}}(b,a)+\gamma\iiint
da^{\prime}db^{\prime}do^{\prime}\
T_{\theta_{t}}(b^{\prime}|b,a)O_{\theta_{o}}(o|b,a)\pi(a^{\prime}|b^{\prime})Q^{\pi}_{\theta}(b^{\prime},a^{\prime})$
(1) $\displaystyle=R_{\theta_{r}}(b,a)+\gamma\iint da^{\prime}db^{\prime}\
\bar{T}_{\theta_{d}}(b^{\prime}|b,a)\pi(a^{\prime}|b^{\prime})Q^{\pi}_{\theta}(b^{\prime},a^{\prime})$
(2)

In practice we replace the full posterior $b$ by a parametric form (such as a
multivariate Gaussian) for which the transitions can be computed tractably,
although this may introduce approximation errors.

To solve the inverse problem by inferring an agent’s internal model that
guides their actions, we begin with the simple observation that the optimal
policy $\pi^{*}(a|b;\theta)$ and optimal state-action value function
$Q^{*}(b,a;\theta)$ function change depending on the dynamics and the rewards
of the problem. That is, the optimal policy and state-action value function
should implicitly be functions of the dynamics and reward parameters. We will
make this explicit by assuming that the dynamics and the reward function are
parameterized functions denoted by $\bar{T}_{\theta_{d}}(b^{\prime}|b,a)$ and
$R_{\theta_{r}}(b,a)$. We choose this parametric form wisely to impose and
exploit structure in the task.

Although the parametric form for the dynamics and reward may be simple or
convenient, the resultant dependence of $Q^{\pi}$ on these parameters is
usually more complicated. Therefore we represent $Q^{\pi}$ by a flexible
function class. This class could be a family of end-to-end trainable deep
networks, for example, or a sum of basis functions as used in nonlinear
support vector machines. We may learn the $Q_{\theta}$ using any of a variety
of reinforcement learning methods. In our work we trained a function
approximator using rewards obtained by policies across many parameters
$\theta$ sampled from a uniform prior on parameter space $\Theta$, although a
good prior over the parameters could make learning faster and more robust.

The optimal policy is to pick the best actions according to the updated value
function. The rewards obtained by these actions can then provide additional
information about the value function, so we may repeat the above procedure
iteratively until the value function converges. However, even choosing the
best action from a given $Q^{\pi}$ can be difficult. When there are few
allowed actions, then the policy is easy to implement: one simply exhaustively
evaluates $Q^{\pi}$ for all actions and picks the most valuable. When there is
a continuum of actions, such that this approach is infeasible, then must
choose another method to maximize value. Here we select actions by
approximating the policy using an actor-critic method implemented by a
supplemental network that is trained to optimize the value function by Deep
Deterministic Policy Gradient (DDPG) [14], although we could also use other
versions of Policy Iteration or $Q$-learning.

Our algorithm for Policy Iteration across model space is given below:

Algorithm 1 Learning optimal value functions across parameter space

1:Given a class of POMDP problems over parameter space $\Theta$

2:Initialize a random policy $\pi$

3:repeat

4: Sample multiple $\theta\sim\mathcal{U}(\Theta)$ and store in
$\mathcal{D}_{\theta}$

5: for all $\theta\in\mathcal{D}_{\theta}$ do

6: Generate $(b,a)$ trajectories $\tau$ using policy $\pi_{\theta}$ and store
in $\mathcal{T}$

7: end for

8: Solve equation 2 for $Q^{\pi}$

9: Improve policy $\pi$ using $Q^{\pi}$ by either $\epsilon$-greedy updates,
softmax action selection [20], or policy gradient for continuous actions.

10:until policy $\pi$ converges

11:return $\pi^{*}\leftarrow\pi$

###  3.2 Inverse Rational Control from optimal control ensembles

Now that we have constructed an ensemble of approximately optimal policies
over the task space, we now aim to recover the true parameters of an agent
whose behavior we observe. Specifically, we try to find parameters which
maximize the likelihood of the agent’s trajectories. For a given
$\theta\in\Theta$, we know the transition dynamics and have already computed
the optimal policy.

Given the observable state and action trajectories indexed by $i\in\\{1\ldots
N\\}$, the log likelihood of the model parameters $\theta$ given the
experimental observations of one trajectory $i$ is

$\displaystyle\mathcal{L}_{i}=\log{p(s^{i}_{0:T},a^{i}_{0:T}|o^{i}_{0:T},\theta)}=\log
p(s^{i}_{0})+\sum_{t}\log\pi(a^{i}_{t}|b^{i}_{t}(o^{i}_{0:t}),\theta)+\log
p(s^{i}_{t+1}|a^{i}_{t},s^{i}_{t},\theta)$ (3)

Note that this likelihood is conditioned on observations $o$, which the agent
has but the external observer does not. When we do not have the observations
but only state action sequence and the initial state, ideally we would
marginalize the likelihood in equation 3 over all latent observation
trajectories, like in traditional Expectation Maximization [7]. Here instead
we use a hard E step, taking the maximum a posteriori (MAP) value for the
observations given the observed state, $\hat{o}={\rm
argmax}_{o}O_{\theta_{o}}(o|s)$. To maximize this likelihood, we rely on auto-
differentiation to compute the gradient of the policy. The algorithm for
learning the maximum likelihood estimate of true parameters given a state-
action trajectory is given by Algorithm 2.

Algorithm 2 Maximum Likelihood estimate for agent’s parameters

1:Initialize $\theta$ randomly by sampling from the prior
$\theta\sim\mathcal{P}(\Theta)$

2:repeat

3: Estimate agent’s observations by MAP inference,
$\hat{o}=\underset{o}{\operatorname{argmax}}\;O_{\theta_{o}}(o|s)$.

4: Estimate beliefs $b$ given $\hat{o}$ using
$\hat{b}_{t}=p(s_{t}|\hat{o}_{1:t},a_{1:t})$.

5: Compute total log-likelihood over trajectories
$\mathcal{L}\leftarrow\sum\nolimits_{i}\mathcal{L}_{i}$ where
$\mathcal{L}_{i}$ is computed according to equation 3

6: Update $\theta$ by one gradient ascent step with learning rate $\alpha$
$\theta\leftarrow\theta+\alpha\nabla_{\theta}\mathcal{L}$

7:until $\mathcal{L}$ converges

8:return $\theta$

In Algorithm 1 we estimate optimal policies over the entire parameter space.
When we then estimate the particular $\theta$ that best explains the agent’s
actions, it is by construction consistent with rational behavior: the best fit
policy is optimal given those parameters, even if they do not match the actual
task. Of course, for real data, this model fit does not imply that the agent’s
policy actually falls into our model class. Where needed, the model class
should be expanded to accommodate other potentially false assumptions the
agent may make.

We can summarize the whole Inverse Rational Control framework in Algorithm 3:

Algorithm 3 Parameter space Inverse Rational Control

1:Define a family of POMDPs for the task, parametrized by $\Theta$.

2:Learn the state-action value functions $Q(b,a;\theta)$ over
$\theta\in\Theta$ using Algorithm 1.

3:Estimate agent’s parameters $\hat{\theta}$ from its observable behavior
using Algorithm 2.

##  4 Demonstration task: ‘Catching fireflies’

We demonstrate that our proposed Inverse Rational Control framework works by
recovering the internal model of simulated agents performing two control
tasks. Ultimately we will apply this approach to understand the internal
control models of behaving animals in neuroscience experiments, where we do
not know the ground truth. However, using simulated agents allows us to verify
the method when we know the ground truth.

In this task, an agent must navigate through a virtual world to catch a
flashing target, called the ‘firefly’ (Figure 1A) [13]. When the agent stops
moving, the trial ends, the agent receives a reward if it is sufficiently
close to the firefly position, and a new target appears. Each target is
visible only briefly, and sensory inputs provide partial (noisy) observations
of self motion, so the agent is uncertain about the current position of its
target as well as its current velocity.

Figure 1: Firefly control task [13]. (A) To reach the transiently visible
firefly target, an agent must navigate by noisy optic flow over a dynamic
textured plane. (B) For a 1D version of this task with only three allowed
actions $a$, we derive sensible state-action value functions $Q(b,a;\theta)$,
here showing that it is best to move toward the target and then stop, unless
the target is too far to justify the effort. Our method accurately recovers
the agent’s assumed parameters within limits imposed by the data, both for the
1D task (C) and the 2D task (D), as shown for several example parameters
$\theta$ inferred from different agents. Error bars show 95% confidence
intervals derived from the curvature of the likelihood given limited data. (E)
Overhead view illustrates one agent’s belief dynamics, depicted by posterior
covariance ellipses centered at each believed location (blue), as well as our
method’s estimates of those beliefs (red). (F) The estimated and true belief
dynamics closely match. Three representative components of the belief
representation are shown here: most likely firefly $x$ position, most likely
angular velocity, and the posterior covariance between $x$ and $y$.

The two example problems both use this task structure, either in one or two
dimensions. In the 1D task, the world state and control variables are
therefore location and forward velocity, and we only allow three discrete
control outputs. In the 2D task we add angle and angular velocity to the state
space, allow continuous control outputs, and allow the agent to make noisy
observations of its state. Given the uncertainty in its state estimate, the
agent must track its belief over the states, and plan its optimal trajectory
based on this belief state space and unknown future belief transitions based
on unknown future observations. Given this task setting, the agent maximizes
its reward expected over the distribution implied by the belief. Each episode
or trial is a variable duration $T$ depending on when the agent stops moving.
The goal is to maximize the total rewards over this finite time horizon.

Our first simulation experiment is a simplified version of the task, where the
agent’s position is restricted to one dimension, state dynamics are linear,
and the agent receives no observations. The belief state is Gaussian-
distributed over the agent’s relative location to the target, with scalar mean
$\mu$ and variance $\sigma^{2}$ evolving as

$\displaystyle\mu_{t+1}$ $\displaystyle=\mu_{t}+g_{a}a\hskip
28.45274pt\sigma^{2}_{t+1}=\sigma^{2}_{t}+\sigma^{2}_{0}$ (4)

The model parameters are the action gain $g_{a}$ and the process noise log
standard deviation $\log{\sigma_{0}}$. We have only three discrete actions,
namely: go left, go right, and stop.

To compute the optimal policies we use a $Q$-learning variant of Algorithm 1.
Recall that we can substitute different learning algorithms, but it is crucial
to learn the optimal action value function $Q^{*}$ or policy as a function of
$\theta$. One approach we took for the 1D demonstration task was to construct
$Q$ from a factorized set of basis functions. We experimented with both
manually constructed polynomial and exponential basis functions, and a
collection of random shallow networks as basis functions for both $\phi(b,a)$
and $\psi(\theta)$. We found that the random network basis was more expressive
and performed better for this task. Figure 1B plots the value function
$Q^{*}(s,a,\theta)$ for each action and a fixed $\theta$.

Now that we have computed the optimal policies over a manifold of parameters,
we create an agent with one particular set of parameters $\theta^{*}$. Using
the corresponding optimal policy and belief dynamics, we simulate this agent’s
belief state and action trajectories. Next we use experimentally observable
data (the action sequence) to update the belief trajectory per our model.
Given this data we follow the maximum likelihood estimation as described in
Algorithm 2 using the most probable belief states to estimate the parameters.
In Figure 1C we plot the estimated parameters recovered by our algorithm
against the agent’s true parameters, along with the 95% confidence interval
for parameter $\theta_{i}$ as given by $2(\mathcal{I}^{-1/2})_{ii}$ where
$\mathcal{I}$ is the Fisher information matrix. In almost all cases the true
value is within the resultant error bars, showing that we correctly recover
the true parameters within the precision allowed by the data.

Our second simulation uses the full 2D firefly task. We use two gain
parameters, one on forward velocity input and another on angular velocity
input, and one process noise parameter that affects $x$ and $y$ position
identically, with no angular process noise. We assume the agent maintains a
Gaussian representation for its beliefs over the world state, which it updates
using an extended Kalman filter given the dynamics parameters $\theta_{d}$.

To compute the optimal policies we use Deep Deterministic Policy Gradient
(DDPG) [14] in Algorithm 1. We sample the parameters uniformly over a moderate
range and for each $\theta$, we run the current policy $\pi_{\theta}$ to
collect sequences of belief states, actions, and rewards. Every time point
$(\hat{b}_{t},a_{t},r_{t})$ in these sequences along with the corresponding
$\theta$ was provided as input to two neural networks, each three layers deep,
64 units wide, and using softplus activations, whose output estimated the
state-action value function and the policy. The learning updates for this are
same as the DDPG algorithm.

As before, we create an agent by choosing parameters $\theta^{*}$, and
generate trajectories using corresponding belief dynamics and optimal policy.
We preserve only the experimentally measurable action trajectories, and apply
Algorithm 2 to compute the maximum likelihood estimate and confidence
intervals. Figure 1D again shows that we can recover agents’ true parameters
up to the intrinsic uncertainty. Additionally, we compare our estimates of
their beliefs to their actual beliefs and find excellent agreement (Figure
1E,F).

##  5 Summary

To summarize our proposed inverse reinforcement learning framework, we express
and learn the optimal action value function generalized over the task
parameter space. This can be thought of as learning optimal value function and
policy for all the parameters in the parameter space. For large, complex
tasks, using an informative prior with sampling focused on relevant regions of
parameter space could greatly accelerate the learning and make the
approximation more robust.

Most other related frameworks have a nested inner loop of policy optimization
or refinement to adapt to the optimal policy for the new updated parameter.
Unlike such methods, our method separates these optimizations into two
separate loops, one for learning optimal policy over parameter space, and
second for maximum likelihood estimation of the true parameters given the
optimal policy computed in the previous loop.

###  5.1 Limitations

Firstly, our assumption parametrized dynamics and reward function induces
obvious model bias. Of course we can reduce this bias by making the parameter
space richer and more flexible by increasing the number and variety of
parameters. However, this would come with a price in computability and
interpretability.

In case of POMDPs, where we need to work with posterior distribution over the
states or the belief, deriving analytical parametric belief update equations
can be a difficult inference problem in general. For special cases like finite
discrete states, the posterior updates can be succinctly represented matrix
products, parameterized by transition matrices. For continuous state and
Gaussian noises, we can use Kalman filter updates which are parameterized by
the gain and covariance matrices, and for small nonlinearity this can be
generalized using extended Kalman filters. But for general cases with general
posterior distributions, a belief update might be a difficult inference
problem. Though this is not necessarily a limitation of our approach but a
challenge for inference problems more generally, still it may be difficult to
apply our method in such cases.

Similarly, as stated several times above, the learning in Algorithm 1 can be
replaced with any suitable reinforcement learning algorithm which guarantees
convergence to an optimal policy, as long as the policy or the $Q$ function is
explicitly a function of model parameters. Our framework theoretically does
not limit the number of parameters as long as there is some algorithm which
can explore and learn optimal policy over that large space.

###  5.2 Outlook

One interesting novelty of our framework is to make optimal action value
$Q^{*}$ and optimal policy $\pi^{*}$ explicit functions of the parameters
$\theta$. We can use this representation to extend the control task to
hierarchical models: Instead of thinking of the parameters as fixed in the
environment, we could consider them to be slowly changing latent variables
describing task demands in a dynamic world. The value function over this
larger space would then provide a means of adaptive control, where the agent
could adjust its policy based on its current belief about $\theta$. To use IRC
for such an agent, we would then need to introduce higher-level parameters
that describe the full dynamics.Now, this looks like any other optimal control
problem, and when we solve for optimal control problem for $\tilde{M}$ as a
function of augmented state $\tilde{s}$, we are learning the optimal control
for $M_{\theta}$ generalized over $\theta$.

We have implemented Inverse Rational Control for neuroscience applications,
but the core principles have value in other fields as well. We can view IRC as
a form of Theory of Mind, whereby one agent (a neuroscientist) creates a model
of another agent’s mind (for a behaving animal). Theory of Mind is a prominent
component of social interactions, and imputing rational motivations to actions
provides a useful description of how people think [3]. Designing useful
artificial agents to interact with others in online environments would also
benefit from being able to attribute rational strategies to others. One
important example is self-driving cars, which currently struggle to handle the
perceived unpredictability of humans. While humans do indeed behave
unpredictably, some of this may stem from ignorance of the rational
computation that drives actions. Inverse Rational Control provides a framework
for better interpretation of other agents, and serves as a valuable tool for
greater understanding of unifying principles of control.

#### Acknowledgments

The authors thank Kaushik Lakshminarasimhan, Dora Angelaki, Greg DeAngelis,
James Bridgewater and Minhae Kwon for useful discussions. SD and XP were
supported in part by the Simons Collaboration on the Global Brain award
324143. PS and XP were supported in part by BRAIN Initiative grant NIH
5U01NS094368. XP was supported in part by an award from the McNair Foundation,
NSF CAREER Award IOS-1552868, and NSF 1450923 BRAIN 43092-N1.

## References

  * [1] Pieter Abbeel and Andrew Y Ng.  Apprenticeship learning via inverse reinforcement learning.  In Proceedings of the twenty-first international conference on Machine learning, page 1. ACM, 2004.
  * [2] Monica Babes, Vukosi Marivate, Kaushik Subramanian, and Michael L Littman.  Apprenticeship learning about multiple intentions.  In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pages 897–904, 2011.
  * [3] Chris Baker, Rebecca Saxe, and Joshua Tenenbaum.  Bayesian theory of mind: Modeling joint belief-desire attribution.  In Proceedings of the annual meeting of the cognitive science society, volume 33, 2011.
  * [4] R Bellman.  Dynamic programming.  Princeton University Press, 1957.
  * [5] Abdeslam Boularias, Jens Kober, and Jan Peters.  Relative entropy inverse reinforcement learning.  In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 182–189, 2011.
  * [6] Jaedeug Choi and Kee-Eung Kim.  Inverse reinforcement learning in partially observable environments.  Journal of Machine Learning Research, 12(Mar):691–730, 2011.
  * [7] Arthur P Dempster, Nan M Laird, and Donald B Rubin.  Maximum likelihood from incomplete data via the em algorithm.  Journal of the Royal Statistical Society: Series B (Methodological), 39(1):1–22, 1977.
  * [8] Krishnamurthy Dvijotham and Emanuel Todorov.  Inverse optimal control with linearly-solvable mdps.  In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pages 335–342, 2010.
  * [9] Chelsea Finn, Sergey Levine, and Pieter Abbeel.  Guided cost learning: Deep inverse optimal control via policy optimization.  In International Conference on Machine Learning, pages 49–58, 2016\.
  * [10] Michael Herman, Tobias Gindele, Jörg Wagner, Felix Schmitt, and Wolfram Burgard.  Inverse reinforcement learning with simultaneous estimation of rewards and dynamics.  In Artificial Intelligence and Statistics, pages 102–110, 2016\.
  * [11] Edwin T Jaynes.  Information theory and statistical mechanics.  Physical review, 106(4):620, 1957.
  * [12] Mrinal Kalakrishnan, Peter Pastor, Ludovic Righetti, and Stefan Schaal.  Learning objective functions for manipulation.  In Robotics and Automation (ICRA), 2013 IEEE International Conference on, pages 1331–1336. IEEE, 2013.
  * [13] Kaushik J Lakshminarasimhan, Marina Petsalis, Hyeshin Park, Gregory C DeAngelis, Xaq Pitkow, and Dora E Angelaki.  A dynamic bayesian observer model reveals origins of bias in visual path integration.  Neuron, 2018.
  * [14] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  arXiv preprint arXiv:1509.02971, 2015.
  * [15] Andrew Y Ng, Stuart J Russell, et al.  Algorithms for inverse reinforcement learning.  In Icml, pages 663–670, 2000.
  * [16] Deepak Ramachandran and Eyal Amir.  Bayesian inverse reinforcement learning.  Urbana, 51(61801):1–4, 2007.
  * [17] Siddharth Reddy, Anca D. Dragan, and Sergey Levine.  Where do you think you’re going?: Inferring beliefs about dynamics from behavior.  In Arxiv 1805.08010, 2018.
  * [18] Stuart Russell.  Learning agents for uncertain environments.  In Proceedings of the eleventh annual conference on Computational learning theory, pages 101–103. ACM, 1998.
  * [19] Felix Schmitt, Hans-Joachim Bieg, Michael Herman, and Constantin A Rothkopf.  I see what you see: Inferring sensor and policy models of human real-world motor behavior.  In AAAI, pages 3797–3803, 2017.
  * [20] Richard S Sutton, Andrew G Barto, et al.  Reinforcement learning: An introduction.  MIT press, 1998.
  * [21] Monica C Vroman.  Maximum likelihood inverse reinforcement learning.  PhD thesis, Rutgers University-Graduate School-New Brunswick, 2014.
  * [22] Zhengwei Wu, Paul Schrater, and Xaq Pitkow.  Inverse POMDP: Inferring what you think from what you do.  arXiv preprint arXiv:1805.09864, 2018.
  * [23] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey.  Maximum entropy inverse reinforcement learning.  In AAAI, volume 8, pages 1433–1438. Chicago, IL, USA, 2008.


# cantnlp@LT-EDI-2024: Automatic Detection of Anti-LGBTQ+ Hate Speech in
Under-resourced Languages

Sidney G.-J. Wong  University of Canterbury, New Zealand  Geospatial Research
Institute, New Zealand  Matthew Durward  University of Canterbury, New Zealand

###### Abstract

This paper describes our homophobia/transphobia in social media comments
detection system developed as part of the shared task at LT-EDI-2024. We took
a transformer-based approach to develop our multiclass classification model
for ten language conditions (English, Spanish, Gujarati, Hindi, Kannada,
Malayalam, Marathi, Tamil, Tulu, and Telugu). We introduced synthetic and
organic instances of script-switched language data during domain adaptation to
mirror the linguistic realities of social media language as seen in the
labelled training data. Our system ranked second for Gujarati and Telugu with
varying levels of performance for other language conditions. The results
suggest incorporating elements of paralinguistic behaviour such as script-
switching may improve the performance of language detection systems especially
in the cases of under-resourced languages conditions.

additions

cantnlp@LT-EDI-2024: Automatic Detection of Anti-LGBTQ+ Hate Speech in Under-
resourced Languages

##  1 Introduction

The purpose of this shared task was to develop a multiclass classification
system to predict instances of homophobia/transphobia in social media comments
across different language conditions Kumaresan et al. (2024). The ten language
conditions were: English (eng), Spanish (esp), Gujarati (guj), Hindi (hin),
Kannada (kan), Malayalam (mal), Marathi (mar), Tamil (tam), Tulu (tcy), and
Telugu (tel).

The main contribution of this paper is that we extend on the work using
spatio-temporally retrained transformer-based language models in Wong et al.
(2023). We have expanded on the synthetic script-switching approach by
incorporating real-world (or organic) samples of script-switching during
domain adaptation in the development of our multiclass classification model
using pretrained language models.

kantelgujmalmarengtamhinesptcy$0$$2$$4$$6$$8$$10$$12$Language
ConditionObservations (000s) Figure 1:  Barplot of labelled training data. The
combined total number of observations (in thousands) by language condition
ordered from the most (kan) to the least (tcy) number of observations.

###  1.1 Problem Description

The organisers of the shared task provided labelled training data for each of
the ten language conditions. Five of the language conditions belong to the
Indo-European language family (eng, esp, guj, hin, and mar) and the remaining
five language conditions belong to the Dravidian language family (kan, mal,
tam, tcy, and tel).

The labelled training data comes from different sources (eng and tam in
Chakravarthi et al., 2021; hin and mal in Kumaresan et al., 2023; and esp in
García-Díaz et al., 2020). The training data is made up of comments from users
reacting to LGBTQ+ related content on YouTube. The labelled training data for
guj, kan, mar, tcy, and tel were introduced for the current shared task.

The total number of social media comments for each language condition
(combining the train and development sets) are shown in Figure 1. kan has the
most observations, followed by tel and guj. tcy has the least number of
observations. The remaining language conditions each have between 1,000 to
5,000 observations.

The social media comments were manually annotated and broadly labelled using
on a three-class classification system Chakravarthi et al. (2021). There were
only two classes for tcy which we have labelled none and homo for consistency
with other language conditions. The classes are:

  * •

Homophobic Content (homo): any comments which were deemed gender-based and
involved pejorative or defamatory language directed towards non-heterosexual
people.

  * •

Transphobic Content (trans): any derogatory or offensive language directed
towards transgender and gender diverse people.

  * •

Non-anti-LGBTQ+ Content (none): counter speech or hope speech as well as
comments which does not contain any homophobic or transphobic content.

| none | homo | trans
---|---|---|---
eng | 0.94 | 0.06 | 0.00
esp | 0.57 | 0.22 | 0.22
guj | 0.47 | 0.28 | 0.25
hin | 0.95 | 0.02 | 0.04
kan | 0.44 | 0.27 | 0.28
mal | 0.79 | 0.16 | 0.06
mar | 0.73 | 0.16 | 0.11
tam | 0.77 | 0.17 | 0.06
tcy | 0.74 | 0.26 | -
tel | 0.39 | 0.32 | 0.30
Table 1:  Class distribution by language condition. Note that tcy has a binary
class distribution.

The class distribution for each language condition is shown in Table 1. We
observe significant class imbalance between language conditions especially in
eng and hin where the homo and trans classes make up less than a tenth of the
labelled training data. Of the 3,726 observations in the eng language
condition, there are only 221 tokens of homo and nine tokens of trans.

Outside the labelled training data and published material, the organisers did
not provide additional corpus or demographic information of the labelled
training data as part of the shared task. Therefore, the classification system
needs to account for the differences in data availability as well as class
imbalance for each language condition.

###  1.2 Related Work

The current shared task is the third shared task on homophobia and transphobia
detection in social media comments. The first shared task involved only three
language conditions: tam, eng, and a separate tam-eng code-mixed condition
Chakravarthi et al. (2022).

The classification system with the best performance for eng had a weighted
Macro $F_{1}$ score of 0.92 was developed by team ablimet Maimaitituoheti et
al. (2022) and for tam was 0.94 developed by team arguably. The best
performing classification system for the tam-eng code-mixed condition was also
developed by team arguably with a weighted Macro $F_{1}$ score of 0.89. The
code-mixed condition had the lowest performance across the three conditions.

Participants took different approaches involving statistical language models
and machine learning. The best performing system used xlm-roberta pretrained
language models Conneau et al. (2020). This BERT-based transformer language
approach structures the relationship between words with language embeddings
Devlin et al. (2019). These language embeddings account for structures across
multilingual conditions.

The second shared task expanded to five language conditions (eng, esp, hin,
mal, and tam) which was broken down by a three-class classification system
similar to the current shared task Chakravarthi et al. (2023). Three of the
language conditions (eng, mal, and tam) were further classified into a seven-
class classification system.

The weighted Macro $F_{1}$ score for the best performing three-class
classification systems was 0.97 for eng and 0.98 for hin developed by
teamplusone using BERT-based transformer models. A weight-space ensembling
technique presented itself as the best solution for esp, mal, and tam language
conditions Ninalga (2023).

The best performing systems for the seven-class classification condition were
all developed using transformer language models. The weighted Macro $F_{1}$
score eng was 0.82 developed by team teamplusone, for mal was 0.88 developed
by team cantnlp Wong et al. (2023), and for tam was 0.87 developed by team
deepblueai.

This suggests BERT-based models, such as xlm-roberta for zero-shot learning,
are particularly effective in carrying out multiclass classification tasks
outlined in the current shared task. More importantly, these systems are
simple to implement and allow for domain adaptation Liu et al. (2019)

Wong et al. (2023) introduced synthetically script-switched instances of
social media data during domain adaptation to account for the high frequency
of script-switching in the labelled data for hin, mal, and tam. The
introduction of script-switched language data improved the performance of the
homophobia/transphobia detection model in hin, but not mal or tam.

The results from Wong et al. (2023) suggest that there is potential for
incorporating paralinguistic behaviour such as script-switching in the
development of multiclass detection language systems. Therefore, this paper
explores this further by incorporating different forms of script-switching.

$0$$0.2$$0.4$$0.6$$0.8$$1$gujkantamtelmarmalengesphintcyProportion of words in
Latin Script(%)Language Condition Figure 2:  Boxplot of labelled training
data. Language condition by the proportion of observations with at least one
word written in Latin script ordered from the lowest (tcy) to the highest
(guj) proportion of observations.

##  2 Methodology

In this section, we provide an overview of our system development methodology.
We took a transformer-based language model approach to develop our system. We
used xlm-roberta as the base plm for our system Conneau et al. (2020). The
embeddings in xlm-roberta were trained on two terabytes of web-crawled data
for over 100 language including nine of the ten language conditions of
interest (with the exclusion of tcy).

A significant advantage of transformer-based plms is the ability for domain
adaptation as discussed in Section 2.1. This means we can retrain the default
language embedding models with additional language data without the need to
train resource-intensive plms from scratch.

We tested different forms of script-switching in order to understand the
impacts of script-switching on our classification system. We then used the
plms developed in Section 2.1 to fine-tune our multiclass classification model
as discussed in Section 2.2. Based on the weighted Macro $F_{1}$ for each
language condition, we submitted the results from the best performing
multiclass classification system to the organisers.

###  2.1 Domain Adaptation

| baseline | synthetic | organic
---|---|---|---
mono | multi | mono | multi | mono | multi
eng | 0.32 | 0.35 | 0.32 | 0.32 | 0.32 | 0.32
esp | 0.80 | 0.82 | 0.87 | 0.84 | 0.76 | 0.82
guj | 0.94 | 0.94 | 0.95 | 0.95 | 0.95 | 0.95
hin | 0.32 | 0.32 | 0.32 | 0.32 | 0.32 | 0.32
kan | 0.92 | 0.93 | 0.94 | 0.94 | 0.94 | 0.94
mal | 0.51 | 0.53 | 0.73 | 0.58 | 0.78 | 0.61
mar | 0.44 | 0.45 | 0.44 | 0.41 | 0.42 | 0.46
tam | 0.48 | 0.42 | 0.54 | 0.49 | 0.48 | 0.56
tcy | 0.72 | 0.43 | 0.43 | 0.43 | 0.43 | 0.43
tel | 0.98 | 0.97 | 0.98 | 0.97 | 0.97 | 0.98
Table 2:  Model performance of candidate classification models by Macro
$F_{1}$ using our test set split from combining the train and validation sets
provided to us by the organisers for each language condition. The three
candidate languages models are: baseline, synthetic, and organic. We also
compared the performance of language-specific (mono) and multilingual (multi)
multiclass classification models. The best performing system is highlighted in
bold.

The first stage in developing our system involved domain adaptation (also
known as retraining). Liu et al. (2019) noted that domain adaptation can
improve the performance of transformer-based language models in downstream
tasks. We can do this by introducing domain (or register) specific text
samples to produce customised retrained PLMs (or retrained language models).
This means we can introduce language data from under-resourced languages such
as tcy as well as additional linguistic information such as script-switching -
a common phenomenon in social media language.

As noted in Wong et al. (2023), we observed varying levels of script-switching
in the labelled training data. Therefore, we first needed to identify the
level of script-switching between language conditions. For each observation,
we calculated the proportion of words written in Latin script using the
alphabet-detector111https://pypi.org/project/alphabet-detector/ Python
package.

The proportion of script-switching between language conditions is shown in
Figure 2 where 0 suggests low usage of Latin-based characters (in the case of
guj, kan, tam, tel) while 1 suggests high usage as expected for eng and esp.
Figure 2 confirms that there is sufficient need to account for the varying-
degrees of script-switching between language conditions.

We retrained xlm-roberta with two forms of script-switching: synthetic and
organic script-switching. These are our candidate models. We describe how we
produced the language data for domain adaptation in Section 2.1.1 and Section
2.1.2. We produced the candidate language models by retraining the language
embeddings using the simpletransformers222https://simpletransformers.ai/
Python library. We did this over four iterations and we evaluated the training
for every 500 steps with AdamW optimisation Loshchilov and Hutter (2019).
Model performance was based on evaluation loss.

####  2.1.1 Synthetic Script-Switching

We took a similar approach as Wong et al. (2023) to produce synthetic samples
of script-switched language data for domain adaptation. We define synthetic as
machine-generated texts. Due to the limited availability of observations for
some language conditions, our main source of human-generated texts come from
the Leipzig Corpus Collection Goldhahn et al. (2012). Each corpus contained
10,000 Wikipedia abstracts produced in 2016 with the exception of tcy which
was produced in 2018.

We then randomly sampled half of the abstracts from each language condition
(excluding the Latin-based eng and esp) and used the
ai4bharat333https://pypi.org/project/ai4bharat-transliteration/ Python library
to transliterate the relevant Brahmic orthographies into Latin script. Once we
produced a subset of synthetically script-switched Wikipedia abstracts, we
combined the original abstracts with the synthetically script-switched
abstracts. Finally, we combined the labelled training data to create train and
evaluation sets. The inclusion of the labelled training data is to ensure
register-specific domain adaptation.

####  2.1.2 Organic Script-Switching

The second form of script-switched language data for domain adaptation involve
organic samples of script-switched language data. We define organic as human-
generated texts. This proved to be a challenge as we were unable to identify
sources of script-switched social media language data for some of the under-
resourced language conditions.

We used the pre-existing labelled training data to produce language profiles
to develop a language identification model with the
langdetect444https://pypi.org/project/langdetect/ These language profiles were
used to detect organic instances of script-switched social media data from the
Global Corpus of Language Use (cglu; Dunn, 2020). This produced a train set
with 230,000 observations and an evaluation set with 12,000 observations which
we could use for domain adaptation.

###  2.2 Classification Model

As discussed in Section 2.1, we developed our multiclass classification models
using the candidate language models during the domain adaptation phase. The
three candidate languages models are: the baseline xlm-roberta language model
(baseline), xlm-roberta retrained with synthetic samples of script-switched
language data (synthetic), and xlm-roberta retrained with organic samples of
script-switched language data (organic).

We resampled the available data to create our own train (80%), validation
(10%), and test (10%) sets to avoid over-fitting on the validation set during
model evaluation. We trained language specific classification models (mono)
and an ensemble multilingual classification model (multi) by combining the
labelled training data. We trained the multiclass classification model using
the simpletransformers Python package for four iterations and we evaluated the
training for every 500 steps with AdamW optimisation Loshchilov and Hutter
(2019). The model performance was based on evaluation loss.

The model performance for each of the candidate models are shown in Table 2.
We have indicated the best performing model based on average Macro $F_{1}$
score (highlighted in bold). In some language conditions, there were multiple
best performing models. Not included in Table 2 are the combined average Macro
$F_{1}$ score for the multilingual models: the average Macro $F_{1}$ for the
baseline model was 0.89; both synthetic and organic models had an average
Macro $F_{1}$ of 0.90.

| cantnlp | Best Performance
---|---|---
eng | 0.323 | 0.496
esp | 0.496 | 0.582
guj | 0.962 | 0.968
hin | 0.326 | 0.458
kan | 0.943 | 0.948
mal | 0.775 | 0.942
mar | 0.433 | 0.626
tam | 0.555 | 0.880
tcy | 0.452 | 0.707
tel | 0.965 | 0.971
Table 3:  The average Macro $F_{1}$ score of our classification system, and
the average Macro $F_{1}$ score of the overall best performing classification
system.

##  3 Results

Based on the average Macro $F_{1}$ score of the candidate models as shown in
Table 2, we nominated the language-specific synthetic classification model as
the best performing classification system. We applied this classification
system and submitted the results to the organisers. The results of our
submitted homophobia/transphobia detection system are shown in Table 3. The
best performing language condition was tel with an average Macro $F_{1}$ of
0.97 and the worst performing language condition was eng with an average Macro
$F_{1}$ of 0.32.

Our final rank for each language conditions are as follows: for eng we came
tenth equal out of ten teams; for esp we came fourth out of five teams; for
guj we came second out of six teams; for hin we came fourth equal out of seven
teams; for kan we came fourth out of eight teams; for mal we came seventh out
of nine teams; for mar we came fifth out of six teams; for tam we came fifth
out of eight teams; for tcy we came third equal out of four teams; and finally
for tel we came second out of nine teams.555Note the final rankings differ
from the published results for esp, tam, and tel as they were not included in
the final rank list due to human error from the organising committee during
submission.

##  4 Discussion

The use of synthetic and organic script-switched language data during domain
adaptation increased the performance for all language conditions from the
baseline model with the exception of eng, hin, and tcy. We expected the eng
and esp language conditions to perform poorly with our proposed methodology as
there were very few instances of script-switching, but the poor performance of
tcy was unexpected.

We hypothesise the poor performance in tcy was due to the limited number of
observation (as shown in Figure 1) the higher than expected usage of Latin-
based script for tcy in the labelled training data (as shown in Figure 2).
This will require robust statistical analysis beyond the scope of the current
paper.

We also posit the poor performance of eng and hin was a result of the class
imbalance between instances of homophobic, transphobic and the non-anti-LGBTQ+
content as demonstrated in Table 1. The performance of our eng and hin
language-specific detection models are in line with other participating teams.

In contrast to the method proposed in Wong et al. (2023), we did not include
any methods to counter the class imbalance in the training data nor did we
include random noise injection to expand the minority classes. It was shown
that random over sampling of minority classes did not significantly improve
the performance of the detection models.

##  5 Conclusion

The main contribution of the current paper is the proposal to use synthetic
and organic script-switching examples of during domain adaptation to improve
the down-stream performance for under-resourced languages. We demonstrated
that our methodology improved the model performance for guj, kan, mal, mar,
and tam even though the improvement was only marginal. Even though our
homophobia/transphobia detection system did not rank first for any of the ten
language conditions, we were pleased with the performance of our detection
system which supports the inclusion of paralinguistic information.

## Ethics Statement

The purpose of the current shared task is to develop a homophobic/transphobic
language detection system in social media texts particularly for under-
resourced Indo-Aryan and Dravidian languages within the fields of
computational linguistics and natural language processing.

We recognise the importance of community-lead research in particular by
members of under-represented and minoritised communities. The lead author
acknowledges his positionality as a member of the LGBTQ+ community. The lead
author is familiar with anti-LGBTQ+ discourse both in online and offline
spaces and the harmful effects of hate speech and offensive language on
members of the LGBTQ+ communities Wong (2023b).

In terms of the authors’ linguistic membership, the authors share proficiency
in eng and esp; however, the authors acknowledge their limited experience with
guj, kan, mar, tam, tcy, and tel with some exposure to hin and mal. We
acknowledge the limitations of our analysis in language conditions where we
have limited proficiency and we will follow the guidance and expertise of
members from the relevant language communities.

We want to thank the organisers of the shared task and the workshop on
Language Technology for Equality, Diversity, and Inclusion. We also want to
thank the contributors of the training data and those who were involved in the
labelling process across the different language conditions.

## Limitations

Under the purview of developing a homophobic/transphobic language detection
system in social media texts, we want to highlight the limitations of our
proposed system and methodology.

Firstly, we acknowledge there are differences in data quality and veracity
between the different language conditions. This is based on the differences in
the corpus size between the different language conditions (as shown in Figure
1) and the distribution of homophobic and transphobic content.

In light of these data quality issues, we have not accounted for these
differences between language conditions. This means we do not entirely
understand the downstream impacts on model performance - although it is clear
that there is a possible relationship between larger and more balanced
language conditions (tel) performing better than smaller and more imbalanced
language conditions (tcy). It is possible these differences could exacerbate
biases already observed in transformer-based language models Bhardwaj et al.
(2021).

Beyond the upstream and downstream impacts of bias in transformer-based
language models, we also recognised that incorporating external data sets from
lcc Goldhahn et al. (2012) and the cglu Dunn (2020) introduces additional
biases not properly addressed in this paper such as geographic bias in social
media language data Wong et al. (2022).

Secondly, there is a need to conduct this form of research under a
sociolinguistic or linguistic anthropological framework. There is a risk that
training data detecting homophobia, transphobia, hate speech, or offensive may
not necessarily reflect the social, political, or linguistic realities of
different populations. This is because some of the features extracted from the
labelled training data may not reflect real-world knowledge.

These differences are particularly evident when we apply these detection
systems across dialect contexts Wong (2023a). For this reason, we propose that
future work in this area should also consider how these systems perform in
real-world context beyond the evaluation of labelled training data. We should
work alongside members of LGBTQ+ communities from culturally and
linguistically diverse backgrounds to understand the effectiveness and
generalisability of our homophobic/transphobic detection systems.

## References

  * Bhardwaj et al. (2021) Rishabh Bhardwaj, Navonil Majumder, and Soujanya Poria. 2021.  Investigating Gender Bias in BERT.  _Cognitive Computation_ , 13(4):1008–1018.
  * Chakravarthi et al. (2023) Bharathi Raja Chakravarthi, Rahul Ponnusamy, Malliga S, Paul Buitelaar, Miguel Ángel García-Cumbreras, Salud María Jiménez-Zafra, Jose Antonio García-Díaz, Rafael Valencia-García, and Nitesh Jindal. 2023.  Overview of Second Shared Task on Homophobia and Transphobia Detection in Social Media Comments.  In _Proceedings of the Third Workshop on Language Technology for Equality, Diversity and Inclusion_ , pages 38–46, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria.
  * Chakravarthi et al. (2022) Bharathi Raja Chakravarthi, Ruba Priyadharshini, Thenmozhi Durairaj, John McCrae, Paul Buitelaar, Prasanna Kumaresan, and Rahul Ponnusamy. 2022.  Overview of The Shared Task on Homophobia and Transphobia Detection in Social Media Comments.  In _Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion_ , pages 369–377, Dublin, Ireland. Association for Computational Linguistics.
  * Chakravarthi et al. (2021) Bharathi Raja Chakravarthi, Ruba Priyadharshini, Rahul Ponnusamy, Prasanna Kumar Kumaresan, Kayalvizhi Sampath, Durairaj Thenmozhi, Sathiyaraj Thangasamy, Rajendran Nallathambi, and John Phillip McCrae. 2021.  Dataset for Identification of Homophobia and Transophobia in Multilingual YouTube Comments.  ArXiv:2109.00227 [cs].
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised Cross-lingual Representation Learning at Scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  ArXiv:1810.04805 [cs].
  * Dunn (2020) Jonathan Dunn. 2020.  Mapping languages: the Corpus of Global Language Use.  _Language Resources and Evaluation_ , 54(4):999–1018.
  * García-Díaz et al. (2020) José Antonio García-Díaz, Ángela Almela, Gema Alcaraz-Mármol, and Rafael Valencia-García. 2020.  UMUCorpusClassifier: Compilation and evaluation of linguistic corpus for Natural Language Processing tasks.  _Procesamiento del Lenguaje Natural_ , 65(0):139–142.
  * Goldhahn et al. (2012) Dirk Goldhahn, Thomas Eckart, and Uwe Quasthoff. 2012.  Building Large Monolingual Dictionaries at the Leipzig Corpora Collection: From 100 to 200 Languages.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_ , pages 759–765, Istanbul, Turkey. European Language Resources Association (ELRA).
  * Kumaresan et al. (2023) Prasanna Kumar Kumaresan, Rahul Ponnusamy, Ruba Priyadharshini, Paul Buitelaar, and Bharathi Raja Chakravarthi. 2023.  Homophobia and transphobia detection for low-resourced languages in social media comments.  _Natural Language Processing Journal_ , 5:100041.
  * Kumaresan et al. (2024) Prasanna Kumar Kumaresan, Ruba Priyadharshini, Bharathi Raja Chakravarthi, Paul Buitelaar, Asha Hegde, Hosahalli Lakshmaiah Shashirekha, Saranya Rajiakodi, Miguel Ángel García-Cumbreras, Salud María Jiménez-Zafra, José Antonio García-Díaz, Rafael Valencia-García, Kishore Kumar Ponnusamy, Poorvi Shetty, and Daniel García-Baena. 2024.  Overview of Third Shared Task on Homophobia and Transphobia Detection in Social Media Comments.  In _Proceedings of the Fourth Workshop on Language Technology for Equality, Diversity and Inclusion_ , Malta. European Chapter of the Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  ArXiv:1907.11692 [cs].
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled Weight Decay Regularization.  ArXiv:1711.05101 [cs, math].
  * Maimaitituoheti et al. (2022) Abulimiti Maimaitituoheti, Yong Yang, and Xiaochao Fan. 2022.  ABLIMET @LT-EDI-ACL2022: A Roberta based Approach for Homophobia/Transphobia Detection in Social Media.  In _Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion_ , pages 155–160, Dublin, Ireland. Association for Computational Linguistics.
  * Ninalga (2023) Dean Ninalga. 2023.  Cordyceps@LT-EDI: Patching Language-Specific Homophobia/Transphobia Classifiers with a Multilingual Understanding.  In _Proceedings of the Third Workshop on Language Technology for Equality, Diversity and Inclusion_ , pages 185–191, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria.
  * Wong et al. (2022) Sidney Wong, Jonathan Dunn, and Benjamin Adams. 2022.  Comparing Measures of Linguistic Diversity Across Social Media Language Data and Census Data at Subnational Geographic Areas.  _Proceedings in New Zealand Geospatial Research Conference_.
  * Wong et al. (2023) Sidney Wong, Matthew Durward, Benjamin Adams, and Jonathan Dunn. 2023.  cantnlp@LT-EDI-2023: Homophobia/Transphobia Detection in Social Media Comments using Spatio-Temporally Retrained Language Models.  In _Proceedings of the Third Workshop on Language Technology for Equality, Diversity and Inclusion_ , pages 103–108, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria.
  * Wong (2023a) Sidney Gig-Jan Wong. 2023a.  Monitoring Hate Speech and Offensive Language on Social Media.  In _Fourth Spatial Data Science Symposium_ , University of Canterbury.
  * Wong (2023b) Sidney Gig-Jan Wong. 2023b.  _Queer Asian Identities in Contemporary Aotearoa New Zealand: One Foot Out of the Closet_.  Lived Places Publishing.