# They were robbed! Scoring by the middlemost
to attenuate biased judging in boxing

Stuart Baumann  Carl Singleton<EMAIL_ADDRESS>Corresponding author:
<EMAIL_ADDRESS>Economics Division, Stirling Management School,
University of Stirling, Stirling, FK9 4LA, Scotland, UK.
We are grateful for comments and advice from Anwesha Mukherjee.
Declarations of interest: none

###### Abstract

Boxing has a long-standing problem with biased judging, impacting both
professional and Olympic bouts. “Robberies”, where boxers are widely seen as
being denied rightful victories, threaten to drive fans and athletes away from
the sport. To tackle this problem, we propose a minimalist adjustment in how
boxing is scored: the winner would be decided by the majority of round-by-
round victories according to the judges, rather than relying on the judges’
overall bout scores. This approach, rooted in social choice theory and
utilising majority rule and middlemost aggregation functions, creates a
coordination problem for partisan judges and attenuates their influence. Our
model analysis and simulations demonstrate the potential to significantly
decrease the likelihood of a partisan judge swaying the result of a bout.

JEL Codes: D91, L83, Z20, Z28
Keywords: Scoring Rules, Judgement Bias, Contests, Pugilism, Combat Sports

## 1\. Introduction

Boxing has a reputation for partisan and corrupt judging. At the amateur
level, some decisions in Olympic gold medal bouts have attracted criticism and
ridicule, becoming boxing folklore, such as Roy Jones Jr.’s defeat in the 1988
(Seoul) light heavyweight final to a South Korean fighter (Ashdown, 2012), and
Joe Joyce’s defeat in the 2016 (Rio de Janeiro) super heavyweight final
(Ingle, 2021; Rumsby, 2021). In professional boxing, there is longstanding
suspicion about the integrity of judges (e.g., US Senate, 2001). Recent
perceived “robberies” include Haney Vs. Lomachenko (Wainwright, 2023) and the
first two editions of Alvarez Vs. Golovkin (Reid, 2023).

This short paper models the decisions of boxing judges and proposes an
alternative scoring method that has the potential to significantly attenuate
judge bias. Currently, scoring at the elite level is on a per-judge basis,
with three judges usually employed for elite professional bouts and five at
the Olympic amateur level. Judges score each round individually and then award
their entire “vote” to the boxer who wins a majority of rounds.111This is a
slight simplification as judges can award additional points for a given round
based on knockdowns, fouls, or particularly dominant performances by one
fighter. The bout is then awarded to the boxer receiving votes from a majority
of judges. If neither boxer receives a majority, due to at least one tied
scorecard among the judges, then the bout is a draw. In this system,
“aggregation over rounds and then judges”, or “majority judges rule”, it is
relatively straightforward for a judge to ensure their vote goes to their
favoured boxer. They just need to award them half the rounds (i.e., 7 of 12
for a world championship level men’s professional bout). They can do this
while minimising backlash, by choosing the best rounds for their favoured
boxer.

The change we propose, “aggregation over judges and then over rounds”, or
“majority rounds rule”, is for each round to be awarded based on the aggregate
scores over all judges. Whoever wins the majority of rounds wins the bout,
rather than whoever wins on a majority of the judges’ scorecards. This
represents a minimalist change to the scoring system in the sport, so that the
aggregation of judges’ scores is first between them within rounds, and then
over rounds, rather than vice versa. The minor nature of this change is
sufficient to introduce a significant coordination problem for a partisan
judge, and may be acceptable among fans.222Fans tend to scrutinise, oppose and
criticise even quite small changes to the rules of their beloved sports. A
notable example from cricket is the LBW rule, which has been continually
‘improved’ over the last century, often under opposition and criticism (Kumar,
2022).

We focus on modelling the simplest practical case, with three judges, one of
whom is biased in favour of one boxer. Under majority judges rule, a partisan
judge can substantially increase the probability of a boxer winning despite
being outnumbered by unbiased judges. Under majority rounds rule, even if the
partisan judge awards a majority of rounds to a favoured boxer, then this will
have no impact on the final outcome unless those rounds align with the
decisions of the other judges. This coordination problem implies that, to
achieve a high probability of victory for their favoured boxer, the partisan
judge would have to award more rounds to their favoured boxer than in the
current system. This exposes them to scrutiny and potential backlash, as
boxing pundits and fans will often criticise poorly awarded rounds on judges’
scorecards.333Criticism of judges in social and sports media is generally
fiercest after bouts where a robbery is perceived to have happened. It often
focuses on specific rounds where a judge’s decision appears to be particularly
poor. While authorities seldom intervene to order a rematch, judges may be
stripped of their status and not employed again (e.g., Slavin, 2017). Our
analysis and simulations of the model demonstrate that the scoring rule change
could be highly effective in diminishing the incentives for biased judging in
boxing and its influence.

Our proposed scoring rule is an application of majority rule and the
middlemost aggregation function from social choice theory, which minimise the
effective manipulability of outcomes by graders (e.g., Arrow, 1963; Balinski
and Laraki, 2007; Young, 1974b, a). This principle is already applied somewhat
to the scoring in boxing, since the decision of the middlemost judge now
determines the bout result. We suggest awarding bouts based on the aggregated
middlemost round-by-round votes instead.

The prevalence of judging bias in combat sports has been documented in a
growing literature of empirical academic papers (e.g., Lee, Cork and
Algranati, 2002; Holmes, McHale and Zychaluk, 2024). This behaviour was also
described vividly in the recent judge-led independent investigation McLaren
(2022) report, which examined unethical conduct in Olympic boxing after being
commissioned by the Association Internationale de Boxe Amateur (AIBA). While
the report did propose improved appointment processes and training of judges,
it did not explore how to make the incentives inherent in the judging process
more resilient to biases and corruption.

Our goal of improving the incentives of judges in boxing is closely related to
the focus of Frederiksen and Machol (1988), who analysed the judging in sports
like figure skating and dance, where judges need to decide between multiple
competitors, a setting where Arrow’s (1963) theorem implies that all possible
ways to combine judge preferences have some undesirable characteristics.
Frederiksen and Machol proposed a new method for aggregating judge scores for
such situations that attenuates some of these issues. Their context though
faced the problem of the Arrow Impossibility Theorem (social choice paradox),
given there were more than two alternative outcomes in the contest. That
theorem does not apply here for a boxing bout since it consists of just two
competitors, only one winner, and potentially biased judges.

In general, this paper contributes to the vast operational research literature
that either post hoc analyses changes to scoring rules and laws in sports or
proposes new changes (for recent surveys see Wright, 2014; Kendall and Lenten,
2017). Our work falls into the latter type of study, particularly where
minimalist changes have been proposed that could still in theory substantially
improve the fairness of sports outcomes. For instance, in the world’s most
popular sport, association football, recent contributions have used
simulations to explore whether incentives and outcomes could be altered
significantly under different tie-breaking rules in round-robin tournaments
(Csató, 2023; Csató, Molontay and Pintér, 2024), whether dynamic sequences in
penalty shootouts could be fairer (Csató and Petróczy, 2022), and whether the
allocation system for the additional slots of the expanded FIFA World Cup
could be improved according to the stated goals of the organisers (Krumer and
Moreno-Ternero, 2023). Finally this paper builds on a growing sports economics
and management literature studying various incentive issues in boxing and
other combat sports (Akin, Issabayev and Rizvanoghlu, 2023; Amegashie and
Kutsoati, 2005; Butler et al., 2023; Butler, 2023; Duggan and Levitt, 2002;
Dietl, Lang and Werner, 2010; Tenorio, 2000). However, to the best of our
knowledge, the incentives of boxing judges have not yet been studied, given
the scoring rules they face, despite a well-developed literature on the
influences and implications of biased decision making by the referees and
judges in other sports (e.g., Dohmen and Sauermann, 2016; Bryson et al., 2021;
Reade, Schreyer and Singleton, 2022), including other combat contests
(Brunello and Yamamura, 2023)).

The remainder of our short paper proceeds as follows. In Section 2, we setup a
styled model of potentially biased judging in a boxing contest. Section 3
describes our analysis and discussion of the model. The detailed proofs of the
main propositions regarding the scoring rules are presented in the Online
Appendix, as are variations on the main results from simulating the model.

## 2\. The Model

Consider a contest between two boxers of equal ability, in the Blue and Red
corners. We assume each sequential round $t\in\\{1,2,\dots,N\\}$ of the
contest has a true result, $\tau_{t}\in\\{B,R\\}$, which is a binomial random
variable with equal probability.

Each judge, $j\in\\{1,2,3\\}$, gets an i.i.d. signal, $x_{t,j}\in\\{B,R\\}$,
about the result of a round. With probability $\alpha\in(0,\frac{1}{2})$ this
signal is the incorrect result, $x_{t,j}\neq\tau_{t}$, while with probability
$1-\alpha$ it is correct, $x_{t,j}\equiv\tau_{t}$.

Judges have a utility of:

$\displaystyle U$ $\displaystyle=S\mathbf{1}_{\text{Blue
wins}}+G\mathbf{1}_{\text{Red wins}}-L\ ,$ $\displaystyle L$
$\displaystyle=\frac{\sum_{t=1}^{N}\mathbf{1}_{s_{t,j}\not=\tau_{t}}}{N}\ ,$
(1)

where $S\geq 0$ and $G\geq 0$ represents the a judge’s value from Blue or Red
winning respectively. $L$ refers to a backlash cost.

We consider the case of two fair judges have $S=G=0$. As these judge’s utility
does not depend on the bout’s winner their optimal behaviour is to minimise
backlash by awarding fairly, defined by choosing a round score of
$s_{t,j}=B\iff x_{t,j}\equiv B$.444We analyse the case where all three judges
are fair in Online Appendix B. The third judge is partisan in favour of Blue
and so has $S>0$ and $G=0$.

Under majority judges rule, the middlemost judge scorecard determines the
bout. Under majority rounds rule, the middlemost judge determines each round’s
winner, and then the middlemost round determines the bout. Judges award rounds
separately and simultaneously.

## 3\. Analysis, Results, and Discussion

The partisan judge ($j=1$) can minimise backlash by awarding rounds
fairly.555If $S\equiv 0$, then this judge’s actions will be congruent to the
fair judges. Under majority judges rule, they can maximally increase the
chance of Blue winning the bout, while minimising backlash, by awarding
$s_{t,1}=B$ in more than $\frac{N}{2}$ rounds. Under majority rounds rule,
their problem is more complex; a judge could award more than $\frac{N}{2}$
rounds to a boxer who then does not win them because the other judges
disagreed.

If $S$ is low, however, then the expected backlash can be sufficient for the
partisan judge to award rounds fairly. We can characterise the critical
$\hat{S}$ where the partisan judge is indifferent between awarding fairly or
gifting an additional round to Blue. We find that this critical value is
higher under majority rounds than majority judges rule, indicating that the
former is more resilient to judge bias.

######  Proposition 1.

For three-round bouts, in which Red won a majority of rounds according to the
true realisations, $\boldsymbol{\tau}=[\tau_{1},\tau_{2},\tau_{3}]$, the
critical $\hat{S}$ is higher under majority rounds than majority judges rule,
$\forall\alpha\in(0,\frac{1}{2})$.

###### Sketch of Proof:.

We can calculate the probability of each fair judge awarding a round for Blue,
denoted by $q$, conditional on the signal seen by the partisan judge:

$\displaystyle q|(x_{t,1}\equiv B)$ $\displaystyle=(1-\alpha)^{2}+\alpha^{2}\
,$ $\displaystyle q|(x_{t,1}\equiv R)$ $\displaystyle=2\alpha(1-\alpha)\ .$
(2)

Under majority rounds rule, the number of fair judges awarding for Blue in a
particular round can be represented as drawing from a binomial distribution
with probabilities as in Equation 2. The survival function of this binomial,
in conjunction with the decision of the partisan judge, is sufficient to infer
the probability of Blue winning the round. From the probabilities for each
round, we can derive the optimal number of rounds for the partisan judge to
award for Blue. Under majority judges rule, we can infer the probability of
another scorecard being in favour of Blue by combining the probabilities in
Equation 2 across rounds. We can use these probabilities to evaluate whether
the partisan judge should award additional rounds such that Blue wins on their
card. Then we can derive, for each scoring rule, expressions for the critical
$\hat{S}$ below which the partisan judge will award rounds fairly (see Online
Appendix A). We find that there is a higher $\hat{S}$ under majority rounds
rule, giving us the proposition. ∎

Proposition 1 establishes that the majority rounds rule is more robust to
partisan judging than the majority judges rule for three-round bouts. We
numerically solve the model to establish the robustness of this result in
longer bouts.666This is done along the lines discussed in the sketch proof for
Proposition 1. The code for calculating partisan judge best responses and bout
simulations are included in the online supplementary material. We use a
benchmark parametrisation of $\alpha=0.1$, $S=0.5$, three judges (one of whom
is partisan), and $N=12$ rounds.777Importantly, Online Appendix B shows that
when all three judges are fair, the majority rounds rule is still more
accurate than the majority judges rule in generating a deserving winner of the
bout. Intuitively this occurs as as there are more combinations of rounds that
could be flipped to change the result. Consider a three-round bout with three
fair judges and a $\mathbf{\tau}$ realisation of $\mathbf{[B,B,R]}$. Consider
that two mistakes happened in judging the bout (in that for two round-judges
the $x_{t,j}$ realisation differs from $\tau$). There are six possible pairs
of $x_{t,j}$ values that can be flipped to change the result of the bout with
the majority rounds rule. For one of the rounds where blue won, we need to
flip two $x_{t,j}$ values and there are six combinations that achieve this.
But there are twelve possible pairs of $x_{t,j}$ values that can be flipped to
change the result of the bout with the majority judges rule. We need to flip
two of the $x_{t,j}$ values awarded to blue on two different scorecards and
there are twelve pairs of values that achieve this.

To demonstrate a partisan judge’s decision making, Figure 1 shows the
probability of Blue winning the bout, given they truly won 6 rounds, for each
number of rounds the partisan judge awards them. Under majority judges rule,
there is a sharp increase in the probability of Blue winning if the partisan
judge awards them more than 6 rounds. If Blue truly deserved to win 4 or 5
rounds, then, to award Blue the win, the partisan judge only needs to risk the
backlash associated with giving them 3 or 2 more rounds on their scorecard. In
contrast, Figure 1 shows that under majority rounds rule, a judge cannot
secure a sharp increase in the probability of Blue winning by giving them a
small number of extra rounds; more rounds only gradually increase Blue’s
chances.

FIGURE 1: Simulated Probability of Blue winning, when both boxers truly won 6
of the 12 rounds, and 1 of the 3 judges favours Blue (in the absence of noise)

Figure 2 shows the impact of these differing incentives for the partisan
judge, from running a series of simulations and counting the proportion of
times each boxer wins under the two scoring systems, conditional on the true
number of rounds won by Blue. When deciding the contest by majority judges,
there is a high probability of erroneous results when Blue truly won only 4-6
rounds. When Blue truly wins the most rounds, the partisan judge unduly helps
to lock in a deserved victory, so there is not a large difference in the
number of incorrectly awarded bouts.

Finally, in the majority rounds case, it can be noted from Figure 2 that the
probability of the blue boxer winning always increases when the biased judge
awards them more rounds. This is in contrast to the majority judges case, when
a judge ceases to impact the result at the point at which they award a
majority of their card to a boxer. For instance, consider a bout where the
fair judge sees 10 rounds with $x_{t,1}\equiv\textit{B}$ and only two rounds
are seen to be won by red. In this case, the biased judge may award additional
rounds to blue to lock in a blue victory, while they would have no such
incentive under majority judges rule.

FIGURE 2: Probability of a “correct” result depending on the number of rounds
truly won by Blue and how judges’ scores are aggregated

This effect, however, does not tend to lead to a greater probability of an
erroneous result under the majority rounds rule. The main reason for this is
that the effect occurs in a context where blue has likely won a large majority
of rounds and is likely to win the bout. The more important case is when a
bout is more even and there is a sharp increase, under the majority judges
rule, in the winning probability at the 7 round level in Figure 2.

This point can be seen in Figure 3, which shows the probability of each
possible outcome on the y-axis and the number of rounds Blue truly won
(excluding noise) on the x-axis. Under majority judges rule (top panel), in
evenly matched bouts, where the true result is a draw, Blue wins 46.9% and Red
wins 11.1%. When evenly matched bouts are awarded under majority rounds rule
(bottom panel), Blue wins 24.3% and Red wins 11.7%.

Figure 3 also shows the frequencies where one boxer wins despite the other
deserving outright victory, e.g., the blue area to the left of the vertical
black line. Under majority judges rule, it is more likely for an erroneous
victory to be in favour of Blue than Red; in this parametrisation, a robbery
in favour of Blue is 12.4 times more likely than a robbery in favour of Red.
Under majority rounds rule, the likelihood of a robbery is still in Blue’s
favour, by 3.6 partisan, because there is still some incentive for the
partisan judge to favour Blue. But this scoring system can substantially
attenuate Blue’s advantage from the presence of a partisan judge. There are
also fewer robberies in absolute terms.

FIGURE 3: Probability of each outcome depending on the number of rounds truly
won by Blue and how judges’ scores are aggregated

For robustness, the Online Appendices demonstrate extensions and checks on our
analysis. Appendix C considers simulations with alternative parametrisations
of the benchmark model, and, in Appendices D-F we repeat the analysis for
setups consistent with women’s professional, men’s Olympic, and women’s
Olympic boxing, respectively (i.e., different numbers of rounds and judges).
The results of all these extensions support our key findings: deciding bouts
by majority rounds, compared with by majority judges, makes it less likely
that a partisan judge sways the outcome of a bout.

## References

  * (1)
  * Akin, Issabayev and Rizvanoghlu (2023) Akin, Zafer, Murat Issabayev, and Islam Rizvanoghlu. 2023\. “Incentives and Strategic Behavior of Professional Boxers.” Journal of Sports Economics, 24(1): 28–49.
  * Amegashie and Kutsoati (2005) Amegashie, J. Atsu, and Edward Kutsoati. 2005\. “Rematches in Boxing and Other Sporting Events.” Journal of Sports Economics, 6(4): 401–411.
  * Arrow (1963) Arrow, Kenneth Joseph. 1963\. Social Choice and Individual Values. Yale University Press.
  * Ashdown (2012) Ashdown, John. 2012\. “50 stunning Olympic moments No14: Roy Jones Jr cheated out of gold.” The Guardian. https://bit.ly/3SQ5KhL.
  * Balinski and Laraki (2007) Balinski, Michel, and Rida Laraki. 2007\. “A theory of measuring, electing, and ranking.” Proceedings of the National Academy of Sciences, 104(21): 8720–8725.
  * Brunello and Yamamura (2023) Brunello, Giorgio, and Eiji Yamamura. 2023\. “Desperately Seeking a Japanese Yokozuna.” Institute of Labor Economics (IZA) IZA Discussion Papers 16536.
  * Bryson et al. (2021) Bryson, Alex, Peter Dolton, J. James Reade, Dominik Schreyer, and Carl Singleton. 2021\. “Causal effects of an absent crowd on performances and refereeing decisions during Covid-19.” Economics Letters, 198(C).
  * Butler et al. (2023) Butler, David, Robert Butler, Joel Maxcy, and Simon Woodworth. 2023\. “Outcome Uncertainty and Viewer Demand for Basic Cable Boxing.” Journal of Sports Economics.
  * Butler (2023) Butler, Robert. 2023\. “An Introduction to the James Quirk Special Issue and the Economics of Combat Sport.” Journal of Sports Economics.
  * Csató (2023) Csató, László. 2023\. “How to avoid uncompetitive games? The importance of tie-breaking rules.” European Journal of Operational Research, 307(3): 1260–1269.
  * Csató and Petróczy (2022) Csató, László, and Dóra Gréta Petróczy. 2022\. “Fairness in penalty shootouts: Is it worth using dynamic sequences?” Journal of Sports Sciences, 40(12): 1392–1398. PMID: 35675384.
  * Csató, Molontay and Pintér (2024) Csató, László, Roland Molontay, and József Pintér. 2024\. “Tournament schedules and incentives in a double round-robin tournament with four teams.” International Transactions in Operational Research, 31(3): 1486–1514.
  * Dietl, Lang and Werner (2010) Dietl, Helmut M., Markus Lang, and Stephan Werner. 2010\. “Corruption in Professional Sumo: An Update on the Study of Duggan and Levitt.” Journal of Sports Economics, 11(4): 383–396.
  * Dohmen and Sauermann (2016) Dohmen, Thomas, and Jan Sauermann. 2016\. “Referee Bias.” Journal of Economic Surveys, 30(4): 679–695.
  * Duggan and Levitt (2002) Duggan, Mark, and Steven D. Levitt. 2002\. “Winning Isn’t Everything: Corruption in Sumo Wrestling.” American Economic Review, 92(5): 1594–1605.
  * Frederiksen and Machol (1988) Frederiksen, Jesper, and Robert Machol. 1988\. “Reduction of paradoxes in subjectively judged competitions.” European Journal of Operational Research. https://www.sciencedirect.com/science/article/abs/pii/037722178890375X.
  * Holmes, McHale and Zychaluk (2024) Holmes, Benjamin, Ian McHale, and Kamila Zychaluk. 2024\. “Detecting individual preferences and erroneous verdicts in mixed martial arts judging using Bayesian hierarchical models.” European Journal of Operational Research).
  * Ingle (2021) Ingle, Sean. 2021\. “Judges ’used signals’ to fix Olympic boxing bouts, McLaren report finds.” The Guardian. https://bit.ly/3sGXk1M.
  * Kendall and Lenten (2017) Kendall, Graham, and Liam J.A. Lenten. 2017\. “When sports rules go awry.” European Journal of Operational Research, 257(2): 377–394.
  * Krumer and Moreno-Ternero (2023) Krumer, Alex, and Juan D. Moreno-Ternero. 2023\. “The Allocation of Additional Slots for the FIFA World Cup.” Journal of Sports Economics, 24(7): 831–850.
  * Kumar (2022) Kumar, Manish. 2022\. “Explained: LBW rules and the controversial umpire’s call in DRS.” The Times of India. https://bit.ly/47F6ef4.
  * Lee, Cork and Algranati (2002) Lee, Herbert, Daniel Cork, and David Algranati. 2002\. “Did Lennox Lewis beat Evander Holyfield?: methods for analysing small sample interrater agreement problems.” Journal of the Royal Statistical Society. Series D (The Statistician).
  * McLaren (2022) McLaren, Richard. 2022\. “Independent investigation of the AIBA.” https://bit.ly/3sHR5e1.
  * Reade, Schreyer and Singleton (2022) Reade, J. James, Dominik Schreyer, and Carl Singleton. 2022\. “Eliminating supportive crowds reduces referee bias.” Economic Inquiry, 60(3): 1416–1436.
  * Reid (2023) Reid, Alex. 2023\. “Boxing’s biggest robberies.” talkSPORT. https://bit.ly/47BO6T8.
  * Rumsby (2021) Rumsby, Ben. 2021\. “Joe Joyce demands Rio 2016 gold medal from IOC after boxing corruption report.” The Telegraph. https://bit.ly/46ndxqw.
  * Slavin (2017) Slavin, Harry. 2017\. “Gennady Golovkin and Canelo Alvarez judge Adalaide Byrd disciplined for lopsided scorecard as she is stood down from major title fights.” Mail Online. https://bit.ly/46nC8vh.
  * Tenorio (2000) Tenorio, Rafael. 2000\. “The Economics of Professional Boxing Contracts.” Journal of Sports Economics, 1(4): 363–384.
  * US Senate (2001) US Senate. 2001\. “Committee on commerce, science and transportation - A review of the professional boxing industry - is further reform needed? Senate Hearing 107-1090.”
  * Wainwright (2023) Wainwright, Anson. 2023\. “Fair or foul? Experts weigh in on Devin Haney-Vasiliy Lomachenko result.” https://bit.ly/49IQUzO.
  * Wright (2014) Wright, Mike. 2014\. “OR analysis of sporting rules – A survey.” European Journal of Operational Research, 232(1): 1–8.
  * Young (1974a) Young, H. 1974a. “A Note on Preference Aggregation.” Econometrica, 42(6): 1129–1131.
  * Young (1974b) Young, H. 1974b. “An axiomatization of Borda’s rule.” Journal of Economic Theory, 9(1): 43–52.

They were robbed! Scoring by the middlemost
to attenuate biased judging in boxing

Online Appendix

##  Appendix A Full proof of Proposition 1

To simplify, we assume there are three rounds. Consequently, there are four
information sets that the partisan judge can receive before they choose how
many rounds to award to Blue. These are { BBB, BBR, BRR, RRR }, which are the
signals of each round of the bout after sorting round results.222Note, in both
scoring systems, no distinctions are made as to when in the bout a particular
round result occurred. Therefore, a bout with true result BRB is the same as a
bout with true result BBR, and we reorder round results to simplify the
analysis.

We start by calculating the probability of a fair judge awarding a round for
the Blue conditional on the partisan judge observing a result of B (we call
this conditional probability $p|B$) and in the complementary case ($p|R$).

$\displaystyle p|B$ $\displaystyle=1-(2\alpha-2\alpha^{2})\ ,$ $\displaystyle
p|R$ $\displaystyle=2\alpha-2\alpha^{2}$ (A.1)

To simplify, we use the notation $\nabla=2\alpha-2\alpha^{2}$. We use this to
calculate the probability of another judge’s scorecard being in favour of Blue
for each information set that the partisan judge observes. We use $C$ to
denote this scorecard probability :

$\displaystyle C|BBB$ $\displaystyle=(1-\nabla)^{2}(2\nabla+1)$ $\displaystyle
C|RRR$ $\displaystyle=\nabla^{2}+2(1-\nabla)\nabla^{2}$ (A.2) $\displaystyle
C|BBR$ $\displaystyle=(1-\nabla)^{3}+(1-\nabla)\nabla^{2}+(1-\nabla)\nabla$
$\displaystyle C|BRR$
$\displaystyle=\nabla^{3}+(1-\nabla)^{2}\nabla+(1-\nabla)\nabla$ (A.3)

### A.1 Majority Judges Rule

The utilities available at each information set (where the first column shows
the information set for each row) and action (the second to fifth columns) are
shown in Table A1.

Award BBB Award BBR Award BRR Award RRR BBB $S(1-(1-C|BBB)^{2})-\alpha$
$S(1-(1-C|BBB)^{2})-\frac{2\alpha+(1-\alpha)}{3}$
$S(C|BBB)^{2}-\frac{\alpha+2(1-\alpha)}{3}$ $S(C|BBB)^{2}-(1-\alpha)$ BBR
$S(1-(1-C|BBR)^{2})-\frac{2\alpha+(1-\alpha)}{3}$ $S(1-(1-C|BBR)^{2})-\alpha$
$S(C|BBR)^{2}-\frac{2\alpha+(1-\alpha)}{3}$
$S(C|BBR)^{2}-\frac{\alpha+2(1-\alpha)}{3}$ BRR
$S(1-(1-C|BRR)^{2})-\frac{\alpha+2(1-\alpha)}{3}$
$S(1-(1-C|BRR)^{2})-\frac{2\alpha+(1-\alpha)}{3}$ $S(C|BRR)^{2}-\alpha$
$S(C|BRR)^{2}-\frac{2\alpha+(1-\alpha)}{3}$ RRR
$S(1-(1-C|RRR)^{2})-(1-\alpha)$
$S(1-(1-C|RRR)^{2})-\frac{\alpha+2(1-\alpha)}{3}$
$S(C|RRR)^{2}-\frac{2\alpha+(1-\alpha)}{3}$ $S(C|RRR)^{2}-\alpha$

Table A1: Utilities for each action and information set in the basic model

The best responses to seeing BBB and BBR are to award BBB and BBR,
respectively. This minimises backlash while still awarding the card of the
partisan judge to the favoured Blue. If the partisan judge sees BRR, however,
then they need to choose between BBR (which awards the card to Blue) and BRR
(which minimises expected backlash). The condition for awarding rounds BRR as
preferred to awarding rounds BBR is:

$\displaystyle\widehat{S}_{\text{Maj. Jgs, BRR}}$
$\displaystyle\leq\frac{1-2\alpha}{6(C|BRR)(1-(C|BRR))}$
$\displaystyle\leq\frac{1-2\alpha}{24\alpha\left(1-\alpha\right)\left(2\alpha^{2}-2\alpha+1\right)\left(4\alpha^{4}-8\alpha^{3}+7\alpha^{2}-3\alpha+1\right)\left(8\alpha^{4}-16\alpha^{3}+10\alpha^{2}-2\alpha+1\right)}$
(A.4)

For a simple example, when $\alpha=0.1$, this expression becomes approximately
$1.09$.

If the partisan judge sees RRR, then they need to choose between BBR (which
awards the card to Blue) and RRR (which minimises backlash). The condition for
RRR to be perferred to BBR is:

$\displaystyle\widehat{S}_{\text{Maj. Jgs, RRR}}$
$\displaystyle\leq\frac{1-2\alpha}{3(C|RRR)(1-(C|RRR))}$
$\displaystyle\leq\frac{1-2\alpha}{12\alpha^{2}\left(\alpha-1\right)^{2}\left(2\alpha^{2}-2\alpha+1\right)^{2}\left(1+4\alpha-4\alpha^{2}\right)\left(4\alpha^{2}-4\alpha+3\right)}$
(A.5)

For a simple example, when $\alpha=0.1$, the right-hand-side here becomes
approximately $12.14$.

It makes intuitive sense that the critical level of blue winning utility will
be higher here than in Equation A.4, as they need to award two more rounds
than they believe Blue won (so higher backlash) and there is lower odds of at
least one other judge awarding in favour of Blue (so less chance that partisan
judging will deliver a victory).

### A.2 Majority Rounds Rule

We can derive the following probabilities of Blue winning a round conditional
on what the partisan judge observes and does:333For instance, we can work out
$q_{1}$ and $q_{3}$ as follows. If we see B, then there is $(1-\alpha)$ chance
that B is the true state and $\alpha$ chance that $R$ is the true state. If B
is the true state, then there is a chance $\alpha$ that a fair judge sees R
and a $1-\alpha$ chance they see B. If R is the true state, then there is a
chance $\alpha$ that a fair judge sees B and a $1-\alpha$ chance they see R.
Adding up the probabilities that B wins on the fair judge’s card we come to
$1-\nabla$. For $q_{3}$, we need both fair judges to score it this way, the
odds of which are $(1-\nabla)^{2}$. For $q_{1}$ we need one (or both) of the
judges to score it this way, the odds of which are $1-\nabla^{2}$. We can work
out $q_{2}$ and $q_{4}$ analogously.

Partisan judge observes B and does B: $\displaystyle q_{1}$
$\displaystyle=1-\nabla^{2}$ (A.6) Partisan judge observes R and does B:
$\displaystyle q_{2}$ $\displaystyle=\nabla(2-\nabla)$ (A.7) Partisan judge
observes B and does R: $\displaystyle q_{3}$ $\displaystyle=(1-\nabla)^{2}$
(A.8) Partisan judge observes R and does R: $\displaystyle q_{4}$
$\displaystyle=\nabla^{2}$ (A.9)

At this point, we define the function $f$ to denote the probability of getting
at least two realisations from 3 binomial distribution trials with
probabilities $p_{1},p_{2},p_{3}$:

$\displaystyle
f(p_{1},p_{2},p_{3})=p_{1}p_{2}(1-p_{3})+p_{1}(1-p_{2})p_{3}+(1-p_{1})p_{2}p_{3}+p_{1}p_{2}p_{3}$
(A.10)

Using this function, we can write the utilities available at each information
set (where the first column shows the information set for each row) and action
(the second to fifth columns), shown in Table A2.

Award BBB Award BBR Award BRR Award RRR BBB
$Sf(q_{1},q_{1},q_{1})-\frac{3\alpha+0(1-\alpha)}{3}$
$Sf(q_{1},q_{1},q_{3})-\frac{2\alpha+1(1-\alpha)}{3}$
$Sf(q_{1},q_{3},q_{3})-\frac{1\alpha+2(1-\alpha)}{3}$
$Sf(q_{3},q_{3},q_{3})-\frac{0\alpha+3(1-\alpha)}{3}$ BBR
$Sf(q_{1},q_{1},q_{2})-\frac{2\alpha+1(1-\alpha)}{3}$
$Sf(q_{1},q_{1},q_{4})-\frac{3\alpha+0(1-\alpha)}{3}$
$Sf(q_{1},q_{3},q_{4})-\frac{2\alpha+1(1-\alpha)}{3}$
$Sf(q_{3},q_{3},q_{4})-\frac{1\alpha+2(1-\alpha)}{3}$ BRR
$Sf(q_{1},q_{2},q_{2})-\frac{1\alpha+2(1-\alpha)}{3}$
$Sf(q_{1},q_{2},q_{4})-\frac{2\alpha+1(1-\alpha)}{3}$
$Sf(q_{1},q_{4},q_{4})-\frac{3\alpha+0(1-\alpha)}{3}$
$Sf(q_{3},q_{4},q_{4})-\frac{2\alpha+1(1-\alpha)}{3}$ RRR
$Sf(q_{2},q_{2},q_{2})-\frac{0\alpha+3(1-\alpha)}{3}$
$Sf(q_{2},q_{2},q_{4})-\frac{1\alpha+2(1-\alpha)}{3}$
$Sf(q_{2},q_{4},q_{4})-\frac{2\alpha+1(1-\alpha)}{3}$
$Sf(q_{4},q_{4},q_{4})-\frac{3\alpha+0(1-\alpha)}{3}$

Table A2: Utilities for each action and information set in the basic model
under majority rounds rule

Similar to the case in Table A1, the utility on the diagonal is better than
the utility from awarding more rounds than this to Red. This implies that if
the partisan judge sees BBB, then they should award rounds as BBB. We can
solve for the levels of $S$ at which the partisan judge prefers to award
rounds fairly rather than giving additional rounds to Blue. Starting at the
BBR information set:

$\displaystyle\widehat{S}_{\text{Maj. Rds, BBR}}$
$\displaystyle\leq\frac{1-2\alpha}{96\alpha^{3}\left(8\alpha^{9}-48\alpha^{8}+124\alpha^{7}-180\alpha^{6}+158\alpha^{5}-80\alpha^{4}+15\alpha^{3}+7\alpha^{2}-5\alpha+1\right)}$
(A.11)

For the BRR information set, the partisan judge could do BBR or BBB rather
than the fair result of BRR. We derive the critical $S$ for both and can
determine that a partisan judge will deviate to BBR at a higher $S$ value than
they would deviate to BBB. Hence, we below report the threshold above which
the partisan judge will not deviate to BBR:

$\displaystyle\widehat{S}_{\text{Maj. Rds, BRR}}\leq$
$\displaystyle\frac{2\alpha-1}{12\alpha\left(64\alpha^{11}-384\alpha^{10}+992\alpha^{9}-1440\alpha^{8}+1264\alpha^{7}-640\alpha^{6}+120\alpha^{5}+56\alpha^{4}-38\alpha^{3}+4\alpha^{2}+3\alpha-1\right)}$
(A.12)

Finally, for the RRR information set, the partisan judge could do BRR, BBR or
BBB, rather than the fair result of RRR. We can establish that if $\alpha$ is
near zero, then the partisan judge will deviate to BBB at a higher $S$ than
they would deviate to the other options. When $\alpha$ is near (but below)
$0.5$, then they will deviate to BBR at a higher $S$ than they would deviate
to the other options. As a result, we have the critical $S$ value:

$\displaystyle\widehat{S}_{\text{Maj. Rds, RRR}}\leq\min\Large[$
$\displaystyle\frac{1-2\alpha}{16\alpha^{2}\left(16\alpha^{10}-96\alpha^{9}+264\alpha^{8}-440\alpha^{7}+504\alpha^{6}-432\alpha^{5}+294\alpha^{4}-162\alpha^{3}+69\alpha^{2}-20\alpha+3\right)},$
$\displaystyle\frac{2\alpha-1}{24\alpha^{2}\left(16\alpha^{8}-80\alpha^{7}+172\alpha^{6}-208\alpha^{5}+152\alpha^{4}-64\alpha^{3}+11\alpha^{2}+2\alpha-1\right)}\Large]$
(A.13)

Now we can summarise Equations A.4, A.5, A.11, A.12 and A.13 with a chart of
$\alpha$ against the critical $S$ ratio below which the partisan judge does
not mis-award rounds. This is shown in Figure A1.

FIGURE A1: Critical $S$ below which partisan judges do award fairly for three-
round bouts with three judges in the BRR and RRR information sets

In all cases where Red wins more rounds, the majority rounds rule has a higher
$S$ value at which the partisan judge is indifferent to awarding fairly and
giving more rounds to Blue. This indicates that the majority rounds rule is
more robust to partisan judging.

##  Appendix B No judges are partisan

When no judges are partisan, the proposed change, from awarding bouts by
majority judges to majority rounds, reduces the probability of bouts being
wrongly decided. This can be seen in Figure B1, which is comparable to Figure
2, with the same parametrisation, but reflects the case where all three judges
are fair.

FIGURE B1: Probability of a “correct” result depending on the number of rounds
truly won by blue and how judges’ scores are aggregated: the case of no
partisan judges

##  Appendix C Other parametrisations

We consider different model parametrisations, to demonstrate the extent to
which the qualitative results of this paper may vary.

High disagreement between different judges

We increase the noise variance, such that the different judges disagree more
often about the outcome of a round. Specifically, we increase $\alpha=0.2$ and
leave the other parameters as they were in the main body of the paper. Figures
1-3 are reproduced below for this new parametrisation as Figures C1\- C3.

FIGURE C1: Probability of favoured boxer winning in one bout where both boxers
won 6 rounds (in the absence of noise) FIGURE C2: Probability of correct
result for bout FIGURE C3: Probability of each outcome for bout

High degree of favouritism

We increase $S$ to 0.8 and leave other parameter values as they are in the
main body of the paper. Figures 1-3 are reproduced below for this new
parametrisation as Figures C4-C6

FIGURE C4: Probability of favoured boxer winning in one bout where both boxers
won 6 rounds (in the absence of noise) FIGURE C5: Probability of correct
result for bout FIGURE C6: Probability of each outcome for bout

##  Appendix D Women’s professional boxing

In women’s professional boxing, there are generally 10 rounds and 3 judges.
Figures 1\- 3 are reproduced below for women’s professional boxing as Figures
D1-D3, with otherwise identical parametrisations.

FIGURE D1: Probability of favoured boxer winning in one bout where both boxers
won 6 rounds (in the absence of noise) - Women’s professional boxing FIGURE
D2: Probability of correct result for bout - Women’s professional boxing
FIGURE D3: Probability of each outcome for bout - Women’s professional boxing

##  Appendix E Men’s Olympic amateur boxing

In men’s Olympic amateur boxing, there are generally 3 rounds and 5 judges.
Figures 1-3 are reproduced below for men’s Olympic amateur boxing as Figures
E1-E3.

FIGURE E1: Probability of favoured boxer winning in one bout where the blue
boxer won 1 round and the red boxer won 2 (in the absence of noise) - Men’s
Olympic amateur boxing FIGURE E2: Probability of correct result for bout -
Men’s Olympic amateur boxing FIGURE E3: Probability of each outcome for bout -
Men’s Olympic amateur boxing

##  Appendix F Women’s Olympic amateur

In women’s Olympic amateur boxing, there are generally 4 rounds and 5 judges.
Figures 1-3 are reproduced below for women’s Olympic amateur boxing as Figures
F1-F3.

FIGURE F1: Probability of favoured boxer winning in one bout where both boxers
won 6 rounds (in the absence of noise) - Women’s Olympic amateur boxing FIGURE
F2: Probability of correct result for bout - Women’s Olympic amateur boxing
FIGURE F3: Probability of each outcome for bout - Women’s Olympic amateur
boxing


# Searching for Lorentz-violating Signatures from Astrophysical Photon
Observations

Jun-Jie Wei1,2,3 1Purple Mountain Observatory, Chinese Academy of Sciences,
Nanjing 210023, China  2School of Astronomy and Space Sciences, University of
Science and Technology of China, Hefei 230026, China  3Guangxi Key Laboratory
for Relativistic Astrophysics,
Nanning 530004, China

###### Abstract

As a basic symmetry of Einstein’s theory of special relativity, Lorentz
invariance has withstood very strict tests. But there are still motivations
for such tests. Firstly, many theories of quantum gravity suggest violations
of Lorentz invariance at the Planck energy scale. Secondly, even minute
deviations from Lorentz symmetry can accumulate as particle travel across
large distances, leading to detectable effects at attainable energies. Thanks
to their long baselines and high-energy emission, astrophysical observations
provide sensitive tests of Lorentz invariance in the photon sector. In this
paper, I briefly introduce astrophysical methods that we adopted to search for
Lorentz-violating signatures, including vacuum dispersion and vacuum
birefringence.

##  1 Introduction

Although experimental tests of Lorentz invariance violation (LIV) have been
performed in a wide range of systems (see Ref. datatables for a compilation of
results), there are still motivations for such tests. From theoretical
consideration, establishing a quantum theory that includes gravity is being
hailed as the Holy Grail of modern physics. However, theories of quantum
gravity (QG) predict that Lorentz invariance may be violated at the Planck
energy scale $E_{\rm Pl}$. From experimental feasibility, even very tiny
deviations from Lorentz symmetry can become measurable at attainable energies
$\ll E_{\rm Pl}$, since Lorentz-violating effects gradually accumulate over
large propagation distances. In brief, both the prospect of relativity
violations arising in a grand unified theory and the feasibility of
discovering LIV have attracted physicists to constantly work on experimental
searches.

Astrophysical observations involving long baselines are very suitable to
search for LIV effects. In vacuum, the LIV-induced modifications to the photon
dispersion relation can produce rich and detectable astrophysical phenomena.
With a slight modification, the speed of light will no longer be independent
of frequency and polarization. If the modified dispersion relation is related
to the frequency of photons, then we may observer a frequency-dependent vacuum
dispersion of light. If the modification is related to the circular
polarization state of photons, the photons with right- and left-handed
polarization states have different velocities, then we may observer vacuum
birefringence. Vacuum dispersion can be tested by measuring the arrival-time
differences of photons with different frequencies originating from the same
astrophysical source (see, e.g., Ref. Amelino-Camelia). Vacuum birefringence
results in a frequency-dependent rotation of the polarization vector of
linearly polarized light, which can be tested by astrophysical
spectropolarimetric measurements (see, e.g., Refs. Kostelecky,Laurent).

In this paper, we present some recent tests of nonbirefringent LIV using
spectral-lag transitions of gamma-ray bursts (GRBs),[5, 6] and some searches
for LIV using polarization measurements of GRBs and blazars.[7, 8]

##  2 Vacuum dispersion

Vacuum dispersion is a potential LIV signature. The arrival time delay between
photons with different energies ($E_{h}>E_{l}$) emitted simultaneously from
the source at redshift $z$ can be derived by introducing the LIV terms in a
Taylor series:[9]

$\Delta t=s_{\pm}\frac{1+n}{2H_{0}}\frac{E_{h}^{n}-E_{l}^{n}}{E_{{\rm
QG},n}^{n}}\int_{0}^{z}\frac{(1+z^{\prime})^{n}dz^{\prime}}{\sqrt{\Omega_{\rm
m}(1+z^{\prime})^{3}+\Omega_{\Lambda}}}\;,$ (1)

where $s_{\pm}=\pm 1$ is the sign of LIV, corresponding to the “subluminal”
($s_{\pm}=+1$) or “superluminal” ($s_{\pm}=-1$) scenarios. $E_{\rm QG}$
denotes the QG energy scale at which LIV effects become significant, $n=1$
($n=2$) corresponds to the linear (quadratic) energy dependence, and ($H_{0}$,
$\Omega_{\rm m}$, $\Omega_{\Lambda}$) are the cosmological parameters of the
standard flat $\Lambda$CDM model.

GRBs are ideal astrophysical phenomena that one can used to perform time-of-
flight tests because they are the most distant transient sources involving a
wide range of photon energies. However, a key challenge in such tests is to
distinguish an intrinsic astrophysical time delay at the source from a time
delay induced by LIV. We proposed that GRB 160625B, the burst having an
apparent transition from positive to negative spectral lags, provides a good
opportunity to disentangle the intrinsic time delay problem.[5] Spectral lag,
the arrival time difference between high- and low-energy photons, is
conventionally defined to be positive if the high-energy photons precede the
low-energy ones. In the subluminal case ($s_{\pm}=+1$), photons with higher
energies would arrive at the observer after those with lower ones, implying a
negative spectral lag due to LIV. Assuming the source-intrinsic time lag to
have a positive dependence on the photon energy, and considering the
contributions to the spectral lag from both the intrinsic positive lag and
LIV-related negative lag, we derived new limits on linear and quadratic
leading-order Lorentz-violating vacuum dispersion by directly fitting the
spectral lag behavior of GRB 160625B. Recently, similar time-of-flight tests
were carried out by analyzing the spectral-lag transition of GRB 190114C.[6]

##  3 Vacuum birefringence

Vacuum birefringence is another potential LIV signature. For a source at
redshift $z$, the LIV-induced rotation angle of the polarization vector of a
linearly polarized wave is[4]

$\Delta\phi(E)\simeq\eta\frac{E^{2}}{\hbar E_{\rm
Pl}H_{0}}\int_{0}^{z}\frac{(1+z^{\prime})dz^{\prime}}{\sqrt{\Omega_{\rm
m}(1+z^{\prime})^{3}+\Omega_{\Lambda}}}\;,$ (2)

where $E$ is the energy of the observed light and $\eta$ is a dimensionless
parameter characterizing the degree of LIV effects.

If the rotation angles of photons with different energies differ by more than
$\pi/2$ over an energy band ($E_{1}<E<E_{2}$), significant depletion of the
initial polarization of the signal is expected. The detection of high
polarization can therefore set upper limits on the birefringent parameter
$\eta$. We gave a detailed calculation on the GRB polarization evolution
arising from the birefringent effect, and confirmed that the initial
polarization is not significantly depleted even if the differential rotation
angle $|\Delta\phi(E_{2})-\Delta\phi(E_{1})|$ is as large as $\pi/2$.[7]
Applying our formulate for calculating LIV-induced polarization evolution to
the gamma-ray polarimetric data of 12 GRBs, we improved existing bounds on the
birefringent parameter $\eta$ by factors ranging from 2 to 10.

If all photons in the observed energy range are assumed to be emitted with the
same intrinsic polarization angle, we expect to observe vacuum birefringence
as an energy-dependent linear polarization vector. We tried to search for an
energy-dependent change of the linear polarization angle in the
spectropolarimetric data of 5 blazars.[8] At the $2\sigma$ confidence level,
the absence of the birefringent effect was limited to be in the range of
$-9.63\times 10^{-8}<\eta<6.55\times 10^{-6}$. As might be expected, optical
polarimetric data of blazars pose less stringent constraints on $\eta$ as
compared to gamma-ray polarizations of GRBs.

##  4 Summary

GRBs are promising astrophysical sources for searching for LIV-induced vacuum
dispersion and vacuum birefringence. Future detections of extremely-high-
energy emission from GRBs with LHAASO, MAGIC, HAWC, and the future
international Cherenkov Telescope Array could improve the limits on LIV using
vacuum dispersion (time-of-flight) tests. Future X-ray/gamma-ray polarization
measurements of GRBs with POLAR-II, TSUBAME, COSI, and GRAPE could also
improve the limits on LIV using vacuum birefringence tests.

## Acknowledgments

This work is partially supported by the National Natural Science Foundation of
China (grant Nos. 11725314 and 12041306), the Key Research Program of Frontier
Sciences (grant No. ZDBS-LY-7014) of Chinese Academy of Sciences, the Major
Science and Technology Project of Qinghai Province (2019-ZJ-A10), the Natural
Science Foundation of Jiangsu Province (grant No. BK20221562), the China
Manned Space Project (CMS-CSST-2021-B11), and the Guangxi Key Laboratory for
Relativistic Astrophysics.

## References

  * [1] Data Tables for Lorentz and CPT Violation, V.A. Kostelecký and N. Russell, 2022 edition, arXiv:0801.0287v15.
  * [2] G. Amelino-Camelia et al., Nature 393, 763 (1998); A.A. Abdo et al., Nature 462, 331 (2009); V. Vasileou et al., Phys. Rev. D 82, 122001 (2013).
  * [3] V.A. Kostelecký and M. Mewes, Phys. Rev. Lett. 87, 251304 (2001); V.A. Kostelecký and M. Mewes, Phys. Rev. Lett. 110, 201601 (2013).
  * [4] P. Laurent et al., Phys. Rev. D 83, 121301 (2011); K. Toma et al., Phys. Rev. Lett. 109, 241104 (2012).
  * [5] J.-J. Wei et al., Astrophys. J. Lett. 834, L13 (2017).
  * [6] S.-S. Du et al., Astrophys. J. 906, 8 (2021).
  * [7] J.-J. Wei, Mon. Not. Royal Astron. Soc. 485, 2401 (2019).
  * [8] Q.-Q. Zhou et al., Galaxies 9, 44 (2021).
  * [9] U. Jacob and T. Piran, J. Cosmol. Astropart. Phys. 01, 031 (2008).


# DroneWorld: High Fidelity Simulation tool for testing small Uncrewed Aerial
System Applications

Bohan Zhang, Yashaswini Shivalingaiah, and Ankit Agrawal
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Department of Computer Science, Saint Louis
University
Saint Louis, MO, USA

# DroneReqValidator: Facilitating High Fidelity Simulation Testing for
Uncrewed Aerial Systems Developers

Bohan Zhang, Yashaswini Shivalingaiah, and Ankit Agrawal
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Department of Computer Science, Saint Louis
University
Saint Louis, MO, USA

###### Abstract

Rigorous testing of small Uncrewed Aerial Systems (sUAS) is crucial to ensure
their safe and reliable deployment in the real world. sUAS developers aim to
validate the reliability and safety of their applications through simulation
testing. However, the dynamic nature of the real-world environment, including
factors such as challenging weather conditions and wireless interference,
causes unique software faults that may only be revealed through field testing.
Considering the high cost and impracticality of conducting field testing in
thousands of environmental contexts and conditions, there exists a pressing
need to develop automated techniques that can generate high-fidelity,
realistic environments enabling sUAS developers to deploy their applications
and conduct thorough simulation testing in close-to-reality environmental
conditions. To address this need, DroneReqValidator (DRV) offers a
comprehensive small Unmanned Aerial Vehicle (sUAV) simulation ecosystem that
automatically generates realistic environments based on developer-specified
constraints, monitors sUAV activities against predefined safety parameters,
and generates detailed acceptance test reports for effective debugging and
analysis of sUAV applications. Providing these capabilities, DRV offers a
valuable solution for enhancing the testing and development process of sUAS.
The comprehensive demo of DRV is available at
https://www.youtube.com/watch?v=Fd9ft55gbO8

###### Index Terms:

Simulation, Automated Analysis, Uncrewed Aerial Vehicles

##  I Introduction

The advancement of artificial intelligence has led to the evolution of smart
small Uncrewed Aerial Systems (sUAS). The applications of smart sUAS have
expanded from package delivery [3] in densely populated cities to military
reconnaissance in remote areas [12], incorporating advanced decision-making
capabilities to enable autonomous missions in various environmental conditions
[5]. These novel applications of Unmanned Aerial Vehicles (UAVs) have posed
numerous challenges in testing their safety and reliability in adverse
environmental contexts such as wind gusts, non-uniform deterioration of
satellite signals, and fluctuations in cellular or 5G signals.

During system validation, UAV software application developers encounter two
primary issues. Firstly, while simulation testing helps in validating the
behavior of the UAV application, it often fails to accurately represent the
realistic and dynamic nature of the operational environment, resulting in
failures during field deployments. Further, certain field tests present
substantial difficulties when performed on physical UAVs, especially those
that involve pushing or exceeding operational limits, such as flying in
extreme weather conditions or close proximity to objects or humans. Secondly,
UAV application developers are usually experts in Software Engineering, while
developing highly realistic 3D environments requires expertise in graphics,
design, and specialized tools such as Unity [13] or Unreal Engine [1]. This
gap in skills presents a major challenge for sUAS developers in their ability
to efficiently and rapidly generate the desired 3D environment for their
simulation testing needs.

Figure 1: Four UAVs surveilling the realistic digital twin model of Chicago
city in DroneReqValidator

Currently, sUAS developers heavily rely on simulations using either 2D maps or
3D simulation environments like Gazebo [6] or AirSim [11]. However, these
existing simulation tools suffer from certain limitations. First, they require
the manual construction of 3D environments, increasing developers’ workload.
This manual process is time-consuming, resource-intensive, and can introduce
inaccuracies and limitations in the realism of the simulated environments.
Second, the responsibility of collecting and analyzing simulation data falls
on the developers as they need to design and implement custom code for
relevant data collection, plotting, and analysis. The lack of automated data
handling tools adds to the complexity of the testing process and further
increases the developers’ workload and reduces productivity [8, 4]. Third,
these simulation tools lack a standardized mechanism for specifying and
evaluating test conditions [16]. Due to this limitation, testing through the
existing simulation tool results in ad-hoc simulation testing. In summary,
current UAV simulation tools do not offer a complete testing harness that can
be effectively used by large teams of sUAS developers.

In contrast, our tool, DroneReqValidator, referred to as DRV in the remainder
of the paper, offers a comprehensive simulation ecosystem for sUAS simulation
testing. The primary goal of DRV is to automate the generation of realistic
environments, based on developer-specified constraints, and facilitate the
monitoring of sUAV activities against predefined safety properties.
Additionally, DRV generates detailed acceptance test reports containing
graphical plots and clear descriptions of failed and passed test cases, which
are useful for effective debugging and analysis of sUAS. DRV leverages the
advancements in Unreal Engine [1], Google Earth digital twin models [7], and
AirSim APIs[11] to enable the automatic generation of a realistic simulation
environment and simulating multiple UAVs in them. DRV’s ultimate goal is to
provide sUAS application developers with a specialized test harness that
serves as a structured framework for simulation testing, facilitate simulation
testing in large-scale sUAS projects, assist developers in identifying
problems and faults resulting from the dynamic nature of the environment
during simulation testing, and increase developer confidence before field
testing.

The remainder of this tool demo paper is organized as follows. Section II
describes DRV’s architecture, and discusses its main components and features,
while Section III outlines preliminary analysis, and open challenges, and
discusses DRV’s current status. Finally, Section IV includes links to the
codebase of every component of DRV.

##  II DRV Architecture

DRV employs a client-server architecture consisting of a front-end and a back-
end to promote collaboration among developers during simulation testing. The
front-end allows developers to configure simulation environments based on the
application requirements and acceptance tests, while the back-end executes
simulations in the digital twin models of specified geographical locations and
generates acceptance test reports. The following section provides details of
each component.

###  II-A Front-End Client

DRV’s front-end component is a user-friendly web application built with the
popular React web framework. It offers an intuitive Graphical User Interface
(GUI) that simplifies the process of setting up the simulation environment,
specifying UAV sensor configuration, and configuring safety test properties.
DRV uses a three-step wizard interface to guide users seamlessly through the
configuration of each aspect of simulation testing.

####  II-A1 Realistic Environment Configuration

The first step in the wizard enables developers to specify the key elements of
a realistic simulation environment. These elements include the geographical
region in which developers wish to conduct the simulations, weather conditions
including wind, and the specific time of day. Based on these inputs, the
system’s back-end creates a 3D digital twin model of the specified region,
incorporates wind in the designated direction and velocity, and adjusts
lighting conditions according to the specified time of day. The details of the
back-end are discussed in Section II-B. Figure 1 shows four UAVs flying in
DRV’s digital twin model of Chicago in the afternoon.

####  II-A2 UAV Sensor Configuration

Multi-UAV software applications could include heterogeneous UAVs and therefore
developers need the ability to easily configure the sensor properties of each
participating UAV in the mission and conduct UAV-specific acceptance testing.
DRV’s GUI caters to this requirement by allowing developers to configure the
specifications of each UAV involved in the mission. This includes defining the
number of sUAVs to be deployed, specifying the sensor configurations for each
individual sUAV, and setting their respective home geolocations within the
simulation environment. The current version of DRV allows developers to
configure major sensors such as GPS, Camera, Lidar, Barmoter, and
Magnetometer. Based on these inputs, the back-end deploys multiple sUAS with
the defined sensor configurations at the specified geo-locations in the
simulated environment. To further explore the user interface and available
configurable properties, please refer to the DRV Demo video.

Figure 2: DRV UI to specify the safety test properties

####  II-A3 Test Property Configuration

One of the primary objectives behind the development of DRV is to facilitate
the seamless specification of pass and fail criteria during simulation
testing. As a crucial step in the configuration, users are required to specify
a range of metrics and conditions that determine the success of a simulation.
Figure 2 demonstrates the user interface for configuring test properties. In
our current version of DRV, developers have the ability to define test cases
based on various metrics. These metrics include configuring the maximum
permissible deviation from the intended flight path, configuring the minimum
separation distance UAVs shall maintain from each other, tracking the
occurrences of collisions with environmental elements, configuring safe
landing spots and enabling tracking UAV landings, configuring no-fly zones and
tracking whether UAVs enter such restricted zones or not during the
simulation. These preliminary test properties were specifically selected based
on sUAS hazards and safety requirements [2, 14].

Figure 3: Flow Chart depicting the interaction between major components of DRV

###  II-B Back-End Server

The back-end server of DRV is built entirely in Python and utilizes the Flask
framework. It comprises of several key components, including the REST
Controller, UAV Controller, and Monitoring layers. The REST controller layer
is primarily responsible for receiving and processing the developers’
specifications regarding the environment, UAV sensors, and the safety test
properties provided through the front-end client. The REST controller layer
also validates the input data and transfers it to the control and monitoring
layer for further processing.

The UAV Controller and Monitoring layer establish communication with Unreal
Engine [1] via Airsim APIs [11], allowing the back-end to instantiate the 3D
environment and accurately simulate the behavior of UAVs within it. UAV
Controller and Monitor manage tasks through a task queue, ensuring that
developers’ simulation requests are processed in the order they are received.
This approach enables effective control and monitoring of multiple UAVs in the
simulation environment while preserving the integrity of the overall
simulation process. The interaction between the different components is shown
in Figure 3. The unique features of the simulation are discussed next.

####  II-B1 Automatic Realistic Environment Generation

DRV’s Automatic Environment Generation component utilizes the advanced
capabilities of the Unreal Engine to deliver highly realistic 3D environments,
as depicted in Fig. 1.

DRV’s current version encompasses four distinctive maps for diverse testing:
Blocks, City Park, Chicago Scanning, and Cesium. The Blocks map offers a
simplified and lightweight 3D environment, suitable for quick testing of sUAS
functionality. The City Park map showcases the Unreal Engine’s cutting-edge
technologies such as Lumen lighting and Nanite, enabling highly realistic
urban environment simulations. The Chicago Scanning map provides a precise 3D
representation of the city of Chicago, serving as an ideal testbed for
evaluating sUAV maneuverability in dense urban settings. Finally, the Cesium
map supports real-time streaming of satellite scanning and terrain data of
real-world geolocations, enabling developers to simulate sUAS in real-world
locations with accurate terrain representation.

####  II-B2 Automated Fuzzy Testing

Fuzzy testing [9] is a software testing technique that aims to discover
vulnerabilities in the system’s behavior by changing the inputs to the system
iteratively. In sUAS applications, where environmental conditions can be
unpredictable and challenging, fuzzy testing becomes particularly valuable.
DRV automatically manipulates the developers’ specified environmental factors
to test UAV application behavior in extreme and diverse environmental
conditions. This automated testing approach enables developers to determine
the system’s performance limits in real-world scenarios.

####  II-B3 Runtime Monitoring and Report Generation

Figure 4: Report snippet showing pass/fail acceptance tests

Runtime monitors collect data and observe the interaction between the system
and the environment [10]. DRV utilizes AirSim APIs to track the activities of
simulated UAVs in the 3D environment. The runtime monitors continuously
collect sensor data from the UAVs, including position, speed, altitude, camera
feed, lidars, and telemetry. This data enables identifying any safety property
violations in the UAV behavior. When a safety violation is detected, potential
issues are logged and made available to the report generator. The details of
the violations, along with data analysis in the form of graphical plots are
then rendered on the user interface for developers to analyze.

Fig. 4 presents two instances of the graphical plots in the acceptance report
for the UAV’s maximum flight path deviation safety property, highlighting the
successful case in green and the failed case in red. This visual
representation offers additional insights and specificity to the developers.

These automatically generated reports are a valuable resource for sUAS
application developers as they streamline the process of identifying and
diagnosing issues, saving time and effort in development and troubleshooting.
Additionally, the automatically generated acceptance test report can be
augmented as evidence in Safety Assurance Cases to argue the safety of the
sUAS [2, 15].

##  III Ongoing Development and Challenges

  * •

Simulation at Scale: As the number of Unmanned Aerial Vehicles (UAVs) in the
simulation increases, the demand on both the Central Processing Unit (CPU) and
Graphical Processing Unit (GPU) also increases. To illustrate this challenge,
a stress test was conducted on a machine with the specifications outlined in
Table I. The findings of the stress test are depicted in Figure 5, which
illustrates the CPU and GPU usage over time during a simulation involving 5
and 20 sUAVs, respectively. First, the graph shows that CPU demand increases
as the number of UAVs in the simulation environment increases. The CPU demand
increases because the UAV Controller and the Monitors component rapidly switch
threads, impacting performance. Second, interestingly, we also observed that
the GPU utilization for a fleet of 20 sUAVs is comparatively low. In this
case, the CPU bottleneck hinders the rendering of the environment, thereby
decreasing the demand for GPUs.

Component | Specification
---|---
GPU | NVIDIA GeForce RTX 3090 Ti 24G
CPU | Intel Core i9-10900
RAM | 64 GB
TABLE I: System Specifications for Preliminary Stress Tests Figure 5: Stress
Test Results: 5 Vs 20 sUAVs simulation

  * •

Geo-Cordinate System Alignment:

DRV integrates the AirSim and Cesium Unreal plugins along with Google Earth
APIs in the Unreal Engine to simulate UAVs in realistic geographical
environments. However, inconsistencies in the coordinate systems between
AirSim and Cesium pose challenges in accurately aligning geo-coordinates and
elevation data. During our initial testing phase, we observed a minor
discrepancy between the physical movement of a sUAV in the real world and its
corresponding translation within the 3D environment of DRV. Our testing also
revealed that Google Maps elevation data occasionally does not match the
elevation in the Cesium, resulting in UAVs flying below terrain
unintentionally. By implementing precise coordinate transformation algorithms,
calibrating the scaling factors, and validating elevation data, we are
actively addressing the challenges associated with coordinate alignment and
elevation.

  * •

Continious Integration Pipeline Integration: DRV currently operates as a
client-server application, but we’re actively working on implementing a
Continuous Integration (CI) pipeline for enhanced testing. CI pipeline will
automatically execute simulation tests in DRV after every commit to the sUAS
application’s Git repository, reducing developers’ manual testing workload.
Moreover, the availability of acceptance test reports after each commit will
enable quick identification of issues resulting from specific commits to the
sUAS applications’s code repository.

##  IV CODE AVAILABILITY

All components of the DRV ecosystem are available on our public Github
respository as open source projects 111https://github.com/UAVLab-
SLU/DRV_public.

## References

  * [1] The most powerful real-time 3d creation tool - unreal engine.  https://www.unrealengine.com/en-US, note = (Accessed on 05/25/2023).
  * [2] A. Agrawal, S. Khoshmanesh, M. Vierhauser, M. Rahimi, J. Cleland-Huang, and R. Lutz.  Leveraging artifact trees to evolve and reuse safety cases.  In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pages 1222–1233. IEEE, 2019.
  * [3] K.-W. Chen, M.-R. Xie, Y.-M. Chen, T.-T. Chu, and Y.-B. Lin.  Dronetalk: An internet-of-things-based drone system for last-mile drone delivery.  IEEE Transactions on Intelligent Transportation Systems, 23(9):15204–15217, 2022.
  * [4] L. Cheng, E. Murphy-Hill, M. Canning, C. Jaspan, C. Green, A. Knight, N. Zhang, and E. Kammer.  What improves developer productivity at google? code quality.  In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 1302–1313, 2022.
  * [5] A. Ilachinski.  Artificial intelligence and autonomy: Opportunities and challenges.  2017\.
  * [6] N. Koenig and A. Howard.  Design and use paradigms for gazebo, an open-source multi-robot simulator.  In Proc. of the 2004 Int’l Conference on Intelligent Robots and Systems, volume 3, pages 2149–2154. IEEE, 2004.
  * [7] G. Maps.  Map tiles api.  https://developers.google.com/maps/documentation/tile, 2021.  Accessed on May 11, 2023.
  * [8] E. Murphy-Hill, C. Jaspan, C. Sadowski, D. Shepherd, M. Phillips, C. Winter, A. Knight, E. Smith, and M. Jorde.  What predicts software developers’ productivity?  IEEE Transactions on Software Engineering, 47(3):582–594, 2019\.
  * [9] H. L. Nguyen and L. Grunske.  Bedivfuzz: integrating behavioral diversity into generator-based fuzzing.  In Proceedings of the 44th International Conference on Software Engineering, pages 249–261, 2022.
  * [10] R. Rabiser, S. Guinea, M. Vierhauser, L. Baresi, and P. Grünbacher.  A comparison framework for runtime monitoring approaches.  Journal of Systems and Software, 125:309–321, 2017.
  * [11] S. Shah, D. Dey, C. Lovett, and A. Kapoor.  Airsim: High-fidelity visual and physical simulation for autonomous vehicles.  In Field and service robotics, pages 621–635. Springer, 2018.
  * [12] P. Stodola, J. Drozd, J. Mazal, J. Hodickỳ, and D. Procházka.  Cooperative unmanned aerial system reconnaissance in a complex urban environment and uneven terrain.  Sensors, 19(17):3754, 2019.
  * [13] Unity.  Unity real-time development platform — 3d, 2d, vr & ar engine.  https://unity.com/.  (Accessed on 05/25/2023).
  * [14] M. Vierhauser, S. Bayley, J. Wyngaard, W. Xiong, J. Cheng, J. Huseman, R. Lutz, and J. Cleland-Huang.  Interlocking safety cases for unmanned autonomous systems in shared airspaces.  IEEE transactions on software engineering, 47(5):899–918, 2019\.
  * [15] M. Vierhauser, M. N. A. Islam, A. Agrawal, J. Cleland-Huang, and J. Mason.  Hazard analysis for human-on-the-loop interactions in suas systems.  In Proc. of the 29th ACM Joint Meeting on European Software Engineering Conf. and Symposium on the Foundations of Software Engineering, pages 8–19, 2021.
  * [16] X. Zheng, C. Julien, M. Kim, and S. Khurshid.  Perceptions on the State of the Art in Verification and Validation in Cyber-Physical Systems.  IEEE Systems Journal, 11(4):2614–2627, 2015.