# Zipr: A High-Impact, Robust, Open-source, Multi-platform,
Static Binary Rewriter

Jason D. Hiser, Anh Nguyen-Tuong, Jack W. Davidson
<EMAIL_ADDRESS>University of Virginia85 Engineer’s
WayCharlottesvileVirginiaUSA22904

(2023; 2 October 2023)

###### Abstract.

Zipr is a tool for static binary rewriting, first published in 2016. Zipr was
engineered to support arbitrary program modification with an emphasis on low
overhead, robustness, and flexibility to perform security enhancements and
instrumentation. Originally targeted to Linux x86-32 binaries, Zipr now
supports 32- and 64-bit binaries for X86, ARM, and MIPS architectures, as well
as preliminary support for Windows programs.

These features have helped Zipr make a dramatic impact on research. It was
first used in the DARPA Cyber Grand Challenge to take second place overall,
with the best security score of any participant, Zipr has now been used in a
variety of research areas by both the original authors as well as third
parties. Zipr has also led to publications in artificial diversity, program
instrumentation, program repair, fuzzing, autonomous vehicle security,
research computing security, as well as directly contributing to two student
dissertations. The open-source repository has accepted accepted patches from
several external authors, demonstrating the impact of Zipr beyond the original
authors.

Binary Rewriting, Retargetable Binary Analysis, Reverse Engineering

††copyright: cc††journalyear: 2023††conference:  Cybersecurity Artifacts
Competition and Impact Award ; December 04–08, 2023; Austin, TX††ccs: Software
and its engineering Software maintenance tools††ccs: Software and its
engineering Translator writing systems and compiler generators††ccs: Security
and privacy Software reverse engineering

##  1\. Introduction

The impetus for creating Zipr was the need to apply many patches to arbitrary
binaries efficiently. We had a static analysis system for binaries that could
identify potential buffer overflows and recommend patches, and we needed to
determine the runtime costs of applying the patches. While we had access to
dynamic binary translators, runtime and memory overheads were not appropriate
for the embedded systems we were targeting. (Scott and Davidson, 2001; Scott
et al., 2003; Hiser et al., 2007) At the time, there were two flavors of
static binary rewriters: Ones that kept a copy of the original program text to
deal with disassembly errors and ones that overwrote the code to be
instrumented with a trampoline to an unused address to execute the
instrumented code snippet, then return to the original code to continue
execution.

Keeping a full second copy of the program was untenable for embedded systems.
Further, we knew that our static analyzer likely had many false positives (as
static analyzers tend to do), so we did not want our instrumentation to suffer
from the cache, branch predictor, and memory penalty overheads of frequent
trampolining inside time-critical kernel loops. We needed a rewriting system
suitable for embedded systems, and nothing available met the requirements: 1)
low overhead (in terms of memory and performance), 2) robust for a large range
of common programs, and 3) the ability to cheaply invoke arbitrary
instrumentation for any instruction in the program. Of course, these features
are desirable in most environments but absolutely necessary for embedded
systems.

We realized that instead of trampolining for patched instructions, we could
trampoline only the (comparatively infrequent) indirect branch instruction
targets. This approach allows instrumentation of any instruction. Zipr works
by putting a trampoline at each indirect branch target, and then places the
remaining code blocks around these trampolines. To save space, Zipr lays out
blocks between trampolines with a best-fit algorithm, and often simply
replaces the trampoline with the correct code, avoiding any overhead at all.

The primary publications describing Zipr were published in 2016 and 2017
(Davidson et al., 2016; Hawkins et al., 2017a; Hiser et al., 2017; Hawkins et
al., 2017b). The open source release (LLC. et al., 2023b) occurred in 2019
after a dependency on a commercial software package, IDA Pro (Eagle, 2011),
was eliminated. Zipr has since been extended for multiple platforms and
demonstrated effective, robust binary rewriting. Section 2 discusses the Zipr
artifact’s current release, while Section 3 describes the impacts that Zipr
has made in artificial diversity, program instrumentation, program repair,
fuzzing, autonomous vehicle security, research computing security, as well as
directly contributing to two student dissertations and numerous papers in key
areas of security and privacy.

##  2\. The Zipr Artifact

#### Zipr Architecture

The Zipr architecture includes a front end that parses binary programs and
lifts it into a low-level Zipr intermediate representation (IR). This IR is
stored in a database we call the IRDB. The front end detects instructions,
functions, data objects, indirect branch targets, and stack unwinding
information (also called exception handling (EH) tables) and deposits this
information in the IRDB. The IRDB contains information about control
transfers, data object locations, and whether the IR is complete enough to
_unpin_ a data object or instruction, etc. One key feature of the IRDB storage
mechanism is that it allows for uncertain and conflicting information. For
example, our disassembly technique allows multiple disassemblers to union
their results. Infrequently, disassemblers disagree on the correct
interpretation of a sequence of bytes. The IRDB stores _both_ representations
in the IRDB, and ultimately, both interpretations will end up in the final
program (assuming no user-specified transformations remove them). While only
one interpretation is correct, the program semantics will be preserved because
only one of them will ever execute dynamically. The incorrect sequence will
result in extra memory and disk space usage, but will not affect correctness.

After building the IR, the user can select one or more composable
transformations to the IR. To ease the task of writing transformations, Zipr
provides an SDK for reading, writing, and modifying the IRDB. (LLC. and Hiser,
2023a) The current SDK is written for C++ programming, but it has always been
envisioned that one could directly modify the IRDB using SQL or write
interfaces for other languages.

Besides direct access to any program item, we recognized that many
transformations would want to analyze the program in some way before making a
transformation. To facilitate this approach, the SDK provides common analysis
techniques, such as the ability to create a control flow graph (CFG) for a
function; the dominator graph for the CFG, a call graph of functions in the
IR; the dead registers at each program point; and the disassembly and set/used
registers for each instruction. While the set of analyses is by no means
complete, the SDK is easily extendable by writing a new transformation and
compiling it into a dynamically-linked library that others can reuse.

To help potential users write Zipr transformations, we have provided a variety
of example transformations:

  * •

An IRDB Cookbook (LLC. and Hiser, 2023c) with well-documented transformations
provides documentation on writing a plugin and includes three example plugins.
initialize_stack inserts code at the beginning of every function to zero-
initialize the stack, preventing exploits relating to uninitialized stack
variables. This example shows how to iterate over functions, inspect a
function’s stack frame, and insert instrumentation. kill_deads runs the IRDB
SDK’s dead register analysis and inserts a set for every dead register
detected at every instruction in the program. This example shows how to invoke
IRDB SDK analyses and act on the results, while simultaneously being a
surprisingly effective test of the analysis results. stack_stamp modifies
function entry and exit code to XOR the program’s return address with a
randomly selected value. Besides being a potential security enhancement, this
example shows how to iterate over functions and modify each function’s entry
and exit points.

  * •

Zafl is an instrumentation pass built on Zipr and the IRDB SDK to instrument a
program with AFL-style instrumentation. (LLC. et al., 2023c; Nagy et al.,
2021a; Wong, 2022; Fioraldi et al., 2020). Zafl yields an instrumented program
that is statistically similar to a program with compiler-inserted
instrumentation in terms of bug-finding abilities.

  * •

The p1transform transformation randomizes the amount of stack space each
function uses, providing an example of how to modify stack frames for a given
function and providing artificial diversity for the transformed program.
(LLC., 2023b)

After completing the (possibly empty) list of user-specified transformations,
Zipr invokes the back end to reconstitute the IR directly into an executable
program. As we recognized that users may want to control the final layout of
the program, a second SDK is provided, the Zipr Backend SDK. (LLC. et al.,
2023a) The primary purpose of this SDK is to allow plugins to the
reconstitution engine to allow the user to control basic block placement, and
apply user-specified relocations. A Zipr relocation is analogous to a linker
relocation in the compiler world. Built-in relocations for changing, for
example, a data item to point at an instruction, are included. The Selective
CFI transformation (LLC. and Hiser, 2023b) demonstrates how to use the Zipr
Backend SDK (described below) to place executable or not-executable nonces in
the code to implement a highly effective version of CFI directly on binary
programs.

To facilitate quick adoption by new users, the Zephyr Gitlab repository
includes docker images with pre-built Zipr installations, ready to run with a
single command.

#### Zipr Platforms

Zipr’s most robust architecture is the x86/64 Linux platform. This platform
supports all common compilers (gcc/g++, icx, clang/clang++ and the obfuscating
LLVM compiler, OLLVM). (Gough and Stallman, 2004; Corporation, 2023; Lattner,
2008; Junod et al., 2015), all common compiler optimization levels and flags (
-O0, -O1, -O2, -O3, -Os, -Ofast, -fomit-frame-pointer, -static, -fPIC -fPID
-pie, etc.), and most languages that are commonly compiled (C, C++, Fortran,
Ada, Rust, including variants of said languages such as C++11, C++17,
Fortran99, etc.) Notably missing is GoLang support due to a currently
undiagnosed race condition with the GoLang custom exception handling format.
Both static and dynamically linked programs and shared libraries are
supported. The platform’s robustness is demonstrated in our test suite for
Zipr testing, which contains thousands of binaries generated from programs
used in industry. (Hiser, 2023c) X86/32 code, though the first target of Zipr,
has yet to achieve as much attention as we have had fewer partners working
with that code base.

The next most supported platform is 32-bit ARM on Linux. We have automated
regular regressions for coreutils and several other programs used in common
software deployments. (Hiser, 2023b) While there are known bugs, many large
programs work and demonstrate the approach’s feasibility for this platform.

We also have preliminary support for 64-bit ARM code and 32- and 64-bit MIPS
code on the Linux platform. While we do not yet have a robust test suite with
widely-used programs, a variety of coreutils programs have demonstrated
successful transformations. Preliminary support is also available for Windows
32- and 64-bit executables.

##  3\. Zipr Impacts

###  3.1. Commercial Adoption

#### GrammaTech

GrammaTech was one of the first commercial adopters of the Zipr technology
with the authors’ collaboration. GrammaTech concluded that Zipr was a good fit
for their entry into the DARPA Cyber Grand Challenge (CGC). CGC was a fully-
autonomous capture the flag (CTF) competition. Each team built a Cyber
Reasoning System (CRS), and the CRSes competed to autonomously protect their
(vulnerable) services and exploit vulnerabilities in competitors’ services. A
CRS scored points for successfully defending against an exploit, but was
harshly penalized for increases in runtime, on-disk space, or memory usage.
Thus, Zipr was a natural fit due to its low runtime overhead, minimal on-disk
expansion, and modest memory footprint. (Nguyen-Tuong et al., 2018)

The primary defensive mechanism was a control flow integrity (CFI)
implementation (Burow et al., 2017; Abadi et al., 2009), which is now open
source. (LLC. and Hiser, 2023b) The CFI implementation, dubbed Selective CFI,
leveraged the IRDB analyses to detect indirect branches that were safe from
exploitation, and elide expensive instrumentation in those cases to help
minimize scoring penalties for using excess resources. Zipr and the Selective
CFI implementation achieved the #1 score for defeating exploits created by the
other competitors.

Later, Zipr’s exception handling parser (Hiser et al., 2017) was adopted by
GrammaTech. They were able to re-use Zipr’s libEHP (Hiser, 2023a) component in
their Datalog Disassembler (ddisasm)(Flores-Montoya and Schulte, 2020).
GrammaTech’s work also helped improve libEHP by submitting source code updates
with bug fixes, build system improvements, and porting it to operate on MS
Windows platforms.

Finally, ONR sponsored GrammaTech to perform a broad comparison of binary
rewriters, which includes both Zipr and ddisasm. (Schulte et al., 2022) Zipr
performed very well. It was the only binary rewriter to transform every
program in the extensive test suite successfully. Zipr and ddisasm were the
only binary rewriters that were able to transform a majority of programs,
while many other rewriters were deemed “too academic” for wide adoption.

#### Dependable Computing

A joint project between Dependable Computing and the authors leveraged formal
verification techniques to meet security and privacy needs for embedded
systems where development artifacts such as source code were unavailable.
(Davidson et al., 2016) Zipr was critical for meeting embedded systems’ strict
overhead limits, and was able to apply patches to meet security requirements.
Overall, Zipr demonstrated that it could rewrite a binary with patches and
yield less than 5% performance penalty.

#### Apogee Software

Apogee software, as part of the DARPA Cyber Fault-tolerance Attack Recovery
(CFAR) project, worked to realize DARPA’s vision of executing multiple diverse
copies of a program in parallel to detect security violations. (Co et al.,
2016) The key insight is that many attacks must be customized to the low-level
details of a program under attack. For example, an attack might include an
absolute address or relative offset to exploit the program. If the same input
is sent to multiple diverse copies of the program, the programs will behave
differently, and that difference can be detected and used to invoke recovery
mechanisms.

In collaboration with the UVA team, Zipr was used to generate artificial
diversity for key web server programs such as Apache and NginX. (Laurie and
Laurie, 2003; Soni, 2016) Zipr was used to randomize code, data layouts, and
locations for the stack, heap, and global address spaces. The Zipr backend SDK
was used to control code and data layout locations so that detection of
certain classes of attacks was provably certain. For example, Zipr could
create two variants of a program such that their address spaces were disjoint.
Thus, any attack that injected an absolute address would crash at least one
variant.

Other teams in the program did similar transformations using compiler
technology instead of operating directly on the binary. As such, when an Ada
program was introduced, the Zipr transformer was the only tool that
successfully produced diverse variants for parallel execution. In fact, thanks
to the robustness of Zipr binary analysis, little additional effort was
required when the source language changed.

Zipr was also leveraged to enact automatic program repair. (Jones et al.,
2019) The program repair worked by instrumenting a program’s input points to
memorize recent program inputs. If the input subsequently caused a security
violation, the input could be marked as malicious. Future attempts to feed the
input to the program caused the input to be ignored and not processed by the
program. This technique helps defeat denial-of-service (DOS) attacks where the
same input causes the program to repeatedly crash and restart, preventing
legitimate requests from being serviced while the program is busy restarting.

#### Red Hat

Red Hat and the authors collaborated to deploy several Zipr-based tools for
ensemble fuzzing with an unnamed collaborator. (LLC. et al., 2023c; LLC.,
2023a) Unfortunately, as is often the case with commercial partners, we can
provide no additional detail due to confidentiality restrictions.

###  3.2. Student Research

Zipr has provided direct, major contributions to two Ph.D. student
dissertations.

#### Dr. William Hawkins

The first student dissertation we will discuss is William Hawkins’s
dissertation who was advised by Dr. Jack W. Davidson. (Hawkins, 2018) Dr.
Hawkins’s dissertation, granted by the University of Virginia, included
content from several of the original Zipr publications (Hawkins et al., 2017a;
Hiser et al., 2017) but also included several security-enhancing
transformations.

One of Dr. Hawkins’s dissertation’s contributions was a technique called
dynamic canary randomization. (Hawkins et al., 2016) The technique was
designed to walk the stack periodically, find the stack canaries used, and
update the canaries to a new value. The primary goal of the technique was to
thwart attacks where an attacker can leak stack canaries and then later
leverage the leaked value to attack critical control flow data on the stack.

The second of Dr. Hawkins’s contributions was a system called Mixr.(Hawkins et
al., 2017c) Mixr was a tool that leveraged a Zipr plugin to lay out the
program into fixed-size code blocks. Each block had metadata about locations
within the block that would need to be patched if the code were to move. When
the transformed program was executed, user-specified mechanisms triggered
randomization of the code locations. Mixr would randomly swap a predetermined
number of code blocks, leveraging the metadata to update code offsets, and
walk the execution stack to update return addresses as needed. Mixr provided a
_moving target defense_ , a mechanism to invalidate information that an
attacker may attempt to learn over time, such as code locations. (Lei et al.,
2018) Dr. Hawkins’s work evaluated differences in code block sizes and re-
randomization times and policies on the performance and security of an
application.

#### Stefan Nagy

Dr. Stefan Nagy and his advisor Dr. Matthew Hicks at Virginia Tech worked on
techniques to improve closed-source application fuzzing. (Nagy, 2022) Two of
Dr. Nagy’s major contributions were published in top tier conferences and
leveraged the Zipr-based plugin, Zafl.(LLC. et al., 2023c)

The first paper examines the features needed to achieve compiler-quality
fuzzing instrumentation on a binary program. (Nagy et al., 2021a) The paper
leveraged the IRDB SDK’s analyses to add AFL-style instrumentation to a binary
program. The work presents several novel techniques, as well as leveraging
IRDB analyses such as dead register analysis to instrument the program to
produce code that has the same performance (in terms of both execution time
and bug-finding capabilities) as compiler-generated instrumentation. These
techniques significantly improved the state of the art in binary-only fuzzing,
which previously relied on heavy-weight instrumentation systems such as QEMU.

Dr. Nagy’s second paper leverages hardware breakpoints to detect when fuzzing
inputs causes new behavior in a program. (Nagy et al., 2021b) When new
behaviors are observed, heavier instrumentation determines the new behaviors,
and remove the corresponding hardware breakpoints before fuzzing continues.
Because most fuzzing inputs do not generate new behaviors, the technique
rarely uses the more expensive instrumentation, which amortizes their costs.
This technique ultimately leads to near-zero overhead for tracing fuzzing
inputs in a program. The technique leverages the IRDB SDK’s loop analysis
methods to insert accounting instrumentation code in loop headers to get
coverage-preserving, coverage-guided tracing.

#### Student Information Requests

We have received numerous requests from students for bug fixes, capability
suggestions, etc. One example is Franziska Maeckel, a student from the
University of Bamberg in Bamberg, Germany. Mr. Maeckel is working on an open-
source version of an address sanitizer for Zafl as part of his dissertation.
(Maeckel, 2023; Serebryany et al., 2012) He contacted us regarding a suspected
bug in the Zipr layout engine due to the Zipr backend reporting an error in
the IR. We worked with him to track the issue to his transformation violating
one of the IR’s invariant properties.

Various other students have contacted us as well, indicating that people are
in fact interested in, and using Zipr. We have seen social media posts from
students elated that Zafl has produced an “order of magnitude improvement in
fuzzing speed” for their CTF challenge problem. Unfortunately, as the project
is freely available at Zephyr’s Gitlab instance, we cannot track the number of
anonymous downloads, how people may be using the technology, or re-sharing of
the project via other means.

###  3.3. Sponsored Research

#### Kevlar

The Kevlar project was sponsored research by the authors with funding from the
Air Force Research Labs (AFRL). The project’s goal was to transition to
practice a variety of research tools for randomizing program layouts and
adding hardening transformations to applications. Many of these
transformations were previously realized using a dynamic rewriter, and the
project leveraged the Zipr technology to apply the transformations statically,
significantly improving the viability for commercial and government
applications for several reasons:

  * •

The Zipr static rewriting technology is more performant and memory frugal,
making adoption more likely.

  * •

The statically rewritten binary can be easily tested in situ, as minimal
environmental changes are needed to add security. Only changing the target
binaries is needed, as opposed to additional runtime software, etc. Such
changes are harder to deploy in many settings due to regulatory approval,
testing restrictions, or other non-technical issues.

  * •

Humans need to gain trust in the system, and this trust is much more difficult
to gain with dynamic translation systems. Trust in static translation systems
can be audited for correctness by experts and enhanced with semi-automatic
formal verification techniques.

Zipr dramatically improved the practicality of the security techniques.

#### Trusted and Resilient Computation

The Trusted and Resilient Mission Operations (TRMO) project, and its follow-on
project, Trusted and Resilient Systems (TRSYS) sought to provide security
enhancements for autonomous vehicles. (Davidson et al., 2019; Leach et al.,
2022) Myriad Zipr-based plugins provided cyber-attack detection capabilities.
Program instrumentation relayed detected attacks to a supervisor module, which
invoked various forms of program repair. For example, in an autonomous quad-
copter, the attack-response module would put the copter into “hover” mode with
a trusted controller while the more capable, yet untrusted controller program
was being analyzed and repaired. If the repair is successful, the repaired
controller could resume mission operations after a cyber attack.

The Zipr instrumentation system was vital to this work. It provided the low-
overhead detection capabilities needed to enact program repair, and stop the
cyber-attack from being undetected or immediately fatal.

#### Helix++

The Helix++ project aimed at transitioning Zipr-based plugins to practice by
leveraging Docker containers. (Davidson et al., 2023; Huband, 2023) By
protecting applications in commonly used Docker containers and making these
protected containers publicly available, we hoped to lower the cost of having
developers use Zipr-protected applications.

We specifically worked with UVA’s Research Computing staff to identify key
services and build a repository of hardened Docker containers. We are in the
process of working with them to deploy their hardened application to end users
and hope that this model can be used to lower the cost of adopting academic
security concepts.

##  4\. Conclusions

This paper presented the Zipr binary rewriter as an artifact, published in
2016 and 2017 (Davidson et al., 2016; Hawkins et al., 2017a; Hiser et al.,
2017; Hawkins et al., 2017b) and open-sourced in 2019 (LLC. et al., 2023b).
Zipr’s SDKs and plugin architecture target multiple platforms, focusing on
efficient binary rewriting to apply diversity and hardening transformations to
improve program security. Zipr’s robust and efficient nature has been
leveraged by the authors, their collaborators, and third parties to enhance
the body of knowledge through publications and dissertations. Zipr also has a
history of providing open-source tools for diversity, program repair, fuzzing,
research computing, and autonomous vehicle security. We expect that Zipr will
continue to be a valuable tool for enhancing program security, and we will
continue to make it available and support user requests for features, bug
fixes, and clarifications of included features.

###### Acknowledgements.

This material is based upon work supported by the National Science Foundation
under Grant No. 2115130.

## References

  * (1)
  * Abadi et al. (2009) Martín Abadi, Mihai Budiu, Ulfar Erlingsson, and Jay Ligatti. 2009.  Control-flow integrity principles, implementations, and applications.  _ACM Transactions on Information and System Security (TISSEC)_ 13, 1 (2009), 1–40.
  * Burow et al. (2017) Nathan Burow, Scott A Carr, Joseph Nash, Per Larsen, Michael Franz, Stefan Brunthaler, and Mathias Payer. 2017.  Control-flow integrity: Precision, security, and performance.  _ACM Computing Surveys (CSUR)_ 50, 1 (2017), 1–33.
  * Co et al. (2016) Michele Co, Jack W Davidson, Jason D Hiser, John C Knight, Anh Nguyen-Tuong, Westley Weimer, Jonathan Burket, Gregory L Frazier, Tiffany M Frazier, Bruno Dutertre, et al. 2016\.  Double Helix and RAVEN: a system for cyber fault tolerance and recovery. In _Proceedings of the 11th Annual Cyber and Information Security Research Conference_. 1–4.
  * Corporation (2023) Intel Corporation. 2023\.  _Intel C Compiler_.  https://www.intel.com/content/www/us/en/developer/tools/oneapi/dpc-compiler.html
  * Davidson et al. (2019) Jack Davidson, Rector, and Visitors of the University of Virginia. 2019\.  Trusted and Resilient Mission Operation.  (2019).
  * Davidson et al. (2023) Jack W Davidson, Jason D Hiser, and Anh Nguyen-Tuong. 2023\.  Helix++: A platform for efficiently securing software.  _arXiv preprint arXiv:2304.04846_ (2023).
  * Davidson et al. (2016) Jack W Davidson, Jason D Hiser, Anh Nguyen-Tuong, Clark L Coleman, William H Hawkins, John C Knight, Benjamin D Rodes, and Ashlie B Hocking. 2016.  A system for the security protection of embedded binary programs. In _2016 46th Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshop (DSN-W)_. IEEE, 234–237.
  * Eagle (2011) Chris Eagle. 2011\.  _The IDA pro book_.  no starch press.
  * Fioraldi et al. (2020) Andrea Fioraldi, Dominik Maier, Heiko Eißfeldt, and Marc Heuse. 2020.  $\\{$AFL++$\\}$: Combining incremental steps of fuzzing research. In _14th USENIX Workshop on Offensive Technologies (WOOT 20)_.
  * Flores-Montoya and Schulte (2020) Antonio Flores-Montoya and Eric Schulte. 2020.  Datalog disassembly. In _29th USENIX Security Symposium (USENIX Security 20)_. 1075–1092.
  * Gough and Stallman (2004) Brian J Gough and Richard Stallman. 2004.  _An Introduction to GCC._ Network Theory Limited.
  * Hawkins et al. (2017b) William Hawkins, Jason D Hiser, Anh Nguyen-Tuong, Jack W Davidson, et al. 2017b.  Securing binary code.  _IEEE Security & Privacy_ 15, 06 (2017), 77–81.
  * Hawkins et al. (2017c) William Hawkins, Anh Nguyen-Tuong, Jason D Hiser, Michele Co, and Jack W Davidson. 2017c.  Mixr: Flexible runtime rerandomization for binaries. In _Proceedings of the 2017 Workshop on Moving Target Defense_. 27–37.
  * Hawkins (2018) William H. Hawkins. 2018\.  _Static Binary Rewriting to Improve Software Security, Safety and Reliability_.  PhD Thesis. University of Virginia, Charlottesville, VA.
  * Hawkins et al. (2017a) William H Hawkins, Jason D Hiser, Michele Co, Anh Nguyen-Tuong, and Jack W Davidson. 2017a.  Zipr: Efficient static binary rewriting for security. In _2017 47th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN)_. IEEE, 559–566.
  * Hawkins et al. (2016) William H Hawkins, Jason D Hiser, and Jack W Davidson. 2016\.  Dynamic canary randomization for improved software security. In _Proceedings of the 11th Annual Cyber and Information Security Research Conference_. 1–7.
  * Hiser (2023a) Jason Hiser. 2023a.  _libEHP Source Code_.  https://git.zephyr-software.com/opensrc/libehp
  * Hiser (2023b) Jason Hiser. 2023b.  _Zipr Regressing Testing for ARM_.  https://git.zephyr-software.com/opensrc/zipr-testing/zipr-arm-testing
  * Hiser (2023c) Jason Hiser. 2023c.  _Zipr Regressing Testing for X86_.  https://git.zephyr-software.com/opensrc/dod_benchmark_suite
  * Hiser et al. (2017) Jason Hiser, Anh Nguyen-Tuong, William Hawkins, Matthew McGill, Michele Co, and Jack Davidson. 2017.  Zipr++ Exceptional Binary Rewriting. In _Proceedings of the 2017 Workshop on Forming an Ecosystem Around Software Transformation_. 9–15.
  * Hiser et al. (2007) Jason D Hiser, Daniel Williams, Wei Hu, Jack W Davidson, Jason Mars, and Bruce R Childers. 2007\.  Evaluating indirect branch handling mechanisms in software dynamic translation systems. In _International Symposium on Code Generation and Optimization (CGO’07)_. IEEE, 61–73.
  * Huband (2023) Jackie Huband. 2023\.  Enabling Research through Federated Access of Compute Resources for Sensitive Data. (2023).  https://pearc.acm.org/pearc23/ Birds of a Feather (BOF) group discussion.
  * Jones et al. (2019) Jessica Jones, Jason D Hiser, Jack W Davidson, and Stephanie Forrest. 2019.  Defeating Denial-of-Service attacks in a self-managing N-Variant system. In _2019 IEEE/ACM 14th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS)_. IEEE, 126–138.
  * Junod et al. (2015) Pascal Junod, Julien Rinaldini, Johan Wehrli, and Julie Michielin. 2015.  Obfuscator-LLVM–software protection for the masses. In _2015 ieee/acm 1st international workshop on software protection_. IEEE, 3–9.
  * Lattner (2008) Chris Lattner. 2008\.  LLVM and Clang: Next generation compiler technology. In _The BSD conference_ , Vol. 5. 1–20.
  * Laurie and Laurie (2003) Ben Laurie and Peter Laurie. 2003.  _Apache: The definitive guide_.  ” O’Reilly Media, Inc.”.
  * Leach et al. (2022) Kevin Leach, Christopher S Timperley, Kevin Angstadt, Anh Nguyen-Tuong, Jason Hiser, Aaron Paulos, Partha Pal, Patrick Hurley, Carl Thomas, Jack W Davidson, et al. 2022\.  START: A Framework for Trusted and Resilient Autonomous Vehicles (Practical Experience Report). In _2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE)_. IEEE, 73–84.
  * Lei et al. (2018) Cheng Lei, Hong-Qi Zhang, Jing-Lei Tan, Yu-Chen Zhang, and Xiao-Hu Liu. 2018.  Moving target defense techniques: A survey.  _Security and Communication Networks_ 2018 (2018).
  * LLC. (2023a) Zephyr Software LLC. 2023a.  _Turbo Source Code_.  https://git.zephyr-software.com/opensrc/turbo
  * LLC. (2023b) Zephyr Software LLC. 2023b.  _Zafl: Zipr-based AFL Instrumentation_.  https://git.zephyr-software.com/opensrc/p1transform
  * LLC. et al. (2023a) Zephyr Software LLC., Will Hawkins, and Jason Hiser. 2023a.  _The Zipr Code Placement SDK Source Code_.  https://git.zephyr-software.com/opensrc/zipr-sdk
  * LLC. et al. (2023b) Zephyr Software LLC., Will Hawkins, and Jason Hiser. 2023b.  _Zipr Source Code_.  https://git.zephyr-software.com/opensrc/zipr
  * LLC. and Hiser (2023a) Zephyr Software LLC. and Jason Hiser. 2023a.  _The IRDB SDK Source Code_.  https://git.zephyr-software.com/opensrc/irdb-sdk
  * LLC. and Hiser (2023b) Zephyr Software LLC. and Jason Hiser. 2023b.  _Selective CFI_.  https://git.zephyr-software.com/opensrc/cfi
  * LLC. and Hiser (2023c) Zephyr Software LLC. and Jason Hiser. 2023c.  _The Zipr Code Placement SDK Source Code_.  https://git.zephyr-software.com/opensrc/irdb-cookbook-examples
  * LLC. et al. (2023c) Zephyr Software LLC., Jason Hiser, and Anh Nguyen-Tuong. 2023c.  _Zafl: Zipr-based AFL Instrumentation_.  https://git.zephyr-software.com/opensrc/zafl
  * Maeckel (2023) Franziska Maeckel. 2023\.  _binary-msan Source Code_.  https://github.com/fransie/binary-msan/tree/instrument-all-functions
  * Nagy (2022) Stefan Nagy. 2022\.  _Practical Feedback and Instrumentation Enhancements for Performant Security Testing of Closed-source Executables_.  PhD Thesis. Virginia Tech, Blacksburg, VA.
  * Nagy et al. (2021a) Stefan Nagy, Anh Nguyen-Tuong, Jason D Hiser, Jack W Davidson, and Matthew Hicks. 2021a.  Breaking through binaries: Compiler-quality instrumentation for better binary-only fuzzing. In _30th USENIX Security Symposium (USENIX Security 21)_. 1683–1700.
  * Nagy et al. (2021b) Stefan Nagy, Anh Nguyen-Tuong, Jason D Hiser, Jack W Davidson, and Matthew Hicks. 2021b.  Same coverage, less bloat: Accelerating binary-only fuzzing with coverage-preserving coverage-guided tracing. In _Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security_. 351–365.
  * Nguyen-Tuong et al. (2018) Anh Nguyen-Tuong, David Melski, Jack W Davidson, Michele Co, William Hawkins, Jason D Hiser, Derek Morris, Ducson Nguyen, and Eric Rizzi. 2018.  Xandra: An autonomous cyber battle system for the Cyber Grand Challenge.  _IEEE Security & Privacy_ 16, 2 (2018), 42–51.
  * Schulte et al. (2022) Eric Schulte, Michael D Brown, and Vlad Folts. 2022.  A Broad Comparative Evaluation of x86-64 Binary Rewriters. In _Proceedings of the 15th Workshop on Cyber Security Experimentation and Test_. 129–144.
  * Scott and Davidson (2001) Kevin Scott and Jack Davidson. 2001.  Strata: A software dynamic translation infrastructure. In _IEEE Workshop on Binary Translation_.
  * Scott et al. (2003) Kevin Scott, Naveen Kumar, Siva Velusamy, Bruce Childers, Jack W Davidson, and Mary Lou Soffa. 2003\.  Retargetable and reconfigurable software dynamic translation. In _International Symposium on Code Generation and Optimization, 2003. CGO 2003._ IEEE, 36–47.
  * Serebryany et al. (2012) Konstantin Serebryany, Derek Bruening, Alexander Potapenko, and Dmitriy Vyukov. 2012.  $\\{$AddressSanitizer$\\}$: A fast address sanity checker. In _2012 USENIX annual technical conference (USENIX ATC 12)_. 309–318.
  * Soni (2016) Rahul Soni. 2016\.  _Nginx_.  Springer.
  * Wong (2022) Cerdic Wei Kit Wong. 2022\.  American fuzzy lop (AFL) fuzzer.  (2022).


# A LLM-Based Ranking Method for the Evaluation
of Automatic Counter-Narrative Generation

Irune Zubiaga  Aitor Soroa  Rodrigo Agerri
HiTZ Center - Ixa, University of the Basque Country UPV/EHU
{irune.zubiaga, a.soroa<EMAIL_ADDRESS>
###### Abstract

The proliferation of misinformation and harmful narratives in online discourse
has underscored the critical need for effective Counter Narrative (CN)
generation techniques. However, existing automatic evaluation methods often
lack interpretability and fail to capture the nuanced relationship between
generated CNs and human perception. Aiming to achieve a higher correlation
with human judgments, this paper proposes a novel approach to asses generated
CNs that consists on the use of a Large Language Model (LLM) as a evaluator.
By comparing generated CNs pairwise in a tournament-style format, we establish
a model ranking pipeline that achieves a correlation of $0.88$ with human
preference. As an additional contribution, we leverage LLMs as zero-shot (ZS)
CN generators and conduct a comparative analysis of chat, instruct, and base
models, exploring their respective strengths and limitations. Through
meticulous evaluation, including fine-tuning experiments, we elucidate the
differences in performance and responsiveness to domain-specific data. We
conclude that chat-aligned models in ZS are the best option for carrying out
the task, provided they do not refuse to generate an answer due to security
concerns.

Warning: Please be advised that this research paper contains instances of hate
speech that may be distressing or offensive to readers. These expressions are
included for analysis and critique purposes only, and they do not reflect the
beliefs or endorsements of the authors or the institution.

A LLM-Based Ranking Method for the Evaluation
of Automatic Counter-Narrative Generation

Irune Zubiaga and Aitor Soroa and Rodrigo Agerri HiTZ Center - Ixa, University
of the Basque Country UPV/EHU {irune.zubiaga, a.soroa<EMAIL_ADDRESS>

##  1 Introduction

The proliferation of misinformation and the dissemination of harmful
narratives has underscored the urgent need for effective strategies to combat
Hate Speech (HS). This necessity has drawn significant attention to the field
of automatic CN generation, where considerable research has been conducted on
the use of LLMs to fulfill this task with impressive results Chung et al.
(2021); Tekiroğlu et al. (2022). However, difficulties in automatically
assessing the quality of the generated CNs remain. As is common in text
generation tasks, while manual evaluation is expensive, time-consuming, and
subjective, existing automatic methods often fail to provide comprehensive
insights or capture the nuanced relationship between generated text and human
perception, overlooking crucial aspects of effectiveness and relevance (Ni’mah
et al., 2023). Finally, the evaluation problem is exacerbated by the lack of a
’universal truth’ and the significant variations among possible references
(see example in Table 1).

HS
---
Muslims do not have anything useful that can enrich our culture.
Candidate CNs
1. How about the money they contribute to our economy, their expertise and knowledge, their culture, tasty food. Should I go on? 2. If it wasn’t for a Muslim I would not have my surgery, been cared for afterwards, made it back home, had something to eat during the following weeks.
Table 1: Sample set of proposed CNs for a single HS instance from the CONAN
corpus. Although only two references are shown in the table, the corpus
includes a total of 36 candidate CNs as the gold standard for the presented
instance of HS.

In this paper we address these shortcomings by presenting a novel approach to
automatically evaluate CNs, aiming for a higher correlation to human
preference than the traditional metrics commonly used to evaluate natural
language generation (NLG) tasks. We propose to evaluate generated CNs pairwise
in a tournament-style format and determine the outcome of said tournaments
without the need for human intervention through the use of a Judge Model
explicitly trained to assess the quality of generated text. The outcomes of
the tournaments will be used to rank models according to their performance.
This approach allows us to decompose a subjective task like CN evaluation into
a series of simpler binary classification problems.

As an additional contribution, we focus on evaluating the inherent ability of
LLMs as ZS CN generators. Leveraging state-of-the-art open-source LLMs, we
seek to explore their potential in generating CNs that effectively challenge
and mitigate the influence of misinformation and harmful narratives. We
examine three variants within the same model family: base, instruction-tuned,
and chat-aligned. This enables us to inspect their unique strengths and
limitations to determine the optimal choice for the task. Finally, we fine-
tune the models on HS-CN pair data to compare their performance against ZS
performance, assessing whether fine-tuning offers any significant improvement
in our scenario. We conclude that chat-aligned models in a ZS setting are the
best option for carrying out the task, provided they do not refuse to generate
an answer due to security concerns111Chat-aligned models, designed to adhere
to safety and ethical guidelines, may sometimes decline to respond to certain
prompts. This refusal is typically in place to prevent the generation of
harmful, inappropriate, or sensitive content.. Code will be made publicly
available upon publication.

##  2 Related Work

In recent years, automatic CN generation has attracted growing research
interest, with numerous methods leveraging NLG technologies for generating
CNs. Nearly all recent systems depend on LLMs to automatically generate CNs
(Ashida and Komachi, 2022; Tekiroğlu et al., 2022; Saha et al., 2024) , driven
by their impressive performance in generation tasks, which often necessitates
minimal or no training data (Zhao et al., 2023a; OpenAI et al., 2024; Zhao et
al., 2023b).

Several datasets have been introduced to aid in the advancement of CN
generation. The first large-scale, multilingual, expert-based dataset, Counter
Narratives through Nichesourcing (CONAN) (Chung et al., 2019), consists of HS-
CN pairs in English, French, and Italian, focusing exclusively on
Islamophobia. This corpus served as the foundation for the development of
MultiTarget CONAN (MT-CONAN) (Fanton et al., 2021a), which includes 8 hate-
speech targets such as women and individuals with disabilities. Additionally,
the DIALOCONAN dataset (Bonaldi et al., 2022), which contains fictitious
dialogues between a hater and a Non-Governmental Organization (NGO) operator,
and the Knowledge-grounded Hate Countering dataset (Chung et al., 2021),
featuring HS-CN pairs with the background knowledge used for constructing the
CNs have been introduced. Some work in adapting these corpora to other
languages has also been done, such as CONAN-EUS Bengoetxea et al. (2024), a
Basque and Spanish translation of the original CONAN dataset, and CONAN-MT-SP
(Vallecillo Rodríguez et al., 2024), a Spanish version of MT-CONAN.

Assessing the impact and effectiveness of the generated CNs remains a crucial
aspect of this research domain. Evaluating CN is particularly challenging
because there are many acceptable answers to a given HS, and is often very
difficult to assess what constitutes a good answer. Evaluation is usually done
either through automatic or manual methods. Automatic methods involve the use
of metrics such as BLEU Papineni et al. (2002), ROUGE Lin (2004), and
BERTScore Zhang et al. (2020), which are standard evaluation metrics in tasks
such as Machine Translation or Text Summarization. However, these metrics are
known to be often weakly correlated with human judgment Sai et al. (2022),
particularly on tasks that require creativity Ni’mah et al. (2023). Other
automatic metrics that have been used for NLG evaluation including Repetition
Rate (RR) (Bertoldi et al., 2013), which measures diversity in the generated
answers, or Novelty Wang and Wan (2018), which encourages the model to
generate answers that differ from the text in the training data. We must note
that none of the aforementioned metrics take the input HS into account,
missing a crucial aspect in the evaluation of CNs. As far as we know, no
previous work has analyzed the correlation of these metrics with human
judgment for CN evaluation.

Due to the limitations of automatic metrics, final assessments frequently rely
on manual evaluations. Nonetheless, manual evaluation is often a costly
process, and finding evaluators with adequate task comprehension can be
challenging. Moreover, the subjective nature of the task adds another layer of
complexity to the evaluation process. To mitigate this subjectivity, various
efforts have been made to identify key aspects that assess the quality of CNs.
Unfortunately, consensus on these key aspects is still lacking, as different
authors consider factors such as relatedness, specificity, richness,
coherence, grammaticality, suitableness, informativeness, diversity,
relevance, language quality, offensiveness or stance Chung et al. (2021);
Ashida and Komachi (2022); Bengoetxea et al. (2024).

Recently, to address the limitations of traditional metrics and manual
evaluation, LLMs are being employed to directly assess the quality of
generated text. Leveraging LLMs to measure NLG has been shown to exhibit
stronger correlation with human assessment compared to conventional reference-
based evaluation Nimah et al. (2023); Chiang and Lee (2023); Wang et al.
(2023); Liu et al. (2023), and it remains an efficient and automated approach.
Some works include using commercial models such as GPT-4 Wang et al. (2023);
Liu et al. (2023), while others focus on training specialized models for
evaluation tasks, resulting in tools like PandaLM (Wang et al., 2024), JudgeLM
Zhu et al. (2023), and UniEval (Zhong et al., 2022). These LLMs can be either
used to ascertain quantitative aspects to measure the quality of the generated
text (Zhong et al., 2022; Ke et al., 2022), or to show preferences between two
texts generated by different systems.

There are few works on automatic CN evaluation. Contemporary to our work,
Jones et al. (2024) use LLMs to evaluate CNs based on five different aspects
considered relevant for their effectiveness in combating HS. In a strategic
shift, we propose to use LLMs to compare outputs from different models between
them and ultimately obtain a ranking of the best models for CN generation that
correlates with human judgments.

##  3 Methodology

This section provides an overview of the key components of the research
methodology. Section 3.1 discusses the specific models that were used for CN
generation. Section 3.2 presents the corpus used in the study. Finally,
Section 3.3 outlines the metrics that were employed to carry out the
evaluation.

###  3.1 Models

In this study, we used auto-regressive models for CN generation. Specifically,
we work with three variants of the Mistral model family (Jiang et al., 2023)
as well as the Llama 2 Chat model (Touvron et al., 2023). The Mistral variants
include the Mistral base model, the Mistral-Instruct model, and Zephyr
(Tunstall et al., 2023), which is a chat-aligned model based on Mistral. The
selection of these three variants enables the comparison of chat-aligned,
instruction-tuned, and base model performance and behavior. Llama 2 was
selected as to compare the results with Mistral models due to its relevance in
the field and potential for providing complementary insights into CN
generation. All models are 7B parameter models, consistent with the available
Mistral model size, to ensure comparability of results. Additionally, the
weights of all the models used in this study are publicly available. The
specific versions of the employed models are listed in Table 3.

###  3.2 Corpus

Dataset | HS-CN Pairs | Unique HS | Unique CN | Avg. CNs / HS | Avg. Words / CN
---|---|---|---|---|---
CONAN | 6648 | 523 | 4040 | 12.71 | 19.48
MT-CONAN | 5003 | 3718 | 4997 | 1.35 | 24.77
Table 2: Statistics of the CONAN and MT-CONAN corpora, showing the number of
HS-CN pairs, the number of unique HS and CN instances, the average number of
CN per HS, and the average number of words per CN.

In order to test the generalizability of our method, we conduct the analysis
on two distinct datasets: Counter Narratives through Nichesourcing (CONAN)
(Chung et al., 2019) and Multi-Target CONAN (MT-CONAN) (Fanton et al., 2021b).
Corpus statistics are presented in Table 2.

#### CONAN

Comprises HS-CN pairs addressing Islamophobia in three languages: English,
Italian, and French. These pairs were collected through nichesourcing
involving 3 different NGOs from the United Kingdom, France, and Italy. As a
result, the CNs are expert-based and crafted by operators specifically trained
to combat online HS. After the data collection phase, three non-expert
annotators were hired to augment the dataset. They paraphrased original hate
content to increase pairs per language and translated content from French and
Italian to English for language parallelism. NGO trainers validated the newly
generated data for each language to ensure quality. In this work, we only
focus on the English partition. During fine-tuning, we used 4833 pairs for
training, 537 for validation, and 1278 for testing. The specific train-val-
test splits are available at https://huggingface.co/datasets/HiTZ/CONAN-EUS.

#### MT-CONAN

Consists of HS-CN pairs in English, collected through a Human-in-the-Loop
approach. This method involves iteratively refining a generative language
model by utilizing its own data from previous loops to generate new training
samples, which are then reviewed and/or post-edited by experts. The HS targets
eight distinct demographics: individuals with disabilities, Jewish people, the
LGBT+ community, migrants, Muslims, people of color, women, and other
marginalized groups. During fine-tuning, we used 3003 pairs for training, 1000
for validation, and 1000 for testing.

###  3.3 Evaluation Metrics

For evaluation we used both reference-based and reference-free metrics.
Additionally, we incorporated the use of a Judge Model as part of our proposed
evaluation methodology.

#### Reference-Based Metrics

Based on previous work on CN generation, we opted to use BLEU, ROUGE-L and
BERTScore. BLEU is a precision-based metric that measures the similarity
between a candidate text and one or more reference texts. It computes the
geometric mean of modified n-gram precision and applies a brevity penalty to
discourage short translations. BLEU is widely used in machine translation
tasks. ROUGE-L, on the other hand, focuses on the recall of content units. It
calculates the longest common subsequence between the candidate sequence and
the reference sequence, normalizing by the length of the reference sequence.
ROUGE-L is commonly employed in text summarization tasks. BERTScore leverages
contextual embeddings from pre-trained BERT models to compute the similarity
between candidate and reference sentences. It computes the score based on the
cosine similarity between BERT embeddings, providing a measure of semantic
similarity. BERTScore has demonstrated effectiveness across various NLG tasks,
including machine translation and text summarization.

#### Reference-Free Metrics

For reference-free metrics, we opted to use Repetition Rate (RR) (Bertoldi et
al., 2013), which is computed by calculating the non-singleton n-grams that
are repeated in the generated text (Bertoldi et al., 2013) and Novelty Wang
and Wan (2018) that is computed by calculating the non-singleton n-grams from
the generated text that appear in the train data. While RR aims to capture the
diversity in the generated text, Novelty measures how different the generated
text is from the training data. It should be noted that Novelty is less
valuable when evaluating models that were used in a ZS setting, as there is no
training involved.

#### LLM-Based Evaluation

Finally, we consider the use of JudgeLM as an evaluator. JudgeLM is a scalable
Judge Model based on Vicuna that was designed to evaluate LLMs in open-ended
scenarios. It was trained using a large-scale dataset consisting on LLM-
generated answers for diverse NLG tasks and detailed judgments from GPT-4.
Remarkably, it achieves an agreement rate exceeding 90% in some tasks,
surpassing even human-to-human agreement levels Zhu et al. (2023). While
JudgeLM supports different evaluation methods, such as comparing single
answers against a reference or comparing multiple answers simultaneously, we
decided to use it to compare generated CNs pairwise, as described in Section
4.1. This eliminates the problem of needing a reliable reference and instead
focuses on determining which of the available options is the best, simplifying
the task. Comparing the CNs against each other also avoids the ambiguity of
the open-ended scenario we would face if we decided to evaluate them
individually. We must note that, as seen in Appendix C.5, where the prompting
format of the model is shown, JudgeLM considers the instance of HS when
comparing the effectiveness of CNs, evaluating their effectiveness in relation
to a specific instance of HS, not just as standalone sentences.

JudgeLM operates in two modes: fast evaluation activated or deactivated. In
fast evaluation mode, the model outputs two scores, one for each CN, providing
an overall assessment of their value. When deactivated, the model supplements
these scores with arguments explaining the rationale behind them. Both during
the development stage and while conducting the result analysis, we deactivated
fast mode to ensure a comprehensive evaluation. However, for creating the
ranks, we opted to deactivate the argumentation feature and solely output the
scores. This decision was made because generating arguments significantly
increases inference time and argumentation was deemed unnecessary for our
specific task.

##  4 Evaluation Framework

In this section we present a cost-effective pairwise rank-based evaluation
paradigm designed to assess the performance of CN generation systems in
alignment with human preference. The method is detailed in Section 4.1. In
addition, in Section 4.2, we describe the additional manual evaluation
conducted to provide a detailed assessment according to various relevant
aspects contributing to the effectiveness of a CN, as presented in Bengoetxea
et al. (2024).

###  4.1 Pairwise Rank-Based Evaluation

We propose an "A vs B" comparative setup to rank models with respect to their
CN generation skills. Suppose we have $n$ models to rank and we want to
evaluate their performance in a test set consisting of $h$ HS instances.
First, each one of the $n$ models will generate a CN for each instance of HS
in the test set. After that, the generated CNs will be pitted against each
other in "A vs B" tournaments. In the end, we are left with ${n\choose 2}\cdot
h=\frac{n!}{2!(n-2)!}\cdot h$ tournaments. We will use either automatic or
manual evaluation to decide the outcome of said tournaments. Based on the
results, each of the $n$ models will receive a performance score using the
following point scheme: the winning model receives $1$ point, the losing model
receives $0$ points, and in the case of a tie, both models receive $0.5$
points. Finally, the models will be ranked based on the obtained score.

#### Automatic Evaluation

For automatic evaluation we will prompt JudgeLM to output 2 scores: one for
each proposed CN. The winner will be determined by comparing these scores: the
CN with the highest score is the winner. If both scores are the same, it
results in a tie.

In our experimentation, all the models in Table 3 were evaluated in both CONAN
and MT-CONAN. Combining their test sets results in 2278 HS instances. We
evaluated both the fine-tuned and ZS versions of each model, along with the
gold standard. This resulted in a total of ${9\choose 2}\cdot 2278=82008$
tournaments.

#### Manual Evaluation

For manual evaluation, we had 3 annotators decide which of the 2 proposed CNs
they believe would be more effective in combating the presented instance of
HS. The specific guidelines provided to the annotators are detailed in
Appendix A.1.

In our experimentation, given the significant cost associated with manual
evaluation, 10 HS instances from each of the test sets of CONAN and MT-CONAN
were randomly selected. This resulted in ${9\choose 2}\cdot 20=720$
tournaments. From the mix of both corpora, 288 tournaments (144 from each
corpus) were evaluated by all 3 evaluators to calculate the inter-annotator
agreement (IAA) using Cohen’s Kappa. For instances from CONAN, the mean IAA
was $0.42$, while for the instances of CONAN-MT, it was $0.58$. The individual
coefficients are presented in Appendix B. The final outcome of these
tournaments was decided using majority voting to reduce subjectivity. The
remaining 432 tournaments were each annotated by a single annotator, with each
annotator evaluating 144 tournaments.

###  4.2 Feature Evaluation

For a comprehensive evaluation of CN quality, we conducted a manual assessment
based on five criteria: Relatedness, Specificity, Richness, Coherence,
Grammaticality and overall score Bengoetxea et al. (2024). Detailed evaluation
guidelines can be found in Appendix A.2. The evaluation was conducted by 2
annotators on 90 HS-CN pairs, which included 10 HS instances from the CONAN
corpus and the corresponding CNs generated by each of the 9 models. Each
evaluated feature used a five-point scale for the answers, with 1 as the
lowest score and 5 as the highest. The mean IAA across all the evaluated
features is $0.69$. The feature-wise IAA is presented in Appendix B.

Model | Version | Type | lrCONAN | lrMT-CONAN
---|---|---|---|---
mistral | v0.1 | Base | 1e-5 | 1e-4
mistral-instruct | v0.2 | Instruct | 6e-6 | 3e-5
zephyr | Beta | Chat | 6e-6 | 1e-4
llama-chat | llama 2 | Chat | 2e-5 | 1e-3
Table 3:  Information regarding the models, along with the learning rates that
resulted in the lowest perplexity in the validation set for each one. It’s
worth noting that the optimal learning rates varied across different corpora:
lrCONAN are the optimal learning rates when fine-tuning on CONAN and lrMT-
CONAN when fine-tuning on MT-CONAN.

##  5 Experimental Setup

This section outlines the specific setup employed for the generation of CNs.
We consider two generation scenarios: ZS (see Section 5.1) and inference with
fine-tuned models (see Section 5.2).

###  5.1 Zero-Shot Generation

In the ZS approach we prompt all models with the sentence _"Provide a brief
counter-narrative in response to the user’s hate speech. Ensure the output
does not contain line breaks"_. This prompt was selected based on preliminary
experimentation. For each model, the prompt was formatted differently, as each
one has its specific prompting format. The various formats are detailed in
Appendix C.

###  5.2 Fine-Tuning Details

Instead of fine-tuning the whole model, Quantized Low-Rank Adaptation (QLoRA)
Dettmers et al. (2023) was used. This approach facilitated a faster and more
accessible training process, as it significantly reduces hardware
requirements. The model was loaded in 4 bit with NF4 quantization data type
and bf16 computational data type. The LoRA update matrices were applied to the
attention blocks and bias parameters were not trained. The LoRA rank and
scaling factor were set to 16 and the dropout to $0.05$. These values were
chosen based on experimentation, guided by those reported in the literature
(Dettmers et al., 2023; Hu et al., 2021), with minimal observed impact on
results. Following usual practice, we used Adam optimizer with a inverse
square root scheduler, half precision, and a batch size of 32. A set of
learning rates ranging from $1e^{-6}$ to $1e^{-3}$ were tested, and the one
yielding the lowest perplexity in the validation set was selected for each
model. The selected learning rates are listed in Table 3. The models were set
to train for a maximum of 10 epochs, with early stopping and a patience of 3
epochs. The checkpoint with the lowest valiadation loss was selected in each
case. Additionally, at inference time, generation was terminated upon
encountering the newline token (\n) to ensure the production of shorter
sentences, addressing the issue of role-playing commonly observed in many
models and particularly prominent in base models due to their challenges in
effectively interpreting prompts.

Rank | Human | Score | JudgeLM | Score
---|---|---|---|---
1 | zephyrzs | 18.02 | zephyrzs | 20.20
2 | gold standard | 17.60 | mistral-instructzs | 16.09
3 | mistral-instructzs | 14.80 | gold standard | 8.98
4 | zephyr | 11.59 | zephyr | 13.30
5 | mistralzs | 10.75 | llama-chatzs | 11.07
6 | mistral | 9.08 | mistralzs | 9.05
7 | mistral-instruct | 7.54 | mistral | 8.70
8 | llama-chatzs | 7.26 | mistral-instruct | 8.50
9 | llama-chat | 3.35 | llama-chat | 4.11
Table 4: Comparison of human and JudgeLM rankings, including the final scores
obtained from the pairwise tournaments. ${zs}$ means that the model was used
in a zero-shot setting.

##  6 Results

First, in Section 6.1, we discuss the correlation between the metrics in
Section 3.3 and human preference, highlighting the implications of the
findings. Then, in Section 6.2, we present the model performance ranking for
the CN generation task derived from the Pairwise Rank-Based evaluation method
(see Section 4).

###  6.1 Correlation of Automatic Metrics with Human Ratings

Figure 1 illustrates the Spearman’s rank correlation coefficients ($\rho$)
among all metrics, including human evaluation. The rankings for Human and J-LM
(JudgeLM) are computed using the pairwise comparison setting described in
Section 4, whereas the rest of the rankings (BLEU, ROUGE-L, etc) are based on
their respective metric scores. All rankings are established across 720
tournaments, as described in Section 4.1.

Figure 1: Matrix with the Spearman’s rank correlation coefficients among
metrics. The last row of the matrix represents the correlation of all the
evaluation metrics to human preference. J-LM is short for JudgeLM.

The figure shows a strong correlation between all variants of JudgeLM and
human preference, as depicted in the last row/column of the matrix, with both
the 7B and 33B parameter JudgeLM achieving a $\rho$ of $0.88$. This high
correlation is supported by a statistically significant Pearson correlation
coefficient of $0.73$ between JudgeLM (33B version) and human preference
(p-value of $0.03$)222We calculate Pearson correlation using the performance
scores obtained by each model in the Pairwise Rank-Based evaluation..

On the contrary, traditional metrics correlate poorly with human preference,
with the highest $\rho$ being the $0.50$ obtained by BERTScore. These results
confirm that commonly used automatic metrics lack alignment with human
preference when evaluating the quality of CNs. Not unsurprisingly, the
correlation between traditional metrics and JudgeLM is also low. They also
correlate poorly among themselves with the exception of ROUGE-L and BERTScore,
which attain a $\rho$ of $0.88$. Despite both being based on n-gram overlap,
ROUGE-L and BLEU only achieve a $\rho$ value of $0.3$.

The aforementioned observations are further reinforced in Appendix D, where
the correlation matrices on the CONAN corpus and the MT-CONAN corpus are
presented separately. In said appendix, we once again see a strong correlation
between JudgeLM variants and human preference, whereas the correlation with
traditional metrics is weak and inconsistent, showing no predictable pattern.

As the concluding point of the correlation analysis, Table 4 presents a
comparison of the final rankings obtained through the manual and automatic
pairwise comparisons as described in Section 4. Both the automatic and the
manual method assign similar scores to almost all systems, with the exceptions
of the gold standard, which obtains a considerably higher score when evaluated
by humans than by JudgeLM, and llama-chatzs, where JudgeLM assigns it a higher
rank than humans. In any case, their final position in the rank only varies
slightly among methods. By analyzing examples of discrepancies between human
and JudgeLM judgements, we observe that the disagreement in the case of the
gold standard might stem from the fact that the JudgeLM model prefers longer,
more detailed CNs, while the annotators preferred shorter, more direct ones.
It might also be related to the fact that the model cannot discern false
information from true information, whereas the human evaluator can penalize
non-factual content resulting in the simpler but veracious CN winning.
Instead, in the case of llama-chatzs, the disagreement might be because
JudgeLM favors answers that start by stating that they can not endorse in hate
speech ("I apologize, but I cannot fulfill your request. I’m just an AI and
it’s not within my programming or ethical guidelines to provide counter-
narratives that promote hate speech… Is there anything else I can help you
with?"). This preference may stem from its training on evaluations made by
ChatGPT, which often responds in a similar manner when asked to provide CNs to
HS.

System | Relatedness | Specificity | Richness | Coherence | Grammaticality | Overall
---|---|---|---|---|---|---
zephyrzs | 4.95 | 4.25 | 4.00 | 5.00 | 5.00 | 4.25
gold standard | 4.10 | 3.75 | 3.25 | 4.80 | 4.30 | 3.50
mistral-instructzs | 4.20 | 3.15 | 3.70 | 4.70 | 5.00 | 3.50
llama-chatzs | 2.90 | 2.55 | 4.30 | 4.90 | 5.00 | 3.05
mistral-instruct | 3.75 | 3.55 | 3.30 | 3.10 | 4.30 | 2.70
mistral | 3.65 | 3.55 | 3.05 | 3.30 | 4.35 | 2.60
zephyr | 4.40 | 4.75 | 3.60 | 3.20 | 4.35 | 2.30
llama-chat | 3.40 | 3.10 | 2.95 | 3.30 | 4.10 | 2.20
mistralzs | 3.10 | 3.30 | 2.40 | 3.55 | 4.60 | 1.90
Table 5: Evaluation of the different aspects that contribute to the
effectiveness of a CN. The values in the table represent the average of the
scores assigned by each annotator.

###  6.2 Ranking by Pairwise Comparison

Figure 2: Ranking through pairwise comparison based on evaluations of all the
JudgeLM size variations across the entire test set.

Figure 2 depicts the ranking of CN generation systems based on the tournament
outcomes according to different sizes of JudgeLM, evaluated across the $82008$
tournaments that comprise the entire test set. Overall, in the ZS scenario
chat-aligned models exhibit superior performance, followed by the instruction-
tuned model, while the base model demonstrates the lowest performance. This
outcome is expected, as base models lack training to understand instructions
and have no prior experience in the task, whereas chat models, in addition to
being capable of understanding instructions, are often trained to fight
toxicity through Safety Fine-Tuning (Touvron et al., 2023; OpenAI et al.,
2024). When fine-tuning the models, we observe a decline in performance across
all models, except for the base model, which exhibits a considerable
improvement. The decline in performance is more pronounced in the chat-aligned
models than in the instruction-tuned model.

When examining the rankings according to the different sizes of JudgeLM, we
observe that as the model size increases, llama-chatzs is positioned lower,
thereby narrowing its performance gap with Zephyrzs.

##  7 Analysis

To confirm which of the models from Section 3.1 is the best for the task, we
conducted a final feature-wise evaluation as explained in Section 4.2. The
results are presented in Table 5. As seen there, the best-performing model is
undoubtedly Zephyr, which considerably surpasses the gold standard.

Analyzing Table 4 in more detail, which presents the final rankings obtained
from manual and automatic pairwise comparisons, we note that while manual
evaluators were instructed to select a winning CN unless both were deemed
ineffective, JudgeLM assigns ties when both responses are of high quality.
This approach may result in a lower correlation between JudgeLM ratings and
human preference.

Lastly, regarding fine-tuning, we observed that including factual CNs in the
fine-tuning process for our CN generation scenario might not be advisable, as
models may mimic the structure but lack factual accuracy.

##  8 Conclusions

CN generation needs a different evaluation framework and metrics than those
used in previous work Chung et al. (2021); Tekiroğlu et al. (2022); Bengoetxea
et al. (2024). This is due to the unique objectives, complexities, and impact
of CNs, which require specialized criteria to assess their effectiveness and
quality accurately. Thus, developing and implementing tailored evaluation
metrics is crucial to advance the field and ensure the successful creation of
impactful CNs.

Consistent with previous research observations, traditional metrics fall short
in evaluating generation tasks that require creativity, including CN
generation to combat HS. In this paper we present a LLM-based ranking method
to provide an alternative automatic evaluation technique that offers promise,
as it has shown increased correlation with human evaluation.

## Limitations

Our work still has some open research questions which can be summarized in the
following limitations. First, we have not addressed truthfulness. Thus,
JudgeLM rewards CNs that provide factual arguments without considering whether
they are truthful. Second, additional tests on larger corpora could be
performed to determine whether the lack of improvement from fine-tuning in
chat and instruct models is due to limitations in the corpus itself.

The corpus used in our experiments was small and, as indicated in Table 2,
exhibited significant repetition of certain HS instances giving them a
different CN each time. We hypothesize that this data structure may
potentially have adverse effects in model performance. Thus, we performed a
preliminary fine-tuning experiment that involved randomly removing duplicate
entries from the corpus, resulting in a smaller but cleaner dataset. Despite
the dataset being smaller, the performance did not degrade. This initial
investigation suggests that reducing duplications could lead to more
consistent learning outcomes.

In future work, we aim to extend this analysis to other languages such as
Spanish, along with Basque, which is considered a low-resource language. We
also intend to explore Retrieval Augmented Generation (RAG) to address the
truthfulness issue, as we anticipate that this approach could substantially
enhance the correlation between human evaluations and those conducted by Judge
Models.

## Acknowledgements

This work has been partially supported by the Basque Government (Research
group funding IT-1805-22). We are also thankful to the following
MCIN/AEI/10.13039/501100011033 projects: (i) DeepKnowledge
(PID2021-127777OB-C21) and by FEDER, EU; (ii) Disargue (TED2021-130810B-C21)
and European Union NextGenerationEU/PRTR; (iii) DeepMinor (CNS2023-144375) and
European Union NextGenerationEU/PRTR. (iv) DeepR3 (TED2021-130295B-C31) and
European Union NextGenerationEU/PRTR.

## References

  * Ashida and Komachi (2022) Mana Ashida and Mamoru Komachi. 2022.  Towards automatic generation of messages countering online hate speech and microaggressions.  In _Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH)_ , pages 11–23, Seattle, Washington (Hybrid). Association for Computational Linguistics.
  * Bengoetxea et al. (2024) Jaione Bengoetxea, Yi-Ling Chung, Marco Guerini, and Rodrigo Agerri. 2024.  Basque and spanish counter narrative generation: Data creation and evaluation.
  * Bertoldi et al. (2013) Nicola Bertoldi, Mauro Cettolo, and Marcello Federico. 2013.  Cache-based online adaptation for machine translation enhanced computer assisted translation.  In _Proceedings of Machine Translation Summit XIV: Papers_ , Nice, France.
  * Bonaldi et al. (2022) Helena Bonaldi, Sara Dellantonio, Serra Sinem Tekiroglu, and Marco Guerini. 2022.  Human-machine collaboration approaches to build a dialogue dataset for hate speech countering.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8031–8049. Association for Computational Linguistics.
  * Chiang and Lee (2023) Cheng-Han Chiang and Hung-yi Lee. 2023.  Can large language models be an alternative to human evaluations? In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 15607–15631, Toronto, Canada. Association for Computational Linguistics.
  * Chung et al. (2019) Yi-Ling Chung, Elizaveta Kuzmenko, Serra Sinem Tekiroglu, and Marco Guerini. 2019.  Conan–counter narratives through nichesourcing: a multilingual dataset of responses to fight online hate speech.  _arXiv preprint arXiv:1910.03270_.
  * Chung et al. (2021) Yi-Ling Chung, Serra Sinem Tekiroğlu, and Marco Guerini. 2021.  Towards knowledge-grounded counter narrative generation for hate speech.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 899–914, Online. Association for Computational Linguistics.
  * Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023.  Qlora: Efficient finetuning of quantized llms.  _arXiv preprint arXiv:2305.14314_.
  * Fanton et al. (2021a) Margherita Fanton, Helena Bonaldi, Serra Sinem Tekiroglu, and Marco Guerini. 2021a.  Human-in-the-loop for data collection: a multi-target counter narrative dataset to fight online hate speech.  _arXiv preprint arXiv:2107.08720_.
  * Fanton et al. (2021b) Margherita Fanton, Helena Bonaldi, Serra Sinem Tekiroğlu, and Marco Guerini. 2021b.  Human-in-the-loop for data collection: a multi-target counter narrative dataset to fight online hate speech.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics.
  * Hu et al. (2021) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7b.
  * Jones et al. (2024) Jaylen Jones, Lingbo Mo, Eric Fosler-Lussier, and Huan Sun. 2024.  A multi-aspect framework for counter narrative evaluation using large language models.
  * Ke et al. (2022) Pei Ke, Hao Zhou, Yankai Lin, Peng Li, Jie Zhou, Xiaoyan Zhu, and Minlie Huang. 2022.  Ctrleval: An unsupervised reference-free metric for evaluating controlled text generation.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu et al. (2023) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023.  G-eval: Nlg evaluation using gpt-4 with better human alignment.
  * Ni’mah et al. (2023) Iftitahu Ni’mah, Meng Fang, Vlado Menkovski, and Mykola Pechenizkiy. 2023.  Nlg evaluation metrics beyond correlation analysis: An empirical metric preference checklist.
  * Nimah et al. (2023) Iftitahu Nimah, Meng Fang, Vlado Menkovski, and Mykola Pechenizkiy. 2023.  NLG evaluation metrics beyond correlation analysis: An empirical metric preference checklist.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1240–1266, Toronto, Canada. Association for Computational Linguistics.
  * OpenAI et al. (2024) OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2024.  Gpt-4 technical report.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Saha et al. (2024) Punyajoy Saha, Aalok Agrawal, Abhik Jana, Chris Biemann, and Animesh Mukherjee. 2024.  On zero-shot counterspeech generation by LLMs.  In _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)_ , pages 12443–12454, Torino, Italia. ELRA and ICCL.
  * Sai et al. (2022) Ananya B. Sai, Akash Kumar Mohankumar, and Mitesh M. Khapra. 2022.  A survey of evaluation metrics used for nlg systems.  _ACM Comput. Surv._ , 55(2).
  * Tekiroğlu et al. (2022) Serra Sinem Tekiroğlu, Helena Bonaldi, Margherita Fanton, and Marco Guerini. 2022.  Using pre-trained language models for producing counter narratives against hate speech: a comparative study.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 3099–3114, Dublin, Ireland. Association for Computational Linguistics.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.
  * Tunstall et al. (2023) Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. 2023.  Zephyr: Direct distillation of lm alignment.
  * Vallecillo Rodríguez et al. (2024) María Estrella Vallecillo Rodríguez, Maria Victoria Cantero Romero, Isabel Cabrera De Castro, Arturo Montejo Ráez, and María Teresa Martín Valdivia. 2024.  CONAN-MT-SP: A Spanish corpus for counternarrative using GPT models.  In _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)_ , pages 3677–3688, Torino, Italy. ELRA and ICCL.
  * Wang et al. (2023) Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023.  Is chatgpt a good nlg evaluator? a preliminary study.
  * Wang and Wan (2018) Ke Wang and Xiaojun Wan. 2018.  Sentigan: Generating sentimental texts via mixture adversarial networks.  In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18_ , pages 4446–4452. International Joint Conferences on Artificial Intelligence Organization.
  * Wang et al. (2024) Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. 2024.  Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020.  Bertscore: Evaluating text generation with bert.
  * Zhao et al. (2023a) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023a.  A survey of large language models.
  * Zhao et al. (2023b) Xuandong Zhao, Siqi Ouyang, Zhiguo Yu, Ming Wu, and Lei Li. 2023b.  Pre-trained language models can be fully zero-shot learners.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 15590–15606, Toronto, Canada. Association for Computational Linguistics.
  * Zhong et al. (2022) Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022.  Towards a unified multi-dimensional evaluator for text generation.
  * Zhu et al. (2023) Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2023.  Judgelm: Fine-tuned large language models are scalable judges.

##  Appendix A Manual Evaluation Guidelines

We carry out two kinds of manual evaluation: rank-based evaluation (see
Section A.1) and feature-based evaluation (see Section A.2).

###  A.1 Rank Evaluation Guidelines

We will present an instance of HS followed by two candidate CNs: CNA and CNB.
Participants will choose which CN they find more effective in countering the
HS. If both CNs are equally unsatisfactory, participants can declare a tie.
Ties will only be applicable when both CN are deemed inadequate in addressing
the HS. Responses lacking specificity and informative content will incur
penalties, as will answers containing false information.

#### Instructions for Annotators

  * •

Carefully read the provided instance of hate speech.

  * •

Evaluate Counter-NarrativeA and Counter-NarrativeB based on their
effectiveness in addressing and countering the hate speech.

  * •

Choose the counter-narrative you find more effective. If both are equally
ineffective, declare a tie.

  * •

Consider the specificity and informative content of each counter-narrative.

  * •

Be vigilant for any false information in the responses, as these should be
penalized.

###  A.2 Feature Evaluation Guidelines

An instance of HS and a CN designed to combat it will be provided. The quality
of the CN will then be evaluated based on the following criteria:

#### Relatedness

Is the CN related to the HS?

  * •

0: No

  * •

1: Barely

  * •

2: Somewhat

  * •

3: More or less

  * •

4: Mostly

  * •

5: Yes

#### Specificity

Does the CN provide detailed and precise information?

  * •

0: Not specific at all

  * •

1: Barely specific

  * •

2: Somewhat specific

  * •

3: Moderately specific

  * •

4: Quite specific

  * •

5: Very specific

#### Richness

Does the CN include a variety of vocabulary and sentence structures?

  * •

0: Very poor vocabulary and structure

  * •

1: Barely rich

  * •

2: Somewhat rich

  * •

3: Moderately rich

  * •

4: Quite rich

  * •

5: Very rich

#### Coherence

Is the CN logically organized and easy to understand?

  * •

0: Not coherent at all

  * •

1: Barely coherent

  * •

2: Somewhat coherent

  * •

3: Moderately coherent

  * •

4: Quite coherent

  * •

5: Very coherent

#### Grammaticality

Is the CN grammatically correct and free of errors?

  * •

0: Completely ungrammatical

  * •

1: Barely grammatical

  * •

2: Somewhat grammatical

  * •

3: Moderately grammatical

  * •

4: Quite grammatical

  * •

5: Completely grammatical

#### Overall Score

How suitable is the CN as a response?

  * •

1: Not suitable (borderline hate speech)

  * •

2: Makes some acceptable points but not suitable

  * •

3: Would be suitable with some modifications

  * •

4: Good, though minor corrections may be needed

  * •

5: Very good as a CN

##  Appendix B Inter-Annotator Agreement

In this section, we show the tables of IAA from the evaluation process. These
tables highlight the consistency among annotators and the reliability of the
annotations. Figure B.1 shows the agreement for the Pairwise Rank-Based
evaluation method presented in Section 4.1, and Table B.1 for the feature
evaluation explained in Section 4.2.

Figure B.1: IAA of the Pairwise Rank-Based evaluation. Feature | IAA
---|---
Relatedness | 0.6397
Specificity | 0.5947
Richness | 0.7806
Coherence | 0.8270
Grammaticality | 0.7087
Overall | 0.6183
Table B.1: IAA of the feature-wise evaluation.

##  Appendix C Prompting

This section presents the exact formats used to prompt each of the models in
Section 3.1.

###  C.1 Mistral Prompt Format

Provide a brief counter-narrative in response to the user’s hate speech.
Ensure the output does not contain line breaks.
###Input:
Hate Speech
###Output:
Counter-Narrative

###  C.2 Mistral-Instruct Prompt Format

<s>[INST] Provide a brief counter-narrative in response to the user’s hate
speech. Ensure the output does not contain line breaks. Hate Speech [/INST]
Counter-Narrative</s>

###  C.3 Zephyr Prompt Format

<|system|>
Provide a brief counter-narrative in response to the user’s hate speech.
Ensure the output does not contain line breaks.</s>
<|user|>
Hate Speech</s>
<|assistant|>
Counter-Narrative</s>

###  C.4 Llama-Chat Prompt Format

<s>[INST] <<SYS>>
Provide a brief counter-narrative in response to the user’s hate speech.
Ensure the output does not contain line breaks.
<</SYS>>Hate Speech [/INST] Counter-Narrative </s>

###  C.5 JudgeLM Prompt Format

You are a helpful and precise assistant for checking the quality of the
answer.
`[`Question`]`
Provide a brief counter-narrative in response to the following hate speech:
Hate Speech It is very important for the counter-narrative to be relevant to
the hate speech.
`[`The Start of Assistant 1’s Answer`]`
Counter-NarrativeA
`[`The End of Assistant 1’s Answer`]`
`[`The Start of Assistant 2’s Answer`]`
Counter-NarrativeB
`[`The End of Assistant 2’s Answer`]`
`[`System`]`
We would like to request your feedback on the performance of two AI assistants
in response to the user question displayed above. Please rate the helpfulness,
relevance, accuracy, level of details of their responses. Each assistant
receives an overall score on a scale of 1 to 10, where a higher score
indicates better overall performance. Please first output a single line
containing only two values indicating the scores for Assistant 1 and 2,
respectively. The two scores are separated by a space. In the subsequent line,
please provide a comprehensive explanation of your evaluation, avoiding any
potential bias and ensuring that the order in which the responses were
presented does not affect your judgment.

##  Appendix D Correlation Matrix

Correlation between the metrics presented in Section 3.3 and human preference
in the CONAN corpus (Table D.1) and in the MT-CONAN corpus (Table D.2).

Figure D.1: Matrix with the Spearman’s rank correlation coefficients among
metrics, created using 360 tournaments from CONAN. The last row of the matrix
represents the correlation of all the evaluation methods to human preference.
Figure D.2: Matrix with the Spearman’s rank correlation coefficients among
metrics, created using 360 tournaments from MT-CONAN. The last row of the
matrix represents the correlation of all the evaluation methods to human
preference.