# Acknowledging Good Java Code
with Code Perfumes

Philipp Straubinger  University of Passau
Passau, Germany  Florian Obermüller  University of Passau
Passau, Germany  Gordon Fraser  University of Passau
Passau, Germany

###### Abstract

Java remains one of the most popular programming languages in education.
Although Java programming education is well supported by study materials,
learners also need more immediate support on the problems they face in their
own code. When this support cannot be offered by educators personally,
learners can resort to automated program analysis tools such as linters, which
provide feedback on potential bugs or code issues. This is constructive
feedback, but it may nevertheless feel like criticism. This paper introduces
code perfumes for Java, a simple program analysis technique similar to
linting, but commending the correct application of good programming practices.
We present a catalogue of 20 Java code perfumes related to common Java
language constructs for beginner to immediate learners. Our evaluation shows
that these code perfumes occur frequently in learners’ code, and programs with
more code perfume instances tend to have better functionality and readability.
Moreover, students who incorporate more code perfumes tend to achieve higher
grades. Thus, code perfumes serve as a valuable tool to acknowledge learners’
successes, and as a means to inform instructors about their learners’
progress.

###### Index Terms:

Linting, Code Quality, Java, Education

##  I Introduction

Java is one of the most frequently used languages to teach programming [1, 2],
and teaching materials and self-study materials are abundantly available.
However, independently of the maturity of these materials, students may
struggle in their attempts to produce functioning programs. Even when they
produce code that implements the desired functionalities, this is often prone
to quality problems, which the students again may struggle to identify and fix
on their own [3]. In a self-study scenario, or when tutors are overwhelmed
with large classrooms and cannot provide help and feedback promptly, learners
can resort to automated program analysis tools such as
SpotBugs111https://spotbugs.github.io/, last accessed 08.06.24 or SonarLint222
https://www.sonarsource.com/products/sonarlint/, last accessed 08.06.24.
Feedback provided by these tools on quality issues can help learners produce
better code and supports their learning processes, and tutors benefit from
insights into their students’ current skills and problems or
misunderstandings. However, such automated feedback tools mostly focus on
pointing out negative aspects.

While this kind of corrective feedback was shown to be very helpful in gaining
further cognitive skills [4], purely negative feedback may be detrimental to
self-efficacy and decrease intrinsic motivation [4]. In contrast, positive
feedback is considered to have better effects on motivational aspects [5].
Consequently, for effective learning feedback should consist of information
about errors as well as correct behaviour to address both cognitive and
motivational aspects. To this end the concept of _code perfumes_ was recently
introduced for the block-based Scratch programming language [6] as a means for
automated positive feedback: Code perfumes are patterns of code that are
considered as good programming practices or provide evidence of correctly
applied code constructs, implying an understanding of a certain programming
concept.

In this paper, we introduce the concept of code perfumes to the domain of
text-based programming languages by proposing a set of concrete code perfumes
for Java. For example, the following code shows a student approach to
programming a scoring method in a chess game assignment. The first line in the
method retrieves the colour of the human player from the board. However, if
the

    ChessBoard} object is
    \mintinline

Javanull this method call results in a

    NullPointerException}.
    \begin

mintedJava public static double score(ChessBoard board) Color hColor =
board.getHumanColor(); // Method logic… A better solution is to use _defensive
programming_ by checking parameter objects for

    null} values before working
    with them. The \emph

defensive null check code perfume acknowledges if learners correctly apply
this approach by checking whether the `ChessBoard} object is
\mintinline`Javanull at the beginning of the method, and if so reacting with
an

    IllegalArgumentException}:
    \begin

mintedJava public static double score(ChessBoard board) if (board == null)
throw new IllegalArgumentException( ”Board is null!”); Color hColor =
board.getHumanColor(); // Method logic… We propose a total of 20 code perfumes
covering various coding concepts, and evaluate them on a dataset of 816
student submissions to programming assignments in a university course. In
detail, the contributions of this paper are as follows:

  * •

We introduce the concept of code perfumes as a means of positive feedback to
Java.

  * •

We describe and implement a catalogue of 20 code perfumes from five
categories.

  * •

We empirically evaluate how frequently these code perfumes occur in student
code.

  * •

We empirically evaluate how the occurrence of code perfumes relates to
correctness of the programs.

Our investigation shows that most of the proposed code perfumes occur
frequently in Java programming assignments. Furthermore, our results suggest a
relation between the occurrence of code perfumes and correctness in the form
of functionality, readability and grades. These encouraging findings suggest
that code perfumes can provide positive feedback for Java and inform teachers
about the learning progress of students.

##  II Background

###  II-A Code Quality Problems

Even though learning to program may be hard, most students will manage to
produce functioning code after some time. However, research has shown that
even functionally correct programs show quality problems and that students
need feedback to stop producing more [3]. For example, a common type of
quality problem is _code smells_ , which are code idioms that increase the
likelihood of errors in future program edits or decrease the readability and
understandability of the code [7]. Prior research has shown that code smells
have negative effects on learning to program as they meddle with the learners’
ability to modify given code [8, 9]. _Bug patterns_ , on the other hand, are
aspects of code that are likely to lead to undesired behaviour (i.e., bugs or
defects) [10]. In particular, misconceptions of learners often manifest in
bugs that follow recurring bug patterns [11, 12, 13]. While finding and
removing such quality problems is inherent to coding and learning to program,
tools can offer support to learners as well as teachers. Static analysis tools
(i.e., linters) like SpotBugs and SonarLint can automatically detect these
quality problems and generate feedback, which has been shown to help students’
code quality skills [14]. Furthermore, static analysis of code patterns is
part of automated feedback and grading tools like ArTEMiS [15] providing
information to teachers for assessing their students.

###  II-B Feedback

Feedback generated by automated program analysis tools is returned regardless
of other characteristics of the student and is accordingly perceived as less
threatening [5]. No hand raising is needed, thus preventing peer
stigmatisation [16]. Computer generated feedback mainly focuses on bad coding
practices or missing functionalities [1]. Such negative feedback may be useful
in programming education in multiple ways: Teachers assessing learners’ code
and supporting students with their individual problems when programming need
to know about their students’ misunderstandings and currently lacking skills
and tools that can help in analysing those [17, 18, 19]. Novices can benefit
as corrective feedback helps to acquire further knowledge and cognitive skills
[4, 14]. On the other hand, positive feedback is important as it leads to
higher intrinsic motivation than negative feedback [5]. Previous research also
suggests that more motivated learners process all kinds of feedback better
[20]. This in turn would possibly lead to even higher effects of the
corrective feedback. Unfortunately, positive feedback generated by automated
tools is rare [1]. Most attempts to introduce positive aspects to automated
feedback are task-specific and give information about what parts of a task are
correct [21]. While this is useful guidance on how to solve specified tasks,
it gives no information about general aspects, like style or quality. One
attempt to address this deficiency in the context of Java is the use of
gamification for reducing warnings produced by analysis tools [22]. While this
keeps motivation high [23] and leads to better code, it only minimises the
negative feedback and does not contain any information about good coding
parts. Positive feedback is more common in educational block-based
programming. For the block-based programming language Scratch [24] the Dr.
Scratch [25] tool displays praise in the form of a short text such as “You’re
doing a great job. Keep it up!!!” and returns points on computational thinking
concepts333http://www.drscratch.org/, last accessed 08.06.24. The LitterBox
[26] static analysis tool can not only point out bug patterns and code smells
but also detects _code perfumes_ , which are code idioms indicating good
programming practices or code showing understanding of certain programming
concepts [6]. Code perfumes can be seen as the opposite of code smells or bug
patterns and are not bound to a specific task but can be applied to any given
project. They have been shown to appear frequently in Scratch projects and can
be used as an indicator of correctness, as projects having a higher count of
code perfumes tend to be more correct [6]. While Java is still one of the most
used languages to learn programming [1, 2], similar positive feedback is
absent, which may limit the learning effects of automated tools like ArTEMiS
[15]. To address this problem, we introduce code perfumes to this domain such
that holistic feedback combining positive and negative aspects can be
generated automatically.

##  III Code Perfumes for Java

Code perfumes are a general concept applicable to any programming language or
skill level. In this paper, we target Java learners who already have some
basic understanding, and are able to produce code using external APIs or
design patterns. A specific concern related to such learners is that even
though they are familiar with the syntax, they struggle to assess the quality
of their code [27]. However, the code perfumes we define may also appeal to
more experienced programmers. Code perfumes are expected to have mostly
positive effects on the code quality, be discernible in source code without
execution, and align with language-dependent concepts. However, programmers
must carefully assess their code and determine whether applying a code perfume
is appropriate in a specific context, or if other solutions should be
considered. This problem could be exacerbated by false positives, which is why
implementations of checks for code perfumes need to aim to minimise false
positives while acknowledging the need for careful application. We categorise
code perfumes based on their purpose:

  * •

Solution Patterns: These are code structures that solve undesirable code
patterns (e.g., code smells or bug patterns). They directly address and avoid
matching negative patterns, but not all bug patterns imply solution patterns
[28]. Ideal solution patterns provide additional positive benefits beyond
eliminating negative patterns.

  * •

Best Practices: Established best practices in Java development can serve as
the foundation for code perfumes. Capturing best practices in code patterns
allow learners to leverage the results of experienced programmers [29].

  * •

Contracts or Conventions: Java contracts, often documented in method
descriptions, form another basis for code perfumes. Adhering to contracts,
especially from the standard library is crucial for developing correct code
and effective collaboration in development teams [30].

To define code perfumes for Java, we draw inspiration from the SonarRules
collection of negative patterns by SonarSource[31] and the popular book
Effective Java by Joshua Bloch [32]. SonarRules provides established code
smells, bug patterns, and vulnerabilities, influencing the creation of
solution pattern code perfumes. Effective Java offers best practices and
guidance for Java programming, focusing on topics such as design patterns,
concurrency, and Java standard library traits. Additionally, the Common
Weakness Enumeration444https://cwe.mitre.org/index.html, last accessed
08.06.2024 was considered as a background source, linking weaknesses
identified by SonarRules and providing additional information. Finally, the
article Mining frequent bug-fix code changes [33] also contributed to a code
perfume definition.

TABLE I: Categories and types of code perfumes

Category | ID | Code Perfume | Type
---|---|---|---
Java Standard Library | 1 | Clone blueprint | solution pattern, contract or convention
2 | Equals blueprint | contract or convention, best practice
3 | Iterator next() follows the contract | contract or convention
4 | Override compareTo with equals | solution pattern, best practice
5 | Override equals of superclass | solution pattern
6 | Paired equals and hashCode | contract or convention, solution pattern
7 | Use optimized collections for Enums | best practice, solution pattern
Java language features | 8 | ‘assert’ in private method | solution pattern, best practice
9 | At least X varargs | solution pattern
10 | Defensive default case | solution pattern
11 | Pattern matching with ‘instanceof’ | solution pattern
12 | Resource management in try-catch | best practice, solution pattern
13 | Synchronize accessors in pairs | solution pattern
Unit Testing Practices | 14 | JUnit 5 tests can be package-private | solution pattern
15 | Single method call when testing for runtime exceptions | solution pattern
Design Patterns | 16 | Builder pattern | best practice
17 | Singleton pattern | best practice
Others | 18 | Copy constructor | best practice
19 | Defensive null check | best practice, solution pattern
20 | No utility instantiation | solution pattern, best practice

###  III-A Java Standard Library Code Perfumes

Code perfumes in this category emphasise the importance of understanding
concepts from the Java standard library, which beginners should encounter
early in their coding journey [32].

##### Clone blueprint:

This code perfume focuses on implementing the `java.lang.Cloneable} interface
to provide a copy functionality. While it may not be the ideal approach, it is
a valid option if done correctly. The code perfume emphasises the importance
of consistently calling \mintinline`Javasuper.clone in

    clone} overrides in an inheritance hierarchy, to maintain the correct runtime typing of the cloned object. The code perfume aligns with the Common Weakness Enumeration (CWE-580) and the corresponding SonarSource rule (RSPEC-1182).
    \begin

mintedJava class DisplayData implements Cloneable @Override public
synchronized DisplayData clone() DisplayData displayData; try displayData =
(DisplayData) super.clone(); catch (CloneNotSupportedException e) throw new
Error(e); // Deep copy return displayData;

##### Equals blueprint:

Overriding the `equals} method in the \mintinline`Javajava.lang. `Object}
class requires adherence to a specific contract, implementing an equivalence
relation between instances of the class. To simplify the coding process and
reduce the likelihood of errors, Joshua Bloch provides a
guideline~\cite`DBLP:books/lib/Bloch08. This code perfume focuses on the first
two steps of Bloch’s guideline, checking object references and type
correctness, as the comparison of class fields can vary.

⬇

@Override

public boolean equals(Object o) {

if (o == this) {

return true;

}

if (!(o instanceof Pawn)) {

return false;

}

// Field checks

}

##### Iterator next() follows the contract:

This code perfume focuses on classes that implement the `java.util.Iterator}
interface in Java. This interface allows for iterating over data structures,
such as lists. The key methods for iteration are \mintinline`JavahasNext and
`next}, with the latter having a contractual requirement to throw a
\mintinline`JavaNoSuchElementException when called without a remaining element
in the iteration. This behaviour is expected by developers using iterators,
and omitting it can lead to unexpected errors. This code perfume checks
whether a class implementing the `Iterator} interface adheres to the necessary
step of throwing a \mintinline`JavaNoSuchElementException in the

    next} method. It is inspired by the rule from SonarSource (RSPEC-2272), which considers the absence of this behaviour a bug.
    \begin

mintedJava public final class Range implements Iterator¡Integer¿ @Override
public boolean hasNext() return contains(start); @Override public Integer
next() if (!hasNext()) throw new NoSuchElementException(); // Iterator logic

##### Override compareTo with equals:

The `java.lang.Comparable} interface includes the \mintinline`JavacompareTo
method for defining a natural ordering between instances of the class. The
method’s contract suggests that if two objects are equal according to the
`equals} method, they should also be equal according to the
\mintinline`JavacompareTo method, and vice versa. While this is only a
recommendation, it is considered good practice to override both methods. The
code perfume serves as a solution pattern to the corresponding SonarSource
rule (RSPEC-1210) checking the presence one of these methods.

⬇

class Command implements Comparable<Command> {

@Override

public int compareTo(Command command) { … }

\par@Override

public boolean equals(Object o) { … }

}

##### Override equals of superclass:

This code perfume addresses a common issue in inheritance hierarchies where
subclasses add fields to those already defined in their superclasses. If a
superclass overrides the `equals} method, but the subclass fails to override
it, there is a potential mistake. SonarSource (RSPEC-2160) identifies this, as
it may lead to unexpected behaviour where the additional fields in the
subclass are ignored by the \mintinline`Javaequals method.

⬇

public class Ancestor {

@Override

public boolean equals(Object o) { … }

}

public class Child extends Ancestor {

int additionalField;

@Override

public boolean equals(Object o) { … }

}

##### Paired equals and hashCode:

This code perfume highlights the importance of maintaining consistency between
the `equals} and \mintinline`JavahashCode methods in Java. While classes can
override these methods independently, it is considered best practice to
override both or neither to avoid violating the contract they form. The
contract, particularly for `hashCode}, states that objects deemed equal by the
\mintinline`Javaequals method should have the same hashCode. Although the
contract, also denoted by SonarSource (RSPEC-1206) and CWE (CWE-581), allows
distinct objects to have the same hashCode, it is recommended to override both
methods together for consistent behaviour and improved performance in hash
tables.

⬇

public class SudokuBoard implements Board {

@Override

public boolean equals(Object other) { … }

\par@Override

public int hashCode() { … }

}

##### Use optimised collections for Enums:

This code perfume focuses on optimising memory usage and execution speed in
Java applications, particularly when dealing with Enum types. It encourages
the use of optimised classes provided by the Java standard library’s
`Collections} API for Enums. The recommendation to use these optimised
collections for Enums is emphasised by Joshua
Bloch~\cite`DBLP:books/lib/Bloch08 and two SonarSource rules (RSPEC-1640 and
RSPEC-1641).

⬇

public class KalahBoard implements Board {

private final EnumMap<Player, PlayerPits>

playerPits;

public KalahBoard( … ) {

this.playerPits =

new EnumMap<>(Player.class);

// Constructor code

}

}

###  III-B Java Language Feature Code Perfumes

Code perfumes in this category encourage the use of language features, idioms,
and keywords in Java coding practices, with a focus on defensive programming
[34] and addressing code smells or bug patterns [35].

##### ‘assert’ in private method:

While the

    assert} keyword is useful for validating
    parameters, it should be avoided in public methods because assertions
    must be explicitly enabled at the JVM level. Best practice involves
    using \mintinline

Javaassert in private methods during development for parameter validation. To
enhance clarity and to ease error identification, the

    assert} expression should be
    accompanied by a message. This aligns with SonarSource’s rule
    (RSPEC-4274) against using \mintinline

Javaasserts to check public method parameters, and is also highlighted in
Effective Java [32].

⬇

private boolean isMovePossible(int row,

int diag, Color color) {

assert isValidPosition(row, diag)

: ”Invalid coordinates”;

// Method logic

}

##### At least X varargs:

This code perfume addresses the use of varargs in Java methods, emphasising a
more efficient approach to set a lower bound on the number of parameters.
Instead of checking the number of parameters within the method’s body
resulting in unnecessary code, it encourages declaring a specific number of
parameters before the varargs. This approach, recommended by Joshua Bloch
[32], allows for compile-time validation of the minimum argument count,
improving efficiency compared to runtime checks.

⬇

public static boolean isWithinCharset(

int charsetSize, int… symbols) {

return symbols != null

&& Arrays.stream(symbols)

.allMatch(i -> i >= 0 && i < charsetSize);

}

##### Defensive default case:

This code perfume underscores the importance of using

    switch} statements effectively
    as a cleaner alternative to chaining
    \mintinline

Javaif-statements. The

    switch}
    construct allows handling every possibility of the
    \mintinline

Javaswitch-variable, with the

    default} case serving as a safety net to address
    unforeseen situations. Including a \mintinline

Javadefault-case at the end of a

    switch} statement demonstrates
    defensive programming, reducing the risk of unexpected behaviour due
    to overlooked cases as reported by SonarSource (RSPEC-131 and CWE
    (CWE-478)).
    \begin

mintedJava switch (firstLetter) case ’h’: cmdHelp(token); break; // More cases
default: printError(”Invalid Command!”); break;

##### Pattern matching with ‘instanceof’:

This code perfume promotes the use of the

    instanceof} operator in
    conjunction with pattern matching, introduced in Java 16. While the
    \mintinline

Javainstanceof operator traditionally required manual casting, pattern
matching now allows for a concise type-check and cast in a single

    instanceof} statement. This enhances
    clarity and readability, as promoted by SonarSource (RSPEC-6201).
    \begin

mintedJava if (copy instanceof Game gameCopy) gameCopy.cells = cells.clone();

##### Resource management in try-catch:

This code perfume encourages the use of the try-with-resources statement in
Java to ensure the proper closing of resources within a try-catch block. The
try-with-resources statement is considered a best practice inspired by
SonarSource (RSPEC-2093) to prevent accidental resource leaks, especially for
beginners working with file reading and I/O streams.

⬇

try (BufferedReader reader = new BufferedReader(

new FileReader(file))) {

Line line = readLine(reader, 0);

// Method logic

}

##### Synchronize accessors in pairs:

This code perfume encourages careful handling of parallelisation, emphasising
the use of the

    synchronized} keyword to synchronise specific parts of the execution flow and prevent race conditions. It particularly focuses on scenarios involving accessor methods (getters and setters) for class fields. The code perfume rewards the practice of synchronising both accessors, addressing a potential bug as identified by SonarSource (RSPEC-2886).
    \begin

mintedJava synchronized void setCell(int rowNr, int colNr, int number) …
synchronized int getCell(int rowNr, int colNr) …

###  III-C Unit Testing Code Perfumes

Code perfumes in this category recognise the standard practice of unit testing
[36] and the need for high-quality test code to avoid maintainability issues
[37, 38, 39].

##### JUnit 5 tests can be package-private:

This code perfume advocates for adhering to the principle of operating with
the least required privileges in Java coding. Specifically, it encourages
developers to choose the narrowest scope possible for class visibility,
including unit tests. With JUnit 5, both tests and test classes do not have to
be public anymore, making them package-private is sufficient for JUnit to
detect and execute them; inspired by SonarSource (RSPEC-5786).

⬇

class PerfumeTest {

@Test

void someTest() { … }

}

##### Single method call when testing for runtime exceptions:

This code perfume emphasises the importance of testing code for expected
RuntimeExceptions using frameworks like JUnit and AssertJ. It discourages
chaining multiple method calls within the `assertThrows} or
\mintinline`JavaassertThatThrownBy methods, as this can obscure which specific
method call caused the expected exception or whether any call did at all. The
code perfume suggests using only a single method call in these methods for
improved code clarity and consistency as taken from SonarSource (RSPEC-5778).

⬇

@Test

void someTest() {

assertThrows(Exception.class,

() -> callMethod());

}

###  III-D Design Pattern Code Perfumes

Code perfumes in this category acknowledge the value of design patterns in
advanced programming education and their contribution to well-formed code [40,
41, 42].

##### Builder pattern:

This code perfume underscores the benefits of using the Builder pattern to
address the challenge of classes with numerous optional fields. Instead of
creating an excessive number of constructor overloads, the Builder pattern
involves an inner class responsible for constructing instances of the outer
class. This approach is more flexible, allowing easier extension of the
builder and the creation of multiple instances using the same builder. Joshua
Bloch [32] recommends considering the Builder pattern when a class’s
constructor would require four or more parameters.

⬇

public class ToBuild {

private ToBuild(Builder builder) { … }

public static class Builder {

public ToBuild build() {

return new ToBuild(this);

}

}

}

##### Singleton pattern:

This code perfume focuses on the Singleton pattern for enforcing the existence
of exactly one instance of a class. Joshua Bloch [32] describes three main
ways to implement the Singleton pattern: using a public instance field, a
public static instance factory method, or defining the class as an enum type
with a single constant. Each approach has its advantages, and all are
considered valid.

⬇

static final class Controller {

private static Controller instance;

private Controller() { }

public static Controller getInstance() {

return instance;

}

}

###  III-E Other Code Perfumes

Finally, some code perfumes fit in no other category.

##### Copy constructor:

This code perfume considers object copying in Java, emphasising the pitfalls
associated with the `Cloneable} interface, as highlighted by Joshua
Bloch~\cite`DBLP:books/lib/Bloch08. A copy constructor is recommended as a
clean solution that takes an object of the same class as its parameter and
performs a deep copy of all fields, including mutable non-primitive ones. This
approach ensures the desired runtime type and is less error-prone than
implementing

    Cloneable}.
    \begin

mintedJava public Tape(Tape clone) content = new LinkedList¡¿(clone.content);
headPosition = clone.headPosition; move(Direction.NOTHING);

##### Defensive null check:

This code perfume underscores the importance of caution when declaring public
methods that deal with non-primitive parameter types. The risk of bugs and
unexpected behaviour is increased when null values are passed as parameters.
To prevent unexpected `NullPointerExceptions} at runtime, the code perfume
recommends checking non-primitive parameters for non-null values or using
annotations like \mintinline`Java@NotNull or `@Nonnull} to specify that null
values are not allowed. Alternatively, if the method permits null values for
certain parameters, using an \mintinline`Java@Nullable annotation is advised
by Joshua Bloch [32].

⬇

public static void callPrint(

TuringMachine turingMachine) {

if (turingMachine != null) {

System.out.println(turingMachine.print());

} else {

errorMsg(”No Turing machine initialized.”);

}

}

##### No utility instantiation:

This code perfume emphasises the definition of a private constructor for
utility classes containing only static methods. Since utility classes do not
need to be instantiated and thus need no implicit default constructor, having
a single private constructor demonstrates an understanding of Java mechanics
and object orientation as described by SonarSource (RSPEC-1118).

⬇

public final class Shell {

private Shell() { }

private static void process(String[] strings)

{ … }

private static void add(String[] strings)

{ … }

}

###  III-F _Perfumator_ : A Tool to Detect Code Perfumes in Java Code

The _Perfumator_ is a static analysis tool implemented in Java 17, with its
primary purpose being the detection of code perfumes in abstract syntax trees
(ASTs). Its functionality involves two main steps: defining static information
in a `JSON} file and implementing a \mintinline`JavaDetector class that
recognises specific patterns in the AST. JavaParser is utilized by the
_Perfumator_ for parsing ASTs and resolving symbols. The tool offers a
command-line interface featuring options like input path, output format, batch
size, specifying dependencies as well as support for internationalization. The
_Perfumator_ identifies instances of code perfumes in designated input files
or directories and generates serialised output in ``JSON˝ format containing
detailed information. To enhance accuracy, the tool allows ignoring specific
files and directories like the build directory during analysis. The “toolname
is designed to be easily extendable with new code perfumes and holds the
potential for integration into other tools. %%% Local Variables: %%% mode:
latex %%% TeX-master: ”../paper” %%% End:

##  IV Evaluation

To gain an initial understanding of the prevalence of the code perfumes
defined in this paper, and how they relate to correctness, we applied the
_Perfumator_ to a large dataset of programming assignments and student
solutions. This allows us to address the following research questions:

  * •

RQ1: How common are code perfumes in Java?

  * •

RQ2: Are code perfumes related to correctness?

###  IV-A Experimental Setup

TABLE II: Assignment submissions and size Task | Name | Total LoC | Submissions | Average LoC
---|---|---|---|---
0 | Practice | 133931 | 359 | 373
1 | Algorithms | 442450 | 809 | 547
2 | CLI Board Game | 659318 | 607 | 1086
3 | GUI Board Game | 609915 | 560 | 1089

In order to study code perfumes, we use a dataset derived from the Programming
II course taught every semester at the University of Passau. This course
focuses on intermediate/advanced Java programming, covering topics such as
Java Swing and parallelism. The coursework consists of four programming
assignments. Although the specific tasks vary between different instances of
the course, they are always variations of the same type of program, and
usually represent the implementation of a board game like
Abalone555https://en.wikipedia.org/wiki/Abalone_(board_game), last accessed
08.06.2024 or
Mastermind666https://en.wikipedia.org/wiki/Mastermind_(board_game), last
accessed 08.06.2024. Due to the similarity of the solution code, we merge
different variants of the same task in our dataset. The tasks include:  • Task
0: Exercise task for familiarity with the automatic grading system  • Task 1:
Mathematical problem-solving to acquaint students with organising programs and
creating complex algorithms  • Task 2: Modelling the board game with a
command-line interface  • Task 3: Implementing a user interface for the
previously modeled game  The first assignment only serves for practice, while
the remaining assignments determine the overall grade. Until the submission
deadline, students can re-submit as often as they wish, receiving feedback
from a set of public tests. After the deadline, functionality is assessed
using a set of secret tests and normalised to the range 1 (all tests pass) to
6 (no tests pass). Additionally, each submission is graded manually on
readability considering coding styles, code structure, and variable/method
naming on a range of 1 to 6. The overall grade is a weighted sum of
functionality and readability ranging from 1 (best) to 5 (worst) with
intermediate steps of .3 and .7. We use data from this course between the
winter semester of 2015/2016 to the winter semester of 2022/2023, covering 15
iterations of the course. Students who registered but did not submit a task
were excluded from the analysis. This resulted in a dataset of 816 students,
acknowledging that some students may have participated more than once to pass
the course. II shows that less than half of the students submitted Task 0,
which was optional. The highest number of submissions was for Task 1, the
first graded task, with subsequent tasks seeing declining submission rates.
The lines of code (LoC) increased with each task, with Tasks 2 and 3 having
almost identical LoC counts. To understand the limitations and applicability
of our code perfumes, we also consider the Progpedia dataset [43] consisting
of 16 programming assignments for undergraduate students. The dataset consists
of 4362 student solutions. The assignments are very basic and cover topics
from reading user input from the console to basic algorithms like Kruskal
[44].

####  IV-A1 RQ1: How common are code perfumes in Java?

To answer this research question, we execute the _Perfumator_ on all student
submissions for both datasets and extract the number of code perfumes per
category.

####  IV-A2 RQ2: Are code perfumes related to correctness?

To answer this research question, we compare the number of code perfumes with
other metrics of their solutions, such as the grade on functionality, the
grade on readability, the overall grades for the course, and the number of
lines of code (LoC), assessed using CLOC777https://github.com/AlDanial/cloc,
last accessed 08.06.2024. To contrast the insights gained from positive code
aspects such as code perfumes, we also examine negative aspects by extracting
the number of code smells using SonarLint2. Subsequently, we use Pearson
correlation [45] to determine potential correlations between these metrics.

###  IV-B Threats to Validity

_Threats to internal validity_ may result from task selection, grading
criteria, and inconsistencies in both automatic and manual grading processes.
However, the grading process remained consistent over the years with the same
supervisor and the same grading criteria. Moreover, the act of testing itself
might influence students’ subsequent performance, and external factors like
changes in course structure or students’ natural skill progression could also
impact the outcomes. The _Perfumator_ might not accurately detect all
perfumes, but we minimized this issue by verifying the results beforehand.
_Threats to external validity_ exist because findings might not generalise
beyond the specific student population or programming course and university.
Additionally, the tasks may not accurately mirror real-world programming
scenarios, and changes over time in technology, education methods, or student
demographics could affect the results. _Threats to construct validity_ may
result from the tools utilised for assessment, which may not fully capture the
intended variables or could introduce errors. All students had the possibility
to analyse their graded results and discuss them with the supervisor,
therefore reducing this threat. Maintaining consistency in task specifics and
evaluation across different course iterations potentially affects treatment
effects.

###  IV-C RQ1: How Common are Code Perfumes in Java?

TABLE III: Code Perfumes overall and per task ID | Total | Submissions | Task 0 | Task 1 | Task 2 | Task 3
---|---|---|---|---|---|---
1 | 196 | 4.0% | 0 | 3 | 97 | 96
2 | 17 | 0.3% | 0 | 7 | 5 | 5
3 | 2 | 0.0% | 0 | 0 | 1 | 1
4 | 281 | 5.8% | 0 | 254 | 12 | 15
5 | 0 | – | 0 | 0 | 0 | 0
6 | 419 | 8.3% | 0 | 259 | 79 | 81
7 | 10 | 0.1% | 0 | 0 | 5 | 5
8 | 520 | 1.7% | 1 | 6 | 257 | 256
9 | 4 | 0.0% | 0 | 1 | 3 | 0
10 | 3656 | 43.3% | 426 | 871 | 1346 | 1013
11 | 21 | 0.3% | 0 | 2 | 2 | 17
12 | 134 | 3.0% | 2 | 80 | 37 | 15
13 | 9 | 0.1% | 0 | 0 | 0 | 9
14 | 0 | – | 0 | 0 | 0 | 0
15 | 0 | – | 0 | 0 | 0 | 0
16 | 0 | – | 0 | 0 | 0 | 0
17 | 17 | 0.2% | 0 | 0 | 6 | 11
18 | 21 | 0.5% | 0 | 5 | 7 | 9
19 | 1048 | 11.5% | 125 | 271 | 250 | 402
20 | 2243 | 43.0% | 4303 | 851 | 671 | 418

III lists the overall number of code perfumes found, their distribution across
tasks, and the percentage of submissions containing them. In total, the
_Perfumator_ identified 8598 instances of perfumes in the dataset. Only a few
of the code perfumes are not found, which is due to the nature of the tasks:
The absence of the override equals of superclass code perfume (ID 5) is
expected, given the relatively simple project structures that may not involve
extensive inheritance hierarchies. Code perfumes related to testing (IDs 14
and 15) are unsurprisingly absent, considering the absence of unit testing in
the course. Similarly, design patterns (IDs 16 and 17) are infrequently used,
possibly due to students’ limited awareness of builder and singleton patterns
or the tasks not necessitating their implementation. The copy constructor code
perfume (ID 18) sees limited use, as most students opt for the cloning
functionality (ID 1). However, all other code perfumes can be found frequently
in the dataset, suggesting that they are relevant and suitable for the type of
code we are targeting. The distribution of code perfumes reveals insights into
the coding behaviour of the students. For example, defensive programming
practices are very prevalent among students and these assignments. Three
specific code perfumes (IDs 10, 19, and 20) collectively constitute over 80%
of all detected instances, indicating a strong emphasis on defensive
programming. Notably, the assert in private method code perfume (ID 8) is
frequently encountered, along with the defensive null check code perfume (ID
19), suggesting a preference for null checks in public methods over
assertions. The code perfume occurrences also suggest adherence to certain
Java language conventions, particularly in cases where students override the

    equals} method alongside either
    \mintinline

JavahashCode or

    compareTo} (IDs 4 and
    6). However, the students notably deviate from the common
    \mintinline

Javaequals blueprint by not consistently checking object references and type
correctness. An intriguing observation is also that students rarely employ
synchronized accessors in pairs (ID 13), a crucial aspect introduced in the
final assignment of each course. This oversight could lead to runtime
exceptions and deadlocks in their implemented games. Optimised collections for
enums (ID 7) are infrequently used, indicating potential unawareness of these
specialised collections. The same may apply to the at least X varargs code
perfume (ID 9), while the pattern matching with instanceof code perfume (ID
11) reflects the limited adoption due to its recent functionality release in
2021. This raises the question whether the inclusion of code perfumes into
student feedback could help improving such deficiencies. It is noteworthy that
the frequency of finding a code perfume does not necessarily imply even
distribution across all submissions. For instance, although code perfume ID 20
is encountered twice as often as code perfume ID 19, the former appears in
nearly four times as many submissions as the latter (see III). Similarly, code
perfumes 6 and 8 exhibit a similar trend, with code perfume 6 appearing in
nearly five times more submissions than code perfume 8\. Consequently, some
students have internalised aspects represented by code perfumes and use them
frequently, while those with fewer occurrences may still be in the process of
adopting these constructs. It is conceivable that positive reinforcement by
reporting code perfumes could support this learning process. To better
understand the limits of the code perfumes defined in this paper, we contrast
this data with the ProgPedia dataset of introductory assignments, where we
only found a total of 14 code perfume instances (IDs 10, 18, and 19) on 4362
student submissions. This can be attributed to the very simplistic nature of
those assignments, they only consist of one file and mostly can be done in 30
to 60 LoC. While this shows that some students are aware of defensive
programming this early in their programming education, more code perfumes
targetting earlier, simpler stages of coding are clearly needed to fittingly
evaluate students in a CS1/Algorithmics class, while our code perfumes are
more fitted for students in a CS2/Software Engineering class, as shown above.

Summary (RQ1): The _Perfumator_ identified 8598 instances of code perfumes,
highlighting a significant emphasis on defensive programming practices among
students, while other conventions are not as commonly embraced. Our code
perfumes target intermediate learners covering design principles; more code
perfumes are needed to cover beginners’ code.

###  IV-D RQ2: Are Code Perfumes Related to Correctness?

(a) Number of code perfumes vs. functionality

(b) Number of code perfumes vs. readability

(c) Number of code perfumes vs. grades

(d) Number of code perfumes vs. lines of code

(e) Number of code perfumes vs. code smells

(f) Lines of code vs. functionality

(g) Lines of code vs. readability

(h) Lines of code vs. grades

(i) Lines of code vs. smells

(j) Number of code smells vs. functionality

(k) Number of code smells vs. readability

(l) Correlation between number of code smells and grades

Figure 1: Correlations between number of code perfumes, code smells, line of
code, functionality, readability, and grades

1 visualises the correlations between the metrics considered. According to
Pearson correlation [45], there is a weak negative correlation between the
number of code perfumes and the functionality (1(a)) and readability (1(b))
assigned to students assignments. Additionally, there is a moderate negative
correlation between the number of code perfumes and the grades (1(c)) received
by the students. Since the best grade is 1, the negative correlations suggest
that code perfumes can serve as an indicator for correctness as well as code
quality, and thus represent a valuable tool for educators. However, we can
generally also see that LoC are correlated with functionality (1(f)),
readability (1(g)), and grades (1(h)), as more complete solutions tend to
consist of more code. When normalising code perfumes per LoC, we find no
correlations between code perfumes per LoC and functionality (0.01),
readability (-0.04), and grades (0.06). An explanation for this is that our 20
code perfumes are too few. While code perfumes can inform tutors about their
students and aspects they perform well on, for code perfumes to serve as a
reliable predictor for student performance, more code perfumes will need to be
defined that cover more aspects of functionality and readability. The
immediate application of code perfumes for educators thus for now primarily
lies in observing which aspects of code are handled well by students, or
identifying students that are lacking understanding of certain concepts. The
complementary nature of code smells and code perfumes is indicated by the
substantially weaker correlations between code smells and functionality
(1(j)), readability (1(k)), and grades (1(l)), even though more code also
leads to more code smells (1(i)). The most common code smells are related to
logging, cognitive complexity, optimising conditionals, multi-threading and
code duplication. The analyses on code smells are influenced by the large
differences between the numbers of code perfumes and code smells, as SonarLint
defines 677 rules [31]. On the one hand, this reinforces the conclusion that
more code perfumes are needed. On the other hand, it also suggests that
neither mechanism on its own can be used for reliable grading, but the
combination of negative and positive aspects provides valuable information.
Thus, classic linters and the perfumator could be used in combination to
create a more holistic view of the learners project helping educators support
and assess their learners.

Summary (RQ2): The presence of code perfumes is associated with improved
readability, functionality, and higher grades, and therefore represent a
valuable tool to inform educators about their students progress. However, the
vast difference in the number of existing code smells and code perfumes
suggests future work on defining more code perfumes.

##  V Conclusions

Program analysis can point out problems in code, but they usually do not
provide encouraging, positive feedback. In this paper, we therefore introduced
the concept of code perfumes for Java, and presented 20 code perfumes to
provide learners with constructive feedback. Our analysis indicates that many
of these code perfumes are common in practice, and we find a link between the
presence of code perfumes and program correctness, highlighting their
potential to aid teachers’ understanding of student learning progress. While
these initial findings are encouraging, our catalogue of 20 code perfumes is
by no means complete, and we anticipate future work to result in many
additional code perfumes. Investigating how to integrate these in the learning
process will be similarly important. For example, code perfumes could be
annotated directly in Integrated Development Environments (IDEs) or in
gamified environments such as Gamekins [46]. Educators may need different ways
to interface with code perfumes, for example by incorporating them as a metric
in grading and learning systems such as ArTEMiS [15]. Finally, although our
motivation originally was to support learners, there is no reason why code
perfumes could not also be used to encourage professional programmers. The
_Perfumator_ is available at:

https://github.com/se2p/perfumator

## Acknowledgements

This work is supported by the DFG under grant FR 2955/2-1, “QuestWare:
Gamifying the Quest for Software Tests”. Thanks to Jakob Silbereisen for his
contributions to the _Perfumator_.

## References

  * [1] W. Conceição, M. Holanda, F. Macedo, E. Ishikawa, V. T. Nunes, and D. Da Silva, “Automatic feedback in the teaching of programming in undergraduate courses: a literature mapping,” in 2022 IEEE Frontiers in Education Conference (FIE), 2022.
  * [2] S. C. dos Santos, P. C. A. R. Tedesco, M. Borba, and M. Brito, “Innovative approaches in teaching programming: A systematic literature review,” in International Conference on Computer Supported Education, 2020.
  * [3] T. Effenberger and R. Pelánek, “Code quality defects across introductory programming topics,” in Proceedings of the 53rd ACM Technical Symposium on Computer Science Education - Volume 1, SIGCSE 2022, Association for Computing Machinery, 2022.
  * [4] B. Wisniewski, K. Zierer, and J. Hattie, “The power of feedback revisited: A meta-analysis of educational feedback research,” Frontiers in Psychology, 2020.
  * [5] J. Hattie, Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement.  01 2009.
  * [6] F. Obermüller, L. Bloch, L. Greifenstein, U. Heuer, and G. Fraser, Code Perfumes: Reporting Good Code to Encourage Learners.  Association for Computing Machinery, 2021.
  * [7] M. Fowler, Refactoring: Improving the Design of Existing Code.  Boston, MA, USA: Addison-Wesley, 1999.
  * [8] F. Hermans and E. Aivaloglou, “Do code smells hamper novice programming? a controlled experiment on scratch programs,” in 2016 IEEE 24th International Conference on Program Comprehension (ICPC), 2016.
  * [9] V. Gutmann, E. Starke, and T. Michaeli, “Investigating code smells in k-12 students’ programming projects: Impact on comprehensibility and modifiability,” 2022.
  * [10] D. Hovemeyer and W. Pugh, “Finding bugs is easy,” SIGPLAN Not., 2004.
  * [11] A. Swidan, F. Hermans, and M. Smit, “Programming misconceptions for school students,” in Proceedings of the 2018 ACM Conference on International Computing Education Research, ICER ’18, Association for Computing Machinery, 2018\.
  * [12] C. Frädrich, F. Obermüller, N. Körber, U. Heuer, and G. Fraser, “Common bugs in scratch programs,” in Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education, ITiCSE ’20, Association for Computing Machinery, 2020.
  * [13] J. Sorva, Misconceptions and the Beginner Programmer.  2018\.
  * [14] J. Jansen, A. Oprescu, and M. Bruntink, “The impact of automated code quality feedback in programming education,” in Post-proceedings of the Tenth Seminar on Advanced Techniques and Tools for Software Evolution (SATToSE), 2017\.
  * [15] S. Krusche and A. Seitz, “Artemis: An automatic assessment management system for interactive learning,” in Proceedings of the 49th ACM Technical Symposium on Computer Science Education, SIGCSE 2018, ACM, 2018.
  * [16] A. Collins and M. Kapur, Cognitive Apprenticeship, p. 109–127.  Cambridge Handbooks in Psychology, Cambridge University Press, 2 ed., 2014\.
  * [17] T. Michaeli and R. Romeike, “Current status and perspectives of debugging in the k12 classroom: A qualitative study,” in 2019 IEEE Global Engineering Education Conference (EDUCON), IEEE, 2019.
  * [18] A. Yadav, S. Gretter, S. Hambrusch, and P. Sands, “Expanding computer science education in schools: understanding teacher experiences and challenges,” Computer Science Education, 2016.
  * [19] S. Sentance and A. Csizmadia, “Computing in the curriculum: Challenges and strategies from a teacher’s perspective,” Education and Information Technologies, 2017.
  * [20] S. DePasque and E. Tricomi, “Effects of intrinsic motivation on feedback processing during learning,” NeuroImage, 2015.
  * [21] V. J. Marin, T. Pereira, S. Sridharan, and C. R. Rivero, “Automated personalized feedback in introductory java programming moocs,” in 2017 IEEE 33rd International Conference on Data Engineering (ICDE), 2017.
  * [22] S. Arai, K. Sakamoto, H. Washizaki, and Y. Fukazawa, “A gamified tool for motivating developers to remove warnings of bug pattern tools,” in 2014 6th International Workshop on Empirical Software Engineering in Practice, 2014\.
  * [23] Z. Zhan, L. He, Y. Tong, X. Liang, S. Guo, and X. Lan, “The effectiveness of gamification in programming education: Evidence from a meta-analysis,” Computers and Education: Artificial Intelligence, 2022.
  * [24] J. Maloney, M. Resnick, N. Rusk, B. Silverman, and E. Eastmond, “The scratch programming language and environment,” ACM Transactions on Computing Education (TOCE), 2010.
  * [25] J. Moreno-León, G. Robles, and M. Román-González, “Dr. scratch: Automatic analysis of scratch projects to assess and foster computational thinking,” RED-Revista de Educación a Distancia, 2015.
  * [26] G. Fraser, U. Heuer, N. Körber, F. Obermüller, and E. Wasmeier, “Litterbox: A linter for scratch programs,” in 43rd IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training, ICSE (SEET) 2021, IEEE, 2021.
  * [27] J. Börstler, H. Störrle, D. Toll, J. van Assema, R. Duran, S. Hooshangi, J. Jeuring, H. Keuning, C. Kleiner, and B. K. MacKellar, “”i know it when I see it”: Perceptions of code quality,” in Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education, ITiCSE 2017, 2017.
  * [28] F. Obermüller, L. Bloch, L. Greifenstein, U. Heuer, and G. Fraser, “Code perfumes: Reporting good code to encourage learners,” in WiPSCE ’21: The 16th Workshop in Primary and Secondary Computing Education, ACM, 2021.
  * [29] L. Porter and B. Simon, “Retaining nearly one-third more majors with a trio of instructional best practices in CS1,” in The 44th ACM Technical Symposium on Computer Science Education, SIGCSE 2013, ACM, 2013.
  * [30] D. Deveaux, P. Frison, and J. Jézéquel, “Increase software trustability with self-testable classes in java,” in 13th Australian Software Engineering Conference (ASWEC 2001), IEEE Computer Society, 2001\.
  * [31] SonarSource, “Unique rules to find bugs, vulnerabilities, security hotspots, and code smells in your java code.” https://rules.sonarsource.com/java/, 2024.  Accessed: 08.06.2024.
  * [32] J. J. Bloch, Effective Java, 3rd Edition.  The Java series … from the source, Addison-Wesley, 2017.
  * [33] H. Osman, M. Lungu, and O. Nierstrasz, “Mining frequent bug-fix code changes,” in 2014 Software Evolution Week - IEEE Conference on Software Maintenance, Reengineering, and Reverse Engineering, CSMR-WCRE 2014, IEEE Computer Society, 2014.
  * [34] J. K. Teto, R. Bearden, and D. C. Lo, “The impact of defensive programming on I/O cybersecurity attacks,” in Proceedings of the 2017 ACM Southeast Regional Conference, 2017.
  * [35] P. Weill-Tessier, A. L. Costache, and N. C. C. Brown, “Usage of the java language by novices over time: Implications for tool and language design,” in SIGCSE ’21: The 52nd ACM Technical Symposium on Computer Science Education, ACM, 2021.
  * [36] E. Daka and G. Fraser, “A survey on unit testing practices and problems,” in 25th IEEE International Symposium on Software Reliability Engineering, ISSRE 2014, 2014.
  * [37] R. Ramler, M. Moser, and J. Pichler, “Automated static analysis of unit test code,” in 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), IEEE, 2016.
  * [38] R. C. Martin, The clean coder: a code of conduct for professional programmers.  Pearson Education, 2011.
  * [39] A. Van Deursen, L. Moonen, A. Van Den Bergh, and G. Kok, “Refactoring test code,” in Proceedings of the 2nd international conference on extreme programming and flexible processes in software engineering (XP2001), Citeseer, 2001.
  * [40] R. Helm, R. E. Johnson, E. Gamma, and J. Vlissides, Design patterns: Elements of reusable object-oriented software.  Braille Jymico Incorporated Quebec, 2000.
  * [41] R. Fojtik, “Design patterns in the teaching of programming,” Procedia-Social and Behavioral Sciences, 2014.
  * [42] A. Blewitt, A. Bundy, and I. Stark, “Automatic verification of design patterns in java,” in 20th IEEE/ACM International Conference on Automated Software Engineering (ASE 2005), ACM, 2005.
  * [43] J. C. Paiva, J. P. Leal, and Álvaro Figueira, “Progpedia: Collection of source-code submitted to introductory programming assignments,” Data in Brief, 2023.
  * [44] J. B. Kruskal, “On the shortest spanning subtree of a graph and the traveling salesman problem,” Proceedings of the American Mathematical society, 1956\.
  * [45] I. Cohen, Y. Huang, J. Chen, J. Benesty, J. Benesty, J. Chen, Y. Huang, and I. Cohen, “Pearson correlation coefficient,” Noise reduction in speech processing, 2009.
  * [46] P. Straubinger and G. Fraser, “Gamifying a software testing course with continuous integration,” in 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, 2024.


# CrowdTruth 2.0: Quality Metrics for Crowdsourcing with Disagreement

Anca Dumitrache, Oana Inel111,
Lora Aroyo
Vrije Universiteit Amsterdam
{anca.dmtrch,oana.inel,
<EMAIL_ADDRESS>
&Benjamin Timmermans
CAS IBM Nederland
<EMAIL_ADDRESS>
&Chris Welty
Google Research, New York
<EMAIL_ADDRESS>
Equal contribution, authors listed alphabetically.

###### Abstract

Typically crowdsourcing-based approaches to gather annotated data use inter-
annotator agreement as a measure of quality. However, in many domains, there
is ambiguity in the data, as well as a multitude of perspectives of the
information examples. In this paper, we present ongoing work into the
CrowdTruth metrics, that capture and interpret inter-annotator disagreement in
crowdsourcing. The CrowdTruth metrics model the inter-dependency between the
three main components of a crowdsourcing system – worker, input data, and
annotation. The goal of the metrics is to capture the degree of ambiguity in
each of these three components. The metrics are available online at
https://github.com/CrowdTruth/CrowdTruth-core.

## Introduction

The process of gathering ground truth data through human annotation is a major
bottleneck in the use of information extraction methods. Crowdsourcing-based
approaches are gaining popularity in the attempt to solve the issues related
to volume of data and lack of annotators. Typically these practices use inter-
annotator agreement as a measure of quality. However, this assumption often
creates issues in practice. Previous experiments we performed (?) found that
inter-annotator disagreement is usually never captured, either because the
number of annotators is too small to capture the full diversity of opinion, or
because the crowd data is aggregated with metrics that enforce consensus, such
as majority vote. These practices create artificial data that is neither
general nor reflects the ambiguity inherent in the data.

To address these issues, we proposed the CrowdTruth (?) method for
crowdsourcing ground truth by harnessing inter-annotator disagreement. We
present an alternative approach for crowdsourcing ground truth data that,
instead of enforcing agreement between annotators, captures the ambiguity
inherent in semantic annotation through the use of disagreement-aware metrics
for aggregating crowdsourcing responses. In this paper, we introduce the
second version of CrowdTruth metrics – a set of metrics that capture and
interpret inter-annotator disagreement in crowdsourcing annotation tasks. As
opposed to the first version of the metrics, published in (?), the current
version models the inter-dependency between the three main components of a
crowdsourcing system – worker, input data, and annotation. This update is
based on the intuition that disagreement caused by low quality workers should
not be interpreted as the data being ambiguous, but also that ambiguous input
data should not be interpreted as due to the low quality of the workers.

This paper presents the definitions of the CrowdTruth metrics 2.0, together
with the theoretical motivations of the updates based on the previous version
1.0. The code of the implementation of the metrics is available on the
CrowdTruth Github.111https://github.com/CrowdTruth/CrowdTruth-core The 2.0
version of the metrics has already been applied successfully to a number of
use cases, e.g. semantic frame disambiguation (?), relation extraction from
sentences (?), topic relevance (?). In the future, we plan to continue the
validation of the metrics through evaluation over different annotation tasks,
comparing CrowdTruth approach with other disagreement-aware crowd aggregation
methods.

## CrowdTruth Methodology

Figure 1: Triangle of Disagreement Figure 2: Example closed and open tasks,
together with the vector representations of the crowd answers.

The CrowdTruth methodology consists of a set of quality metrics and best
practices to aggregate inter-annotator agreement such that ambiguity in the
task is preserved. The methodology uses the triangle of disagreement model
(based on the triangle reference (?)) to represent the crowdsourcing system
and its three main components – input media units, workers, and annotations
(Figure 1). The triangle model expresses how ambiguity in any of the corners
disseminates and influences the other components of the triangle. For example,
an unclear sentence or an ambiguous annotation scheme would cause more
disagreement between workers (?), and thus, both need to be accounted for when
measuring the quality of the workers.

The CrowdTruth methodology calculates quality metrics for workers, media units
and annotations. The novel contribution of version 2.0 is that the way how
ambiguity propagates between the three components of the crowdsourcing system
has been made explicit in the quality formulas of the components. So for
example, the quality of a worker is weighted by the quality of the media units
the worker has annotated, and the quality of the annotations in the task.

This section describes the two steps of the CrowdTruth methodology:

  1. 1.

formalizing the output from crowd tasks into annotation vectors;

  2. 2.

calculating quality scores over the annotation vectors using disagreement
metrics.

### Building the Annotation Vectors

In order to measure the quality of the crowdsourced data, we need to formalize
crowd annotations into a vector space representation. For _closed tasks_ , the
annotation vector contains the given answer options in the task template,
which the crowd can choose from. For example, the template of a _closed task_
can be composed of a multiple choice question, which appears as a list
checkboxes or radio buttons, thus, having a finite list of options to choose
from. Figure 2 shows an example of a closed and an open task, indicating also
what the media units and annotations are for both cases.

While for _closed tasks_ the number of elements in the annotation vector is
known in advance, for _open-ended tasks_ the number of elements in the
annotation vector can only be determined when all the judgments for a media
unit have been gathered. An example of such a task can be highlighting words
or word phrases in a sentence, or as an input text field where the workers can
introduce keywords. In this case the answer space is composed of all the
unique keywords from all the workers that solved that media unit. As a
consequence, all the media units in a closed task have the same answers space,
while for open-ended tasks the answer space is different across all the media
units. Although the answer space for open-ended tasks is not known from the
beginning, it still can be further processed in a finite answer space.

In the annotation vector, each answer option is a boolean value, showing
whether the worker annotated that answer or not. This allows the annotations
of each worker on a given media unit to be aggregated, resulting in a media
unit vector that represents for each option how often it was annotated. Figure
2 shows how the worker and media unit vectors are formed for both a closed and
an open task.

### Disagreement Metrics

Using the vector representations, we calculate three core metrics that capture
the media unit quality, worker quality and annotation quality. These metrics
are mutually dependent (e.g. the media unit quality is weighted by the
annotation quality and worker quality), based on the idea from the triangle of
disagreement that ambiguity in any of the corners disseminates and influences
the other components of the triangle. The mutual dependence requires an
iterative dynamic programming approach, calculating the metrics in a loop
until convergence is reached. All the metrics have scores in the $[0,1]$
interval, with $0$ meaning low quality and $1$ meaning high quality. Before
starting the iterative dynamic programming approach, the quality metrics are
initialized with $1$.

To define the CrowdTruth metrics, we introduce the following notation:

  * •

$workers(u):$ all workers that annotate media unit $u$;

  * •

$units(i):$ all input media units annotated by worker $i$;

  * •

$WorkVec(i,u):$ annotations of worker $i$ on media unit $u$ as a binary
vector;

  * •

$MediaUnitVec(s)=\sum_{i\in workers(s)}WorkVec(i,s)$, where $s$ is an input
media unit.

To calculate agreement between 2 workers on the same media unit, we compute
the cosine similarity over the 2 worker vectors. In order to reflect the
dependency of the agreement on the degree of clarity of the annotations, we
compute $Wcos$, the weighted version of the cosine similarity. The Annotation
Quality Score (AQS), which will be described in more detail at the end of the
section, is used as the weight. For open-ended tasks, where annotation quality
cannot be calculated across multiple media units, we consider annotation
quality equal to 1 (the maximum value) in all cases. Given 2 worker vectors,
$vec_{1}$ and $vec_{2}$ on the same media unit, the formula for the weighted
cosine score is:

$\displaystyle Wcos(vec_{1},vec_{2})=$
$\displaystyle=\dfrac{\sum_{a}vec_{1}(a)\;vec_{2}(a)\;AQS(a)}{\sqrt{(\sum_{a}vec_{1}^{2}(a)\;AQS(a))\;(\sum_{a}vec_{2}^{2}(a)\;AQS(a))}},$
$\forall a\text{ - annotation}.$

The Media Unit Quality Score (UQS) expresses the overall worker agreement over
one media unit. Given an input media unit $u$, $UQS(u)$ is computed as the
average cosine similarity between all worker vectors, weighted by the worker
quality ($WQS$) and annotation quality ($AQS$). Through the weighted average,
workers and annotations with lower quality will have less of an impact on the
final score. The formula used in its calculation is:

$UQS(u)=\dfrac{\sum\limits_{i,j}WorkVecWcos(i,j,u)\;WQS(i)\;WQS(j)}{\sum\limits_{i,j}WQS(i)\;WQS(j)},$
$\displaystyle WorkVecWcos(i,j,u)=Wcos($ $\displaystyle WorkVec(i,u),$
$\displaystyle WorkVec(j,u)),$ $\forall i,j\in workers(u),i\neq j.$

The Worker Quality Score (WQS) measures the overall agreement of one crowd
worker with the other workers. Given a worker $i$, $WQS(i)$ is the product of
2 separate metrics - the worker-worker agreement $WWA(i)$ and the worker-media
unit agreement $WUA(i)$:

$WQS(i)=WUA(i)\;WWA(i).$

The Worker-Worker Agreement (WWA) for a given worker $i$ measures the average
pairwise agreement between $i$ and all other workers, across all media units
they annotated in common, indicating how close a worker performs compared to
workers solving the same task. The metric gives an indication as to whether
there are consistently like-minded workers. This is useful for identifying
communities of thought. $WWA(i)$ is the average cosine distance between the
annotations of a worker $i$ and all other workers that have worked on the same
media units as worker $i$, weighted by the worker and annotation qualities.
Through the weighted average, workers and annotations with lower quality will
have less of an impact on the final score of the given worker.

$\displaystyle WWA(i)=$
$\displaystyle\dfrac{\sum\limits_{j,u}WorkVecWcos(i,j,u)\;WQS(j)\;UQS(u)}{\sum\limits_{j,u}WQS(j)\;UQS(u)},$
$\displaystyle\forall j\in workers(u\in units(i)),i\neq j.$

The Worker-Media Unit Agreement (WUA) measures the similarity between the
annotations of a worker and the aggregated annotations of the rest of the
workers. In contrast to the $WWA$ which calculates agreement with individual
workers, $WUA$ calculates the agreement with the consensus over all workers.
$WUA(i)$ is the average cosine distance between the annotations of a worker
$i$ and all annotations for the media units they have worked on, weighted by
the media unit ($UQS$) and annotation quality ($AQS$). Through the weighted
average, media units and annotations with lower quality will have less of an
impact on the final score.

$WUA(i)=\dfrac{\sum\limits_{u\in
units(i)}WorkUnitWcos(u,i)\;UQS(u)}{\sum\limits_{u\in units(i)}UQS(u)},$
$\displaystyle WorkUnitWcos(u,i)$ $\displaystyle=Wcos(WorkVec(i,u),$
$\displaystyle MediaUnitVec(u)-WorkVec(i,u))$

The Annotation Quality Score (AQS) measures the agreement over an annotation
in all media units that it appears. Therefore, it is only applicable to closed
tasks, where the same annotation set is used for all input media units. It is
based on $P_{a}(i|j)$, the probability that if a worker $j$ annotates $a$ in a
media unit, worker $i$ will also annotate it.

$\displaystyle
P_{a}(i|j)=\frac{\sum\limits_{u}UQS(u)\;WorkVec(i,s)[a]\;WorkVec(j,s)[a]}{\sum\limits_{u}UQS(u)\;WorkVec(j,u)(r)},$
$\displaystyle\forall u\in units(i)\cap units(j).$

Given an annotation $a$, $AQS(a)$ is the weighted average of $P_{a}(i|j)$ for
all possible pairs of workers $i$ and $j$. Through the weighted average, input
media units and workers with lower quality will have less of an impact on the
final score of the annotation.

$\displaystyle AQS(a)$
$\displaystyle=\dfrac{\sum\limits_{i,j}WQS(i)\;WQS(j)\;P_{a}(i|j)}{\sum\limits_{i,j}WQS(i)\;WQS(j)},$
$\displaystyle\forall i,j\text{ workers, }i\neq j.$

The formulas for media unit, worker and annotation quality are all mutually
dependent. To calculate them, we apply an iterative dynamic programming
approach. First, we initialize each quality metric with the score for maximum
quality (i.e. equal to 1). Then we repeatedly re-calculate the quality metrics
until each of the values are stabilized. This is assessed by calculating the
sum of variations between iterations for all quality values, and checking
until it drops under a set threshold $t$.

The final metric we calculate is the Media Unit - Annotation Score (UAS) – the
degree of clarity with which an annotation is expressed in a unit. Given an
annotation $a$ and a media unit $u$, $UAS(u,a)$ is the ratio of the number of
workers that picked annotation $u$ over all workers that annotated the unit,
weighted by the worker quality.

$UAS(u,a)=\dfrac{\sum\limits_{i\in
workers(u)}WorkVec(i,u)(a)\;WQS(i)}{\sum\limits_{i\in workers(u)}WQS(i)}.$

## Conclusion

In this paper, we present ongoing work into the CrowdTruth metrics, that
capture and interpret inter-annotator disagreement in crowdsourcing. Typically
crowdsourcing-based approaches to gather annotated data use inter-annotator
agreement as a measure of quality. However, in many domains, there is
ambiguity in the data, as well as a multitude of perspectives of the
information examples. The CrowdTruth metrics model the inter-dependency
between the three main components of a crowdsourcing system – worker, input
data, and annotation.

We have presented the definitions and formulas of several CrowdTruth metrics,
including the three core metrics measuring the quality of workers,
annotations, and input media units. The metrics are based on the idea of the
triangle of disagreement, expressing how ambiguity in any of the corners
disseminates and influences the other components of the triangle. Because of
this, disagreement caused by low quality workers should not be interpreted as
the data being ambiguous, but also that ambiguous input data should not be
interpreted as due to the low quality of the workers. The metrics have already
been applied successfully to use cases in topic relevance (?), semantic frame
disambiguation (?) and relation extraction from sentences (?).

## References

  * [Aroyo and Welty 2013] Aroyo, L., and Welty, C.  2013\.  Crowd Truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard.  Web Science 2013. ACM.
  * [Aroyo and Welty 2014] Aroyo, L., and Welty, C.  2014\.  The Three Sides of CrowdTruth.  Journal of Human Computation 1:31–34.
  * [Aroyo and Welty 2015] Aroyo, L., and Welty, C.  2015\.  Truth Is a Lie: CrowdTruth and the Seven Myths of Human Annotation.  AI Magazine 36(1):15–24.
  * [Dumitrache, Aroyo, and Welty 2017] Dumitrache, A.; Aroyo, L.; and Welty, C.  2017\.  False positive and cross-relation signals in distant supervision data.
  * [Dumitrache, Aroyo, and Welty 2018] Dumitrache, A.; Aroyo, L.; and Welty, C.  2018\.  Capturing ambiguity in crowdsourcing frame disambiguation.
  * [Inel et al. 2014] Inel, O.; Khamkham, K.; Cristea, T.; Dumitrache, A.; Rutjes, A.; van der Ploeg, J.; Romaszko, L.; Aroyo, L.; and Sips, R.-J.  2014\.  Crowdtruth: Machine-human computation framework for sing disagreement in gathering annotated data.  In The Semantic Web–ISWC 2014. Springer.  486–504.
  * [Inel et al. 2018] Inel, O.; Li, D.; Haralabopoulos, G.; Van Gysel, C.; Szlávik, Z.; Simperl, E.; Kanoulas, E.; and Aroyo, L.  2018\.  Studying topical relevance with evidence-based crowdsourcing.  In To Appear in the Proceedings of the 27th ACM International Conference on Information and Knowledge Management.  ACM.
  * [Knowlton 1966] Knowlton, J. Q.  1966\.  On the definition of “picture”.  AV Communication Review 14(2):157–183.