# Fluorescent Neuronal Cells v2: Multi-Task, Multi-Format Annotations for Deep
Learning in Microscopy

Luca Clissa  National Institute of Nuclear Physics, Bologna, Italy  Antonio
Macaluso  German Research Center for Artificial Intelligence (DFKI), Agents
and Simulated Reality Department, Saarbruecken, Germany  Roberto Morelli
University of Bologna, Department of Physics and Astronomy, Bologna, Italy
Alessandra Occhinegro  University of Bologna, Department of Biomedical and
Neuromotor Sciences, Bologna, Italy  Emiliana Piscitiello  University of
Bologna, Department of Biomedical and Neuromotor Sciences, Bologna, Italy
Ludovico Taddei  University of Bologna, Department of Biomedical and
Neuromotor Sciences, Bologna, Italy  Marco Luppi  University of Bologna,
Department of Biomedical and Neuromotor Sciences, Bologna, Italy  Roberto
Amici  University of Bologna, Department of Biomedical and Neuromotor
Sciences, Bologna, Italy  Matteo Cerri  University of Bologna, Department of
Biomedical and Neuromotor Sciences, Bologna, Italy  Timna Hitrec  University
of Bologna, Department of Biomedical and Neuromotor Sciences, Bologna, Italy
Lorenzo Rinaldi  National Institute of Nuclear Physics, Bologna, Italy
University of Bologna, Department of Physics and Astronomy, Bologna, Italy
Antonio Zoccoli  National Institute of Nuclear Physics, Bologna, Italy
University of Bologna, Department of Physics and Astronomy, Bologna, Italy

###### Abstract

Fluorescent Neuronal Cells v2 is a collection of fluorescence microscopy
images and the corresponding ground-truth annotations, designed to foster
innovative research in the domains of Life Sciences and Deep Learning. This
dataset encompasses three image collections in which rodent neuronal cells’
nuclei and cytoplasm are stained with diverse markers to highlight their
anatomical or functional characteristics. Alongside the images, we provide
ground-truth annotations for several learning tasks, including semantic
segmentation, object detection, and counting. The contribution is two-fold.
First, given the variety of annotations and their accessible formats, we
envision our work facilitating methodological advancements in computer vision
approaches for segmentation, detection, feature learning, unsupervised and
self-supervised learning, transfer learning, and related areas. Second, by
enabling extensive exploration and benchmarking, we hope Fluorescent Neuronal
Cells v2 will catalyze breakthroughs in fluorescence microscopy analysis and
promote cutting-edge discoveries in life sciences. The data are available at:
https://amsacta.unibo.it/id/eprint/7347.

## Background & Summary

Fluorescence microscopy is as a pivotal imaging technique in life-science
experiments, allowing researchers to study biological structures or processes
with remarkable precision. It employs fluorescent dyes or proteins that emit
light at specific wavelengths depending on the illuminating wavelength they
absorb. Exploiting this phenomenon, specific molecules can be tagged (stained)
with fluorescent markers, and visualized by filtering only their emitted
light, thus providing valuable insights into their localization, activity, and
interactions.

Despite its widespread use, current practices in fluorescence microscopy
analysis heavily rely on semi-automatic procedures, often necessitating manual
recognition and/or counting of specific neuronal structures of interest [1, 2,
3]. For instance, in the study of torpor mechanisms, researchers depend on
laborious hand-crafted operations to identify neuronal networks associated
with this process [4]. This manual aspect typically delays the analyses, also
introducing potential errors due to limitations of human operators. Moreover,
the similarity between structures of interest and the background often leads
to challenges in distinguishing and accurately recognizing biological
compounds, resulting in inherent arbitrariness and interpretation bias.

For these reasons, there is a growing interest in automating the recognition
and counting of tagged elements in fluorescence microscopy[5, 6, 7, 8]. Deep
learning approaches have demonstrated great promise in various object
recognition tasks. However, their performance can deteriorate when applied to
data from domains significantly different from those adopted for pre-training
(domain shift[9, 10]). Furthermore, the effectiveness of these approaches
typically relies heavily on the availability of well-annotated data[11], which
is often scarce and limited in the fluorescence microscopy domain.

To mitigate these issues, we present the Fluorescent Neuronal Cells v2 (FNC)
dataset. This archive features 3 data collections, for a total of 1874 high-
resolution images of rodents brain slices capturing a diverse range of
neuronal structures and staining patterns. To facilitate research in this
field, we also provide 750 annotations in various formats, tailored to popular
supervised learning tasks such as semantic segmentation, object detection, and
counting. Apart from serving as an additional benchmark for testing model
generalization in microscopy applications, the FNC dataset opens up several
research opportunities. Firstly, the heterogeneity of biological structures
and their visual characteristics enable testing the generalization of trained
models, and validating transfer learning and domain adaptation methods [12,
13]. Also, the availability of multiple annotation types allows the
exploration of different learning paradigms, ranging from supervised and
unsupervised approaches to self-/weakly-supervised techniques. Moreover, the
specific challenges of our data well suit investigations into methodological
advancements, e.g., assessing the effectiveness of different annotation
formats and uncertainty estimation.

The design of the data collection process involved two distinct stages.
Firstly, data collection was conducted following standardized experimental
protocols. Specifically, controlled experimental conditions were applied to
the animals, whose brains were sliced and processed by a classical
immunofluorescence protocol to stain various neuronal substructures.
Subsequently, a fluorescence microscopy was employed to capture high-
resolution images of the areas of interest. Secondly, domain experts performed
data annotation providing ground-truth labels necessary for supervised
learning.

Figure 1: Study design. The study was designed in two phases: data collection,
where high-resolution pictures of rodent brain slices were acquired; and data
annotation, where expert researchers collected annotations needed for
supervised learning approaches.

Despite the presence of open-source fluorescence microscopy datasets, several
issues hinder their utilization for training deep learning models. Firstly,
these collections typically lack accompanying ground-truth annotations, thus
precluding the adoption of supervised learning techniques. Secondly, labelled
datasets often include just a few dozens of images[14, 15], that can be
restrictive considering the data-intensive nature of deep learning models.
Also, the moderate resolution of images in open datasets[16, 17] hampers the
effectiveness of resorting to crops as an alternative to whole images for
augmenting sample size. Thirdly, most existing datasets predominantly include
a single marker type[14, 15], thus lacking diversity and limiting robust model
training. Alongside these aspects, public datasets typically provide label
types as dot-annotations or bounding boxes[14, 15, 16, 17], which prevents
their extension to fine-grained segmentation tasks. Additionally, the data
accessibility is sometimes restricted due to the use of domain-specific
formats[10], which complicates integration into deep learning frameworks and
wide dissemination.

In response to these challenges, we present a large archive comprising high-
resolution fluorescent microscopy images, encompassing different markers and
cell types. Furthermore, the data are shared as easily accessible PNG files,
and the corresponding annotations are provided in various types and formats,
enabling the exploration of different learning approaches and tasks, thereby
significantly expanding the scope of potential applications.

## Methods

The FNC dataset compiles images acquired from multiple studies and
experimental conditions, while maintaining a consistent structure in the
acquisition pipeline. Minor modifications were made to accommodate the
specific requirements of each study and adapt to the current experimental
circumstances and equipment. The data collection process consisted of two
distinct and independent stages: image acquisition and data annotation. This
section provides a comprehensive description of the data acquisition design,
including the dedicated measures implemented for each image collection (refer
to Figure 1 for a visual summary).

(a) Green image example

(b) Red image example

(c) Yellow image example

(d) Green mask example

(e) Red mask example

(f) Yellow mask example

Figure 2: Data preview. The figures show examples of fluorescence micrscopy
pictures (Figures 2(a), 2(b) and 2(c)) and the corresponding ground-truth
binary masks (Figures 2(d), 2(e) and 2(f)).

### Image acquisition

In the image acquisition phase, a total of 68 rodents were subjected to
controlled experimental conditions to study torpor and thermoregulatory
mechanisms. At the end of the experimental session, the animals were deeply
anaesthetized and transcardially perfused with 4% formaldehyde[4]. This
process allowed for the tagging of several neuronal substructures located
within the nucleus or cytoplasm of the neurons.

Rodents brains were then sectioned into 35 $\mu m$ thick tissue slices, with
sampling conducted at regular intervals (105 $\mu m$ for mice and of 210 $\mu
m$ for rats) to avoid redundant data and ensure comprehensive coverage while
maintaining manageable data size. Brain slices were finally stained for
distinct markers following a standard immunofluorescence protocol [4]. Only
some areas of interest were observed, namely the Raphe Pallidus (RPa),
Dorsomedial Hypothalamus (DMH), Lateral Hypotalamus (LH), and Ventrolateral
Periaqueductal Gray (VLPAG). These specific brain regions were chosen based on
their relevance to the study of torpor mechanisms. The resulting specimens
were observed by means of a fluorescence microscope equipped with a high-
resolution camera. A specific wavelength of excitation light was selected for
each collection based on the excitation wavelength of the chosen marker,
resulting in pictures acquired with the application of green, yellow/orange or
red filters. For simplicity, the image collections are named according to
their prevalent hue. The original images were acquired as either TIF or JPG
files depending on the camera default settings. To ensure traceability, a file
naming convention was adopted to indicate their respective sample
origins111<animal_id>_S<sample_id>C<column_id>R<row_id>_<brain_area>_<zoom>_<collection_id>.
During the analysis phase, the raw data were converted to uncompressed PNG
format, taking care to preserve the extensive set of associated metadata. This
conversion aimed to enhance accessibility and facilitate broader utilization
of the data, allowing for inspection and manipulation without the need for
specialized software. Consequently, the FNC archive includes both these
derived images and the original raw images, which are retained for data
recovery and reproducibility purposes.

#### Green and Yellow collections

The images within these collections were obtained during the same experiment
[4], in which brain sections from C57BL/6J mice were stained with two markers
to highlight specific substructures present in the neurons’ nucleus and
cytoplasm. The resulting brain slices were then observed using a Nikon Eclipse
80i microscope, equipped with a Nikon Digital Sight DS-Vi1 color camera, at a
magnification of 200x.

More specifically, the green collection corresponds to cFOS staining (cf.
Figure 2(d)). This staining method was employed to emphasize the nuclei of
active neuronal cells[18], enabling the topographic analysis of brain areas
that exhibit neuronal activity under specific experimental conditions. This
approach is widely employed to identify neuronal cells responsible for
regulating specific physiological phenomena.

In contrast, the yellow collection (cf. Figure 2(c)) utilized staining for the
b-subunit of Cholera Toxin (CTb). This monosynaptic retrograde neuronal tracer
migrates within the soma and axons of neuronal cells projecting to the brain
area where CTb was previously injected during in vivo experiments[19].
Consequently, this staining technique facilitates the identification of
morphological connections between different brain regions.

#### Red collection

The red collection comprises images obtained from multiple unpublished
experiments, concerning specimens of both mice and rats (cf. Figure 2(b)).
Despite sharing the same experimental setup as green and yellow collections,
this time the brain tissues were stained for various elements to
phenotypically characterize the cells involved in the neural circuits
underlying the physiological phenomena of torpor and thermoregulation.
Specifically, slices were stained for orexin, tryptophan hydroxylase, and
tyrosine hydroxylase. In this case, image acquisition was conducted using both
the aforementioned Nikon Eclipse 80i microscope and an ausJENA JENAVAL
microscope, equipped with a Nikon Coolpix E4500 color camera, at a
magnification of 250x. For further details, please refer to the accompanying
metadata for each image.

### Data annotation

The data annotation process was carried out by multiple proficient
experimenters according to a fixed annotation protocol222check Annotations
protocol.pdf inside Annotations.zip in the data archive, with multiple
revision rounds to ensure data quality and minimize operator bias. We adopted
the Visual Geometry Group Visual Image Annotator (VIA) annotation tool[20,
21], which employs a web interface for image visualization and allows for the
overlaying of annotations in different forms. In our study, the tagging
process involved creating polygon contours, and the resulting annotations were
exported into CSV format. To generate the binary masks required for training,
the polygon contours were transformed using programming libraries such as
OpenCV and scikit-image. For the yellow collection, we utilized the binary
masks available from version 1[22] as pre-annotations. Specifically, we
employed erosions and dilations techniques to address fragmented contours
resulting from semi-automatic labeling based on thresholding. Furthermore, we
applied methods to fill small holes within segmented objects, and removed
spurious objects that went overlooked in the previous annotations or were
erroneously added by prior processing. Subsequently, these pre-annotations
were refined manually using VIA, enhancing their accuracy and ensuring better
consistency across the annotations (see Figure 3). In contrast, the green and
red collections were annotated from scratch.

(a) v2

(b) v1

(c) v2

(d) v1

Figure 3: Yellow masks v1 review. Figures 3(a) and 3(c) illustrate how binary
masks were reviewed compared to version 1, respectively Figures 3(b) and 3(d).
Improvements include: small objects removal, contour smoothing, holes filling
and more consistent labelling.

Upon completion of the labeling process, the polygon contours exported from
VIA were also converted into multiple annotation types and formats. This
conversion aims at facilitating accessibility for a wide range of users and
promoting the exploration of various learning problems related to our data.
For a more comprehensive understanding of the available formats and annotation
types, please refer to the Section Data Records.

## Data Records

The FNC dataset is a collection of 1874 high-resolution fluorescent microscopy
pictures, 750 of which also have their corresponding ground-truth segmentation
masks, while the remaining 1124 are unlabelled. It is hosted on AMS
Acta333available at: https://doi.org/10.6092/unibo/amsacta/7347 (link will be
active after acceptance), the open access repository managed by the University
of Bologna. The data are organized into three standalone image collections,
named for simplicity green, yellow, and red, each available under the
corresponding folder (see Figure 4(c)). The collections share a common layout
to facilitate easy access and analysis (see Figure 4(a)).

To aid users in navigating the archive, the metadata_v2.xlsx file provides a
comprehensive overview of the FNC data collection. It includes high-level
metadata for each image, such as the corresponding animal, acquisition
details, data partition, and annotation information.

.1 <collection>. .2 test. .3 ground_truths. .3 images. .3 metadata. .2
trainval. .3 ground_truths. .3 images. .3 metadata. .2 unlabelled. .3 images.
.3 metadata.

.1 ground_truths. .2 COCO. .2 masks. .2 Pascal_VOC. .2 rle. .2 VIA.

(a) Collection folder structure

(b) Annotations folder structure

(c) Dataset composition

Figure 4: FNC dataset structure. Figure 4(a) shows the structure of each image
collection folder, while Figure 4(b) gives more details on the organization of
the annotations directory. Figure 4(c) summarizes the composition of each
image collection, with the amounts of training, testing and unlabelled images.

### Image collection folder structure

The trainval and test folders contain all labelled images for each collection.
These data partitions were obtained through a random 75%/25% split and are
recommended as a suggested configuration to ensure reproducibility and
comparability in future studies. The remaining images were collected under the
unlabelled folder.

Inside each data partition folder, the images folder contains fluorescence
microscopy images in PNG format. All the images are accompanied by a rich set
of metadata, stored both in their EXIF tags and as a separate TXT file under
the metadata folder. The ground_truths folder contains annotations in various
formats commonly used within the machine learning community (see Figure 4(b)).

### Annotation types and formats

The FNC collection provides annotations of multiple types, encoded in several
standard formats. In the masks folder, we find the binary masks typically used
for segmentation tasks (cf. Figures 2(d), 2(e) and 2(f)). The correspondence
between the masks and the respective images can be established based on the
filenames. The other folders store a light-weight encoding of the binary
masks, enriched with additional annotation types/formats.

The rle directory contains Running Length Encoding (RLE) of the binary masks,
stored as pickle files. This encoding is a compressed representation that can
effectively save disk space while preserving the complete segmentation
information. It is particularly convenient for high-resolution images like
those present in our dataset.

The other directories provide several annotation types, and they are named
after their annotation format. Polygon annotations are available in each of
the VIA, COCO and Pascal_VOC directories, in the form of json or xml files.
COCO[23] and Pascal_VOC[24] formats also features bounding boxes and dot
annotations for object detection tasks, and count labels for object counting.

## Technical Validation

In order to demonstrate the potential for successful model training and
analysis using the provided dataset, we conducted three types of checks to
ensure the accuracy and quality of the annotations.

Firstly, polygon annotations were obtained by experienced researchers. During
this phase, the annotations underwent multiple rounds of double-checking to
ensure that the polygons did not have intersecting edges and that they
accurately represented the objects when transformed into binary masks.

Secondly, we leveraged domain knowledge to validate the annotations.
Precisely, we tested the binary masks against our expectations regarding the
sizes and shapes of the biological structures involved. This validation
process relies on a quantitative evaluation concerning objects’ area and
diameter, complemented by a visual scrutiny of the masks to ensure they align
with the expected shapes and exhibit smooth contours.

| signal % |  | cell count |  | area (${\mathrm{\SIUnitSymbolMicro m}}^{2}$) |  | Feret diameter ($\mathrm{\SIUnitSymbolMicro m}$) |  | equivalent diameter ($\mathrm{\SIUnitSymbolMicro m}$)
---|---|---|---|---|---|---|---|---|---
collection | green | red | yellow |  | green | red | yellow |  | green | red | yellow |  | green | red | yellow |  | green | red | yellow
count | 283 | 184 | 283 |  | 4606 a | 4486 a | 2659 a |  | 4600 | 4483 | 2621 |  | 4600 | 4483 | 2621 |  | 4600 | 4483 | 2621
mean | $0.64$ | $2.61$ | $0.65$ |  | $28.65$ | $34.96$ | $25.76$ |  | $74.78$ | $246.52$ | $132.55$ |  | $12.20$ | $25.68$ | $17.24$ |  | $9.64$ | $17.13$ | $12.63$
std | $0.57$ | $1.74$ | $0.80$ |  | $17.49$ | $17.90$ | $19.88$ |  | $23.39$ | $130.24$ | $66.02$ |  | $1.92$ | $8.86$ | $4.86$ |  | $1.48$ | $4.50$ | $3.04$
min | $0$ | $0$ | $0$ |  | $0$ | $0$ | $0$ |  | $12.41$ | $24.81$ | $17.62$ |  | $5.48$ | $7.08$ | $6.09$ |  | $3.97$ | $5.62$ | $4.74$
10% | $0.10$ | $0.37$ | $0$ |  | $9$ | $14$ | $4$ |  | $48.74$ | $106.72$ | $66.16$ |  | $9.85$ | $15.67$ | $11.93$ |  | $7.88$ | $11.66$ | $9.18$
25% | $0.25$ | $1.34$ | $0.13$ |  | $15$ | $21$ | $9$ |  | $58.58$ | $144.04$ | $88.02$ |  | $10.88$ | $18.64$ | $13.90$ |  | $8.64$ | $13.54$ | $10.59$
50% | $0.45$ | $2.34$ | $0.40$ |  | $25$ | $31$ | $19$ |  | $71.67$ | $218.80$ | $117.75$ |  | $12.07$ | $24.38$ | $16.51$ |  | $9.55$ | $16.69$ | $12.24$
75% | $0.92$ | $3.53$ | $0.82$ |  | $39.75$ | $50$ | $44$ |  | $87.33$ | $320.75$ | $161.17$ |  | $13.35$ | $30.99$ | $19.76$ |  | $10.54$ | $20.21$ | $14.33$
90% | $1.38$ | $4.89$ | $1.49$ |  | $56$ | $58$ | $57$ |  | $104.66$ | $422.64$ | $218.27$ |  | $14.70$ | $37.64$ | $23.59$ |  | $11.54$ | $23.20$ | $16.67$
max | $3.36$ | $8.40$ | $5.54$ |  | $69$ | $90$ | $72$ |  | $227.04$ | $842.66$ | $548.20$ |  | $21.21$ | $74.39$ | $41.66$ |  | $17$ | $32.76$ | $26.42$

  * a

The difference compared to counts in following columns comes from the
contribution of empty images. These amount to 6, 3 and 38 images for green,
red and yellow collections, respectively.

Table 1: Summary statistics of key features’ distribution for each image
collection. The top portion highlights global indicators, while the bottom one
reports given percentiles of each distribution.

Table 1 reports summary statistics for the distribution of key features at the
image and object levels, that can be leveraged for technical validation. For
instance, the annotated objects display an average area of nearly 75, 247, and
133 $\mu m^{2}$ for green, red, and yellow cells, respectively. These values
align with the expected size of the biological structures represented in each
image collection. Additionally, the analysis of Feret and equivalent diameters
provides an understanding of the typical form of the stained objects. In
particular, the Feret diameter[25] can be interpreted as a measure of the
maximum extension of an object, whereas the equivalent diameter represents the
diameter the object would have if it were a perfect circle with the same area.
Thus, comparing these two metrics can offer insight into the objects’ shape
regularity. For green cells, the values for the two measurements are
relatively close (roughly 12 VS 10 $\mu m$), suggesting that these cells are
broadly circular or oval in shape. A similar conclusion can be drawn for the
yellow stains, albeit with slightly more variability (approximately 17 and 13
$\mu m$), indicating generally regular shapes with occasional deviations. In
the case of red objects, instead, the comparison is markedly different. This
time we observe a Feret diameter around 26 $\mu m$ against an equivalent
diameter of 17, which suggests that these stains are typically elongated in
one direction rather than concentrated around a center of mass. All these
observations are also corroborated when visually inspecting annotated cells,
which confirms prior expectations about objects size and shapes based on the
nature of the marked structures.

### Learning

Thirdly, we conducted a sample training phase for each image collection using
a cell ResUnet architecture[5], specifically designed for this type of
application. Specifically, we trained a network from scratch for each
collection using a Dice Loss[26] and the Adam optimizer[27]. The initial
learning rate was set based on the “learning rate test"[28] implemented by
fastai’s[29] lr_find() method. The training phase continued for 100 epochs
with cyclical learning rates[30, 31], and the best model was selected based on
the best validation dice coefficient. For all technical details please refer
to the GitHub repository444available at: https://github.com/clissa/fluocells-
scientific-data (link will be active after acceptance). This training phase
aims to verify the effectiveness of the data in facilitating the learning of
beneficial cell features. Additionally, the intent is to highlight the
relevant metrics for result evaluations. In particular, we suggest performance
should be assessed differently depending on the end goal of future analyses.

For segmentation tasks, we provided an implementation where matching of actual
and predicted neurons555by this we intend the calculation of True Positives
(TP), False Positives (FP) and False Negatives (FN) is done based on their
overlap, quantified as Intersection-over-Union (IoU)[32]. This approach not
only ensures a 1-1 correspondence of true and predicted objects but also
assesses how closely the predictions reconstruct the shape of ground-truth
cells. Building on top of this definition, standard metrics such as precision,
recall and $F_{1}$ score can be computed as measures of global performance.

Detection tasks, on the other hand, would benefit from a looser matching
criterion, comparing predicted and true objects’ centers instead of overlaps.
This approach prioritizes recognition over precise shape reconstruction.

Finally, for counting tasks we suggest common regression metrics such as Mean
Absolute Error (MAE), Median Absolute Error (MedAE) and Mean Percentage Error
(MPE).

| segmentation | detection | counting
---|---|---|---
metrics |  $F_{1}$ Score | Precision | Recall |  $F_{1}$ Score | Precision | Recall | MAE | MedAE | MPE (%)
Green | 0.69 | 0.79 | 0.62 | 0.77 | 0.86 | 0.69 | 1.20 | 1.00 | 9%
Red | 0.28 | 0.33 | 0.25 | 0.60 | 0.69 | 0.53 | 7.13 | 7.00 | 42%
Yellow | 0.65 | 0.67 | 0.63 | 0.77 | 0.78 | 0.75 | 1.54 | 1.00 | 18%
Table 2: Performance metrics by learning task. The segmentation portion refers
to TP, FP, and FN computed based on objects overlapping (IoU). For detection
metrics, predicted and true cells are associated based on their centers’
distance. Counting metrics simply consider the difference between predicted
and true objects.

Table 2 shows the results of the sample training for each image collection.
These results are not intended to be a comprehensive exploration of the
model’s capabilities on FNC data, but rather to showcase some characteristics
of various evaluation methods. Nonetheless, they may serve as a baseline for
future studies.

Despite no optimization of the training pipeline, the initial results are
mainly satisfactory (except for red segmentation), confirming the technical
robustness of the data collection process. Going into more details, we observe
a marked discrepancy between segmentation and detection metrics. As expected,
the $F_{1}$ scores based on the distance between true and predicted centers of
mass are significantly higher than the corresponding segmentation indicators.
Moreover, the discrepancy is greater for image collections where the objects
have more irregular shapes (green < yellow < red). This is a consequence of
the more inclusive matching criterion used for detection tasks.

In terms of counting, performance is already very satisfactory. However, these
metrics may not fully represent the model’s performance as good results could
arise due to a balancing effect between true positives and false negatives.
Interestingly, despite low absolute errors, the percentage error is relatively
high due to the impact of errors in images with few or no cells. To address
this issue, we adopt the following formula for MPE computation:
$\text{MPE}=\dfrac{\left(\text{predicted}-\text{true}\right)}{\max{\left(\text{true},1\right)}}$.
In this way, the fraction is not over-inflated when there are no cells in the
original mask.

## Usage Notes

The Fluorescent Neuronal Cells collection is available both as a comprehensive
archive and as individual image collections for specific research
requirements. This enables users to download the data efficiently and
selectively, based on their specific needs. The code provided is based on the
Python and PyTorch frameworks, offering a robust foundation for analysis and
modeling. However, thanks to the popularity of the annotation formats and the
use of PNG images, users can easily employ their preferred deep learning
framework.

### Peculiar traits

In all image collections, the visual representation is characterized by the
prevalence of two distinct color tones, which result from the deliberate
selection of a specific wavelength. One tone appears darker, indicating areas
where light has been filtered out, while the other tone is brighter and more
intense, emitted by the fluorophore corresponding to the color of each
collection (see Figures 2(a), 2(b) and 2(c)). As a result, the images can
generally be depicted using variations of a single color. Consequently, a 1-D
representation may be sufficient, or an alternative color space other than RGB
could provide more informative and less redundant data.

Notice, however, that the specific colors employed in our studies were
dictated not by any inherent or functional property of the stained biological
structures, but rather by their accessibility and practicality during the time
of the experiments. Therefore, it would be a misinterpretation to associate
specific colors to particular neuronal substructures. In fact, these colors
serve only as contrasting elements to discern the stained foreground objects
from the background. Consequently, the emphasis should lie primarily on
learning this discrimination rather than matching specific colors with the
neuronal structures. Thus, the particular colors should not be considered
indicative of the type of neuronal cells or their functional attributes, but
merely as a practical tool aiding in the overall visualization and
interpretation.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

Figure 5: Challenges. FNC data present several difficulties to take into
account during their analysis. Common challenges are represented by
overcrowding (Figures 5(a), 5(d), 5(f) and 5(g)), ambiguity (Figures 5(a),
5(d) and 5(f)), and artifacts (Figures 5(b), 5(c), 5(d), 5(e), 5(f) and 5(g)).

### Challenges

Some important insights for future studies can be drawn examining ground-truth
masks at the pixel level, revealing significant characteristics that impact
the training process.

The two classes, namely cells (1) and background (0), exhibit an extreme class
imbalance, with background pixels being overwhelmingly predominant, typically
exceeding cell pixels by over a factor of 100 (cf. Table 1, signal %). These
observations highlight the necessity for specialized training strategies to
address this pronounced class imbalance and effectively learn the pixel
classification.

Additional challenges are associated with the macroscopic content of the
images. The Fluorescent Neuronal Cells data showcase a diverse collection of
11704 subnuclear neuronal structures, varying in shape, size, and extension
(cf Table 1, area, Feret diameter and equivalent diameter columns). The
distribution of these structures across the collections is uneven, with some
images containing numerous cells while others are devoid of them.
Consequently, the model needs to be flexible enough to handle both scenarios.

Furthermore, despite considerable efforts to stabilize the acquisition
procedure, several technical challenges persist. Firstly, there is a high
variability in terms of color, saturation, and contrast from one image to
another. For instance, there are instances where the tissues absorb some of
the markers (see Figures 5(b), 5(c), 5(d), 5(e) and 5(g)), causing irrelevant
compounds to emit light which is then captured by the microscope.
Consequently, the background’s hue may shift towards values similar to those
of faint neuronal cells (see Figures 5(b), 5(c), 5(d), 5(e) and 5(f)). In such
circumstances, relying solely on pixel intensity is insufficient to
distinguish between signal and background, necessitating the consideration of
additional characteristics such as saturation and contrast. However, even the
analysis of these characteristics is not straightforward, as fluorescent
emissions are naturally unstable, leading to fluctuations in the saturation
levels exhibited by cell pixels (cf. Figures 5(a), 5(b) and 5(c) or Figures
5(f) and 5(g)).

Moreover, the substructures of interest have a fluid nature. Also, the shot
can capture different two-dimensional sections depending on how the cells are
oriented within the tissues. As a consequence, the size and the shape of the
stained cells can vary significantly (cf. objects dimension in Figures 2(d),
2(e) and 2(f)), further complicating the discrimination between cells and the
background.

Another challenge arises from the occasional presence of accumulations of
fluorophore in narrow areas, resulting in emissions that closely resemble
those of cells. These artifacts can manifest as small areas, such as point
artifacts and filaments, or larger structures, like lateral stripes (see
Figures 5(b), 5(c), 5(d), 5(e), 5(f) and 5(g)). Again, their presence hampers
the detection task, making the recognition and the understanding of cells
structure and size mandatory for the model.

A further source of complexity is represented by overcrowding (Figures 5(a),
5(d), 5(f) and 5(g)). When several cells are close-by, maybe partially
overlapping, precisely localizing cell boundaries can be challenging, thus
requiring adjustments to prevent the model from merging nearby cells into
single agglomerations.

Last but not least, in some occasions the recognition of cells may be
ambiguous even for human operators(cf. marked and non-marked instances in
Figures 5(a), 5(d), 5(f) and 5(g)). Of course, this poses an issue of
intrinsic subjectivity in the annotation process, which in turn affects both
the training and assessment phases.

By and large, all of these factors make the recognition and counting tasks
harder and complicate the learning process. Likewise, borderline annotations
hinder model evaluation as their subjectivity deprives the model of a reliable
and indisputable testbed.

### Research lines

As long as potential applications, the FNC dataset offers rich opportunities
for diverse research directions, including:

  * •

Object Segmentation, Detection, and Counting: The dataset’s comprehensive
annotations and diverse neuronal structures support studies focusing on
accurate segmentation, detection, and counting of cells. Particularly, FNC may
be a challenging benchmark for class imbalance, object
overlapping/overcrowding, and uncertainty estimation

  * •

Transfer Learning: With the availability of multiple image collections within
FNC, researchers can explore transfer learning techniques, leveraging
knowledge from one collection to improve performance on another.

  * •

Unsupervised or Self-/Weakly-Supervised Learning: The presence of both labeled
and unlabeled data within the FNC dataset provides an ideal testbed for
evaluating unsupervised or self-/weakly-supervised learning approaches.

  * •

Evaluation of Annotation Types: Researchers can investigate the effectiveness
of different annotation types for specific tasks, allowing for a comparative
analysis and selection of the most suitable annotations considering the
cost/performance requirements of a given use-case.

### Limitations

Despite the Fluorescent Neuronal Cells collection presenting a variety of
images in many aspects, it has limitations in terms of diversity across
several parameters.

Firstly, all the images were collected by the same research laboratory in
Bologna, utilizing fixed experimental conditions and acquisition settings.
Furthermore, the images were captured using epifluorescence microscopy, which
limits the range of techniques employed. However, we believe that the adopted
acquisition settings represent a more challenging scenario. Therefore, pre-
training on FNC data should enable generalization to modern equipment such as
confocal microscopy, which produces higher-quality images with sharper object
boundaries and improved signal-to-noise ratio.

Another limitation lies in the lack of diversity in the cell types depicted
and the animal species involved. Our dataset only focuses on subcellular
components of rodent neurons. This might potentially impact the generalization
of the models to different use cases and restrict their application to other
cell types or animal species.

## Code availability

The code associated with this work is available on GitHub
(https://github.com/clissa/fluocells-scientific-data). The repository contains
utils to:

  * •

perform data operations (dataOps/: i) converting raw TIFF images into PNG with
metadata, ii) recreating expected data folders structure, iii) convert VIA
annotation to binary masks, iv) encode binary masks into various annotation
formats and types, v) preprocess yellow masks from previous FNC version[22])

  * •

implement deep learning modelling strategies (fluocells/models/: contains
network blocks to implement c-ResUnet architecture[5]; compute_metrics.py,
evaluate.py and training.py: contain utils to implement model training and
evaluation)

  * •

explore, analyze and evaluate models interactively (notebooks/: contains
jupyter notebooks with examples of how to deal with standard stages of data
analysis, namely i) exploratory data analysis, ii) implementation of model
architecture and training pipeline, and iii) experiments

## References

  * [1] Dentico, D. _et al._ C-fos expression in preoptic nuclei as a marker of sleep rebound in the rat.  _European Journal of Neuroscience_ 30, 651–661, 10.1111/j.1460-9568.2009.06848.x (2009).
  * [2] Luppi, M. _et al._ c-fos expression in the limbic thalamus following thermoregulatory and wake–sleep changes in the rat.  _Experimental Brain Research_ 237, 1397–1407, 10.1007/s00221-019-05521-2 (2019).
  * [3] Chiocchetti, R. _et al._ Phosphorylated tau protein in the myenteric plexus of the ileum and colon of normothermic rats and during synthetic torpor.  _Cell and Tissue Research_ 384, 287–299 (2021).
  * [4] Hitrec, T. _et al._ Neural control of fasting-induced torpor in mice.  _Scientific Reports_ 9, 10.1038/s41598-019-51841-2 (2019).
  * [5] Morelli, R. _et al._ Automating cell counting in fluorescent microscopy through deep learning with c-ResUnet.  _Scientific Reports_ 11, 22920, 10.1038/s41598-021-01929-5 (2021).
  * [6] Cao, Y., Liu, S., Peng, Y. & Li, J.  Denseunet: densely connected unet for electron microscopy image segmentation.  _IET Image Processing_ 14, 2682–2689 (2020).
  * [7] Riccio, D., Brancati, N., Frucci, M. & Gragnaniello, D.  A new unsupervised approach for segmenting and counting cells in high-throughput microscopy image sets.  _IEEE Journal of Biomedical and Health Informatics_ PP, 1–1, 10.1109/JBHI.2018.2817485 (2018).
  * [8] Kumar, N. _et al._ A multi-organ nucleus segmentation challenge.  _IEEE Transactions on Medical Imaging_ 39, 1380–1391, 10.1109/TMI.2019.2947628 (2020).
  * [9] Ouyang, C. _et al._ Causality-inspired single-source domain generalization for medical image segmentation.  _IEEE Transactions on Medical Imaging_ 42, 1095–1106, 10.1109/TMI.2022.3224067 (2023).
  * [10] Poon, C., Teikari, P., Rachmadi, M. F., Skibbe, H. & Hynynen, K.  A dataset of rodent cerebrovasculature from in vivo multiphoton fluorescence microscopy imaging.  _Scientific Data_ 10, 141 (2023).
  * [11] Xie, J., Kiefel, M., Sun, M.-T. & Geiger, A.  Semantic instance annotation of street scenes by 3d to 2d label transfer.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ (2016).
  * [12] Haq, M. M. & Huang, J.  Adversarial domain adaptation for cell segmentation.  In _Medical Imaging with Deep Learning_ , 277–287 (PMLR, 2020).
  * [13] Brieu, N. _et al._ Domain adaptation-based augmentation for weakly supervised nuclei detection.  _arXiv preprint arXiv:1907.04681_ (2019).
  * [14] Raza, S. E. A. _et al._ Micro-net: A unified model for segmentation of various objects in microscopy images.  _Medical Image Analysis_ 52, 160–173, https://doi.org/10.1016/j.media.2018.12.003 (2019).
  * [15] Taschner-Mandl, S. _et al._ An annotated fluorescence image dataset for training nuclear segmentation methods.  https://www.ebi.ac.uk/biostudies/bioimages/studies/S-BSST265 (2020).
  * [16] Waithe, D. _et al._ Fluorescence Microscopy Data for Cellular Detection using Object Detection Networks., 10.5281/zenodo.2548493 (2019).
  * [17] Stringer, C., Wang, T., Michaelos, M. & Pachitariu, M.  Cellpose: a generalist algorithm for cellular segmentation.  _Nature methods_ 18, 100–106 (2021).
  * [18] Kovács, K.  Measurement of immediate-early gene activation-c-fos and beyond.  _Journal of neuroendocrinology_ 20, 665–672 (2008).
  * [19] Lencer, W. I. & Tsai, B.  The intracellular voyage of cholera toxin: going retro.  _Trends in biochemical sciences_ 28, 639–645 (2003).
  * [20] Dutta, A., Gupta, A. & Zissermann, A.  VGG image annotator (VIA).  http://www.robots.ox.ac.uk/ vgg/software/via/ (2016).  Version: 2.0.12, Accessed: 2023.
  * [21] Dutta, A. & Zisserman, A.  The VIA annotation software for images, audio and video.  In _Proceedings of the 27th ACM International Conference on Multimedia_ , MM ’19, 10.1145/3343031.3350535 (ACM, New York, NY, USA, 2019).
  * [22] Clissa, L. _et al._ Fluorescent neuronal cells (2021).
  * [23] Lin, T.-Y. _et al._ Microsoft coco: Common objects in context (2015).  1405.0312.
  * [24] Everingham, M., Van Gool, L., Williams, C. K., Winn, J. & Zisserman, A.  The pascal visual object classes (voc) challenge.  _International journal of computer vision_ 88, 303–338 (2010).
  * [25] Pabst, W. & Gregorova, E.  Characterization of particles and particle systems.  _ICT Prague_ 122, 122 (2007).
  * [26] Sudre, C. H., Li, W., Vercauteren, T., Ourselin, S. & Jorge Cardoso, M.  Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations.  In _Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: Third International Workshop, DLMIA 2017, and 7th International Workshop, ML-CDS 2017, Held in Conjunction with MICCAI 2017, Québec City, QC, Canada, September 14, Proceedings 3_ , 240–248 (Springer, 2017).
  * [27] Kingma, D. P. & Ba, J.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * [28] Smith, L. N.  A disciplined approach to neural network hyper-parameters: Part 1–learning rate, batch size, momentum, and weight decay.  _arXiv preprint arXiv:1803.09820_ (2018).
  * [29] Howard, J. & Gugger, S.  Fastai: A layered api for deep learning.  _Information_ 11, 108 (2020).
  * [30] Smith, L. N.  Cyclical learning rates for training neural networks.  In _2017 IEEE winter conference on applications of computer vision (WACV)_ , 464–472 (IEEE, 2017).
  * [31] Smith, L. N. & Topin, N.  Super-convergence: Very fast training of neural networks using large learning rates.  In _Artificial intelligence and machine learning for multi-domain operations applications_ , vol. 11006, 369–386 (SPIE, 2019).
  * [32] Kirillov, A., He, K., Girshick, R., Rother, C. & Dollár, P.  Panoptic segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 9404–9413 (2019).

## Acknowledgements

The study was supported with funding by the Italian National Institute for
Nuclear Physics (INFN). The collection of original images was supported by
funding from the University of Bologna and the European Space Agency (Research
agreement collaboration 4000123556).

## Author contributions statement

L.C. conceived the idea of curating and sharing the annotated data. M.L.,
R.A., M.C. and T.H. designed the biological study and M.L., A.O., E.P. and
L.T. collected the original microscopic fluorescent images. L.C. defined and
orchestrated the annotation pipeline and instruments. M.L., A.O., E.P. and
L.T. performed data annotation and review. L.C., A.M. and R.M implemented the
code for data conversion, pre-processing, exploration, visualization and
modelling. L.C. and A.M. performed technical validation. L.C. and A.M drafted
the manuscript. L.C., A.M., R.M., M.L., A.O., E.P. and L.T. proofread and
revised the manuscript. All authors read and approved the final manuscript.

## Competing interests

The authors declare no competing interests.


# CopyScope: Model-level Copyright Infringement Quantification in the
Diffusion Workflow

Junlei Zhou  Southern University of Science and TechnologyChina
<EMAIL_ADDRESS>,  Jiashi Gao  Southern University of Science and
TechnologyChina<EMAIL_ADDRESS>,  Ziwei Wang  Southern
University of Science and TechnologyChina<EMAIL_ADDRESS>and
Xuetao Wei  Southern University of Science and TechnologyChina
<EMAIL_ADDRESS>
###### Abstract.

Web-based AI image generation has become an innovative art form that can
generate novel artworks with the rapid development of the diffusion model.
However, this new technique brings potential copyright infringement risks as
it may incorporate the existing artworks without the owners’ consent.
Copyright infringement quantification is the primary and challenging step
towards AI-generated image copyright traceability. Previous work only focused
on data attribution from the training data perspective, which is unsuitable
for tracing and quantifying copyright infringement in practice because of the
following reasons: (1) the training datasets are not always available in
public; (2) the model provider is the responsible party, not the image.
Motivated by this, in this paper, we propose CopyScope, a new framework to
quantify the infringement of AI-generated images from the model level. We
first rigorously identify pivotal components within the AI image generation
pipeline. Then, we propose to take advantage of Fréchet Inception Distance
(FID) to effectively capture the image similarity that fits human perception
naturally. We further propose the FID-based Shapley algorithm to evaluate the
infringement contribution among models. Extensive experiments demonstrate that
our work not only reveals the intricacies of infringement quantification but
also effectively depicts the infringing models quantitatively, thus promoting
accountability in AI image-generation tasks.

Accountability, Generative AI, Copyright traceability, Infringement

##  1\. Introduction

AI image generation programs, namely Artificial Intelligence Generated Content
(AIGC) tools such as Stable Diffusion (sta, 2023b), DALL·E2 (Dal, 2023),
Midjourney (Mid, 2022), are setting off a new revolution of artwork creation.
These programs allow users to effortlessly generate target images by taking
some descriptions as input into the model. However, these AI-generated
artworks inherit the characteristics of the images that are used to train
models (Wang et al., 2023; Marra et al., 2019; Sha et al., 2022), which might
be pretty similar to the original ones, as shown in Figure 1. Such similarity
has aroused concerns about copyright infringement disputes. For example, three
artists (Sarah Andersen, Kelly McKernan, and Karla Ortiz) have recently
accused Stable Diffusion of unlawfully scraping copyrighted images from the
Internet to mimic their art styles(Get, 2023). To this end, research on
responsible AI image generation is in urgent need to address such copyright
issues.

Figure 1. The image (1) was created by painter Erin Hanson in 2021, and the
image (2) was created by Stable Diffusion using the ”style of Erin Hanson” as
a prompt. The styles of these two images are so similar that it is impossible
to tell them apart.

Oil painting on crystal, light and shadow, back lit tree, strong silhouette,
stained glass, modern impressionism, style of Erin Hanson

Previous work on data attribution (Mahmood et al., 2022; Jia et al., 2021;
Kwon and Zou, 2021; Sim et al., 2022) focused on how the images in the
training data contributed to the model’s outcome, which is not suitable for
the context of copyright traceability. This is because of the following
reasons: (1) the training data is not known in advance in real-world
practices; as shown in Figure 2, the models are usually publicly available,
while the training datasets are not(Wang et al., 2023; Dai and Gifford, 2023).
(2) the responsible party is the model provider, namely, the personnel or the
organization who abused the online image collections without the owners’
consent, not the image itself. As long as the model that generates the
infringing image is identified, the corresponding infringer (the model
provider) can be found, and the degree of infringement can be quantified.
Thus, we need to develop a new approach to quantify copyright infringement
from the model level, which is the focus of our paper.

To this end, we propose a new framework CopyScope at the model level towards
AIGC copyright traceability. Our framework CopyScope includes three closely
intertwined stages (Identify- Quantify-Evaluate). In the Identify stage, we
conduct an extensive and in-depth analysis of 16,000 generated images under
six themes and rigorously identify four components (Base Model(Rombach et al.,
2022), Lora(Hu et al., 2021), ControlNet(Zhang et al., 2023), and Key
Prompt(Radford et al., 2021)) that are involved in the infringement in
diffusion workflow. In the Quantify stage, we extensively compare and analyze
five metrics, which include Cosine(cos, 2023), DHash (Difference Hash
similarity)(dHa, 2023), Hist (Histogram similarity)(his, 2023), SSIM
(Structural similarity)(Wang et al., 2004), and FID (Fréchet Inception
Distance)(Heusel et al., 2017) methods, from multiple dimensions, such as
style, structure, etc., to measure the similarity between the generated and
original images. We find that the FID metric is the most effective
quantification method because FID can capture the similarity that fits human
perception naturally, which could reflect accurate quantification of each
model’s contribution. In the Evaluate stage, we model our scenario into a
cooperative game model and propose the FID-based Shapley method to evaluate
the contribution of each infringement model. In the end, we conduct extensive
experiments to demonstrate that our proposed FID-based Shapley algorithm could
effectively quantify the infringement in the diffusion workflow, which offers
a promising approach to help us unveil the intricacies of infringement in the
emerging domain of AI image generation tasks.

We summarize our contributions as follows:

  * •

To our knowledge, we are the first to propose a new copyright infringement
quantification framework CopyScope at the model level, which facilitates
stakeholders to investigate the emerging intricate infringement case.

  * •

We propose the FID-based Shapley algorithm to effectively quantify the
infringement in the diffusion workflow, which takes advantage of Fréchet
Inception Distance (FID) to effectively capture the image similarity that fits
human perception naturally, and the Shaplely value scheme to quantify the
infringement contribution among models.

  * •

We conduct extensive experiments to demonstrate the effectiveness of our
proposed framework CopyScope, which depicts the infringing models
quantitatively, thus promoting the legally compliant use of AI-generated
content.

Figure 2. The interaction between user and model. It shows that the training
images are invisible other than the model provider, and each model is uniquely
associated with its provider, which is possible to conduct a model-side
infringement tracing.

Figure 3.  Illustration of the diffusion workflow, which consists of two
parts: (a) the image creation process, in which multiple components jointly
contribute to the generation of images; (b) the Evaluate stage, in which the
contribution of multiple models on the generated image’s copyright is
evaluated.

##  2\. Preliminary and Background

###  2.1. AI Image Generation

In Figure 3, we show the basic diffusion workflow to generate images with text
as input: ❶ First, a user sets up a fundamental component to create images,
which includes the text encoder model, U-Net $\&$ scheduler, and image
decoder. ❷ The user selects specific prompts and adjusts related parameters
such as seed, sampling steps, image size, etc. However, due to the
universality of the fundamental component, the above process alone can merely
generate coarse images. Therefore, additional components such as low-rank
adaptation of large language models (Lora)(Hu et al., 2021) and conditional
control of text-to-image diffusion models (ControlNet)(Zhang et al., 2023) are
required to create fine images with specific themes. These components jointly
contribute to the output image’s characteristics, including content, style,
and background, which refer to the originality of the image and pose a
potential risk of copyright infringement. The functionality of each component
is described as follows:

  1. (1)

Base model: The most famous model for AI image generation is the SD series
model (e.g., SD1.5(Rombach et al., 2022), SD2.0, and SD XL(SD-, 2023).)
released by StabilityAI(sta, 2023a) and Runway (run, 2023). However, as these
models are trained using hundreds of millions of images with mixed styles,
they are difficult to generate images with a specific theme. Therefore, the
image generation usually relies on secondary-trained models based on these
basic models (e.g., DreamShaper(dre, 2023), GhostMix(gho, 2023), Anything(any,
2023), SDMv10(SDM, 2023)). These secondary-trained models have unique themes
and styles and are more likely to infringe the existing works. We thereby
conduct copyright traceability based on the secondary-trained model and refer
to it as the Base Model.

  2. (2)

U-Net $\&$ Scheduler: U-Net iteratively denoises the Gaussian noise matrix in
a diffusion loop, and the noise of each prediction is guided by text and
timesteps. The predicted noise is removed from the random Gaussian noise
matrix, and finally, the random Gaussian noise matrix is converted to the
latent features of the image. The scheduler is responsible for the forward and
backward propagation of the entire Diffusion model. It processes the model’s
output during training and inference according to the set mathematical rules
and the number of timesteps.

  3. (3)

Lora: A fine-tuning model generates a specific type of image with a particular
style of artist (e.g., Ghibli Style(Ghi, 2023), Davinci Style(Leo, 2023), One
Piece Style(one, 2023)). Such models are trained at lower cost and are more
likely to generate images with particular topics, making them more susceptible
to producing infringing images.

  4. (4)

ControlNet: A neural network model is deployed to regulate the stable
diffusion model, which enables control over the primary structure of the
generated image. We can transfer the composition or human pose from the
reference image to the target image by utilizing ControlNet. Furthermore,
ControlNet is almost inseparable in some specific generation tasks, such as
specific image structure, design layout, architectural form, line drawing
coloring, and other scenes(Dep, 2023).

  5. (5)

Prompt: Users can leverage their imagination and employ appropriate prompt
words to describe their images to AI to generate desirable results. Based on
the content, description instructions can be categorized into type prompts,
content prompts, composition prompts, and painter prompts. Among these,
painter prompts are primarily intended for generating the image’s subject,
which can be an artist’s painting style or the protagonist of a specific work.
However, it is essential to note that such prompts often directly infringe
upon the original author’s rights, which are called Key Prompt in our
research.

Components | Description | Models
---|---|---
Base Model | The basic model of stable diffusion determines the style of generated image. |  SDv1-5(SDv, 2022): Universal model without specific topic.
SDMv10(SDM, 2023): Based on SDv1-5 and added training of classical works of
art.
ControlNet |  A category of models that control image structure by adding additional conditions. |  Depth(Dep, 2023): Capture the original image’s structural depth to control the
generated image’s structure.
Lora | Fine-tune the generated image. |  Leonardo(Leo, 2023): Use Davinci’s portfolio training to adjust images to more
closely resemble Davinci’s creative style.
Key Prompt | Instruct an AI on what to paint. |  Davinci: Tips for generating Leonardo da Vinci style images.
MonaLisa: Make the generated image closer to MonaLisa.

Table 1. Four infringement components have been identified, each representing
a type of model set. A brief description of their function is in the
Description column. The Models column shows the specific models we used in the
Mona Lisa experiment (please see section 4.)

###  2.2. Similarity Metrics

To study the issue of copyright tracing more accurately and effectively, we
need to quantify the models involved in infringement. Regarding AI-generated
images, we need to examine their relationship with the original image from
multiple dimensions, such as style, structure, and other characteristics.
Therefore, we quantify the models associated with infringement based on these
dimensions and compare the following quantitative indicators:

Cosine similarity (Cosine). Cosine similarity calculates the cosine of the
angle between the vectors, equivalent to the vectors’ dot product divided by
the product of their lengths. In the image similarity calculation, we can
convert the image into a feature vector and then use cosine similarity to
compare the similarity of these feature vectors. The image cosine similarity
ranges from $[0,1]$, where $1$ indicates the same vectors and $0$ indicates
orthogonality or decorrelation.

Difference Hash similarity (DHash). The essence of the hash algorithm for
image similarity recognition is to hash the image to generate a set of binary
numbers and then find similar images by comparing the hash value distance of
different images. DHash(dHa, 2023) is a differential hashing algorithm that
compares the size of the left and right pixels when hashing an image to obtain
the hash sequence. A larger hash value indicates a more tremendous difference
between the two images.

Histogram similarity (Hist). It measures the similarity of two pictures in
color distribution. The histogram algorithm counts the number of pixels of
different colors in the image, presents it as a histogram, and then compares
the image similarity. The value range is $[0,1]$. The closer to $1$, the more
similar the two are. However, the histogram method only considers the color
distribution but ignores texture and structure information.

Structural similarity (SSIM). SSIM(Wang et al., 2004) simultaneously compares
the similarity of images from three aspects: brightness, contrast, and
structure. The SSIM algorithm can enhance the structural similarity of images
in a group and better detect subtle differences.

(1)
$\textbf{SSIM}(r,g)=\frac{{(2\mu_{r}\mu_{g}+C_{1})(2\sigma_{rg}+C_{2})}}{{(\mu_{r}^{2}+\mu_{g}^{2}+C_{1})(\sigma_{r}^{2}+\sigma_{g}^{2}+C_{2})}}.$

Among them, $\mu$ represents the average brightness of the image,$\sigma$
represents the standard deviation of image brightness, $\sigma$ represents the
variance of image $r$, $g$, and $C$ is a constant. The value range of SSIM is
$[0,1]$. The larger the value, the more similar the two images are.

FID method (FID). Fréchet Inception Distance(Heusel et al., 2017), originally
used as an evaluation index of the generative model to calculate the distance
between the real image and the generated image feature vector. FID directly
considers the distance between the generated data and the accurate data at the
feature level, and the smaller the data, the more similar. From a theoretical
perspective, FID measures the distance between two multivariate normal
distributions, and its calculation formula is as follows,

(2)
$\textbf{FID}={\parallel\mu_{r}-\mu_{g}\parallel}_{2}^{2}+Tr(\Sigma_{r}+\Sigma_{g}-2(\Sigma_{r}\Sigma_{g})^{1/2}).$

Among them, $\mu_{r}$ and $\mu_{g}$ represent the mean value of the feature
vectors extracted from the real image and the generated image, respectively;
$\Sigma_{r}$ and $\Sigma_{g}$ represent the covariance matrix of the original
image and the generated image.

| Celebrity | Film$\&$TV | Artwork | Popular models | Design | Game
---|---|---|---|---|---|---
Base Model | 3,164(100$\%$) | 3,210(100$\%$) | 3,031(100$\%$) | 3,383(100$\%$) | 1,962(100$\%$) | 1,711(100$\%$)
ControlNet | 1,526(48$\%$) | 1,037(32$\%$) | 1,196(39$\%$) | 1,338(39$\%$) | 352(17$\%$) | 576(33$\%$)
Lora | 2,959(93$\%$) | 2,893(90$\%$) | 2,167(71$\%$) | 2,764(81$\%$) | 1,158(59$\%$) | 1,711(100$\%$)
Key Prompt | 3,079(97$\%$) | 2,574(80$\%$) | 2,386(93$\%$) | 1,996(59$\%$) | 1,286(65$\%$) | 1,077(62$\%$)
Table 2. The dataset for identifying infringement components in the CopyScope
framework. The generated images in the dataset come from AI-generated images
platform Civitai(civ, 2023). The generated images in the dataset are divided
into six major themes, and the usage ratio of the four types of infringement
components in each category is marked.

##  3\. Methodology

In this section, we present an Identify-Quantify-Evaluate framework called
CopyScope to address the issue of AI-generated image copyright traceability at
the model level. In the Identify stage, we first rigorously select four
pivotal components for describing infringing models by analyzing images
generated from Civitai(civ, 2023). In the Quantify stage, we adopt FID to
measure the similarity between the original images and the images generated by
our designed model under different alliances of models. In the Evaluate stage,
we trace back the possible infringing model by computing the contribution of
models using the FID-Shapley value.

###  3.1. Identify Influential components

We initiate our study by determining components that have the most significant
impacts on the generated images. Components make up the AI image generation
workflow, which is used to characterize generation models in our proposed
copyright tracking approach.

This stage is based on a survey from the world’s largest AI image generation
exchange and sharing platform Civitai, where we collect more than 16,000
generated image data from over 5,000 models to find commonalities in generated
images. The generated images are divided into 6 themes: celebrity, film&TV,
artwork, popular models, design, and game. We explore the distribution of
models that generate images involving copyright infringement by calculating
the usage rate of components: Base Model, Lora, ControlNet, and Key Prompt.
Table 2 shows the frequency of these components, indicating that they have a
high usage rate in AI image-generation tasks.

We identify four components that are used in AI image generation at a high
frequency: ❶ The Base Model is essential for each generated image. ❷ The
second is the Lora. Although the Lora is not a necessary option for generating
images, we can find from Table 2 that it has a high application rate in each
category, indicating that the use of Lora for adjustment in AI image
generation has become a norm. ❸ Prompts with particular specificity are called
Key Prompts. Key Prompt can make the generated image close to the
characteristics of these keywords to a large extent, thus infringing on the
original author’s rights. ❹ The overall usage rate of ControlNet is relatively
lower than the other components. This is because the ControlNet is challenging
to use as it needs higher environment configuration requirements than Lora and
Key Prompt(ext, 2022). However, ControlNet is an essential components in
generating particular themes as it controls the structure of the image. From
the perspective of copyright tracing, the ControlNet is a critical suspected
infringement component that our CopyScope framework considers.

###  3.2. Quantify Model Performance

As shown in Table 2.1, we choose specific models for each component. We select
SDv1-5 and SDMv10 for Base Model, Depth for ControlNet, Leonardo for Lora,
Davinci and MonaLisa for Key Prompt. We use these models to simulate 30
different alliances, where the two specific models of Base Model is essential
for any alliances of models and the other four specific models of ControlNet,
Lora, Key Prompt can be used to form an alliance at the same time. In this
way, the alliance of models is calculated as
$2\times(C_{4}^{1}+C_{4}^{2}+C_{4}^{3}+C_{4}^{4})=30$. We then use each
alliance to generate $100$ batches of images to generate the dataset of
$3,000$ images for quantification and evaluation.

We propose to explore which model can affect the generated image and thereby
cause infringement by studying the generated and original images. Therefore,
accurately measuring the similarity between the generated and original images
is crucial to quantification. We quantify the performance of each alliance by
measuring the similarity of the images generated by the alliance and the
original image. To this end, we present the results of multiple indicators
such as Cosine, Hist, DHash, SSIM, and FID as described in section 2.2.

According to the results in the following experiment, we comparatively select
FID as the quantification method because the results of images’ similarity are
more approximate to the human intuition. Furthermore, in section 4.2, we
compare these indicators and found that FID has a more accurate similarity
description ability than other quantitative methods. Additionally, the
discrimination between the generated results of different alliances is more
remarkable, which is more helpful for our subsequent evaluation.

###  3.3. Evaluate Contributions of Models

In the sections above, we have identified the copyright-related components in
the diffusion workflow and proposed quantitative metrics to measure image
similarity. To achieve copyright traceability and determine the level of
infringement for different models, we introduce the Evaluate stage as the
final step of the CopyScope framework, as shown in Figure 3. The Evaluate
stage aims to assess the contribution of each model in the output image. We
formally define the Evaluate stage as follows:

###### Definition 3.1.

(Evaluate stage) Given a set of models
$\mathcal{M}=\\{z_{1},z_{2},\dots,z_{n}\\}$, where each $z_{i}$ is a specific
model of the component in the diffusion workflow, the Evaluate stage aims to
find the value of $v(z)$ of each model $z$, which represents how much it
contributes to the copyright of the generated image.

Let $\mathcal{L}=\left\\{z_{1},...,z_{N}\right\\}$ be an alliance formed by
$N$ models and $U(\cdot)$ be the value function that can be applied on any
subset of the alliance $\mathcal{L}$. To evaluate the contribution of $i$-th
model $z_{i}$ to the overall value of the alliance $\mathcal{L}$, two widely
used solutions from cooperative game theory are Leave-one-out (LOO) (Ying and
Keong, 2004) and Shapley value (SV) (Rozemberczki et al., 2022).

LOO-based Method. The idea of LOO is to measure the marginal contribution of
each model $z$ to the alliance by removing it from the alliance and observing
the difference. For the $i$-th model $z_{i}$, its contribution
$v_{LOO}(z_{i})$, also referred to as the LOO value, can be obtained as
follows,

(3) $v_{LOO}(z_{i})\propto U(\mathcal{L})-U(\mathcal{L}\backslash{z_{i}}).$

SV-based Method. The SV-based method was originally used to provide a fair way
of dividing the benefits for players in a coalition based on their individual
and joint contributions. To calculate the contribution of a model in the
alliance, the SV-based method considers all possible sub-alliances that
include that model and then takes the weighted average of the differences
between the value of each sub-alliance with and without that model. For the
$i$-th model $z_{i}$, its contribution $v_{SV}(z_{i})$, also referred to as
the Shapley value, can be obtained as follows,

(4)
$v_{SV}(z_{i})\propto\frac{1}{N}\sum_{\mathcal{L}\subseteq\mathcal{M}\setminus{z_{i}}}[U(\mathcal{L}\cup{z_{i}})-U(\mathcal{L})]\frac{|\mathcal{L}|!(N-1-|\mathcal{L}|)!}{(N-1)!}.$

Algorithm 1 FID-Shapley Algorithm

1:Input: All models: $\mathcal{M}=\left\\{z_{1},...,z_{N}\right\\}$; the
alliances set
$\mathcal{S}=\\{\mathcal{L}_{1},\mathcal{L}_{2},\dots,\mathcal{L}_{n}\\}$,
where $\mathcal{S}$ contains all possible alliance $\mathcal{L}$ from
$\mathcal{M}$; the FID-based value function $U(\cdot)$.

2:Initialize: An FID-Shapley value set $\mathcal{R}=\mathcal{\varnothing}$,
FID-Shapley value $v=0$.

3:for each model $z$ in $\mathcal{M}$ do

4: Initialize margin contribution $c(z)=0$;

5: for each alliance $\mathcal{L}$ in $\mathcal{S}$ do

6: if $z\notin\mathcal{L}$ then

7: Continue;

8: else

9: Update the margin contribution of model $z$:

10: $c(z)\leftarrow
c(z)+[U(\mathcal{L})-U(\mathcal{L}-z)]\times\frac{|\mathcal{L}|!(N-1-|\mathcal{L}|)!}{(N-1)!}$;

11: end if

12: end for

13: Calculate the FID-Shapley value of $z$: $v(z)=\frac{c(z)}{N}$;

14: Append to FID-Shapley value set $\mathcal{R}\leftarrow v(z)$;

15:end for

16:Return: $\mathcal{R}$.

We represent the models in the diffusion workflow as an alliance in a
cooperative game(Ghorbani and Zou, 2019; Pandl et al., 2021; He et al., 2021),
as each model jointly forms an image generation alliance, as shown in Table 3,
and the value of the model can be calculated by a value function, which is the
similarity metrics in our work. Each model is independent and does not affect
each other, and the diffusion workflow can be built by any sub-alliance from
the models. Meanwhile, the models’ alliance satisfies the following
properties(Shapley, 1951), which enables us to design a contribution
evaluation method based on the Shapley value.

Figure No. | Alliances |  Cosine $\uparrow$ | Hist$\uparrow$ | DHash$\uparrow$ | SSIM$\uparrow$ |  RGB-SSIM $\uparrow$ | FID$\downarrow$
---|---|---|---|---|---|---|---
Figure 4(1) | SDv1-5 | 0.8802 | 0.1511 | 0.5468 | 0.2934 | 0.8424 | 310.18
Figure 4(2) |  SDv1-5+Depth | 0.8927 | 0.3518 | 0.5000 | 0.4541 | 0.9251 | 289.24
Figure 4(3) |  SDv1-5+Depth+Davinci | 0.9817 | 0.5582 | 0.7656 | 0.9281 | 0.9971 | 239.17
Figure 4(4) |  SDv1-5+Depth+Davinci+MonaLisa | 0.9087 | 0.5789 | 0.7500 | 0.9684 | 0.9963 | 233.21
Figure 4(5) |  SDv1-5+Depth+Davinci+MonaLisa+Leonardo | 0.8279 | 0.5356 | 0.7656 | 0.7463 | 0.9689 | 220.40
Figure 4(6) |  SDv1-5+Davinci | 0.9184 | 0.3734 | 0.4843 | 0.5275 | 0.9200 | 265.01
Figure 4(7) |  SDv1-5+Davinci+MonaLisa | 0.9328 | 0.4591 | 0.6562 | 0.7235 | 0.9420 | 241.18
Figure 4(8) |  SDv1-5+Davinci+MonaLisa+Leonardo | 0.8876 | 0.4085 | 0.5937 | 0.6065 | 0.9458 | 275.92
Figure 4(9) |  SDv1-5+MonaLisa | 0.8778 | 0.4417 | 0.3593 | 0.1982 | 0.8251 | 336.24
Figure 4(10) |  SDv1-5+Leonardo | 0.8705 | 0.0964 | 0.6718 | 0.2673 | 0.9148 | 307.06
Figure 4(11) | SDMv10 | 0.8759 | 0.0372 | 0.7031 | 0.2568 | 0.8837 | 320.48
Figure 4(12) |  SDMv10+Depth+Davinci | 0.8898 | 0.5227 | 0.7343 | 0.6214 | 0.9819 | 209.06
Figure 4(13) |  SDMv10+Depth+Davinci+MonaLisa | 0.8876 | 0.5891 | 0.7656 | 0.5766 | 0.9834 | 212.13
Figure 4(14) |  SDMv10+Depth+Davinci+MonaLisa+Leonardo | 0.8605 | 0.4096 | 0.8593 | 0.4481 | 0.9733 | 184.69

  * •

Bold: Best performance compared to all alliances.

  * •

$\uparrow$: Higher value corresponds to the generated image being more similar
to the original image.

  * •

$\downarrow$: Lower value corresponds to the generated image being more
similar to the original image.

Table 3. Quantitative results under different Quantitative metrics. The Figure
No. column corresponds to each generated Mona Lisa image in Figure 4. The
Alliances column gives the model alliances used in each generated image(more
quantitative data from the alliance, see Appendix A). The Cosine$\sim$FID
column is the similarity score between each generated and original image under
different quantification methods.

Figure 4. The different models (e.g., SDMv10, Depth, Davinci, etc.) are
alliances to simulate the generation of Leonardo da Vinci’s Mona Lisa images.
In this example, we explore 30 alliances, including six models. We generate
100 batches of images for each alliance to explore the similarity between the
images generated by different alliances and the original images.

Figure 5. The pixel difference between the generated image and the original
image, where darker color indicates larger differences, and lighter color
indicates smaller differences.

  * •

Property 1. ${\textstyle\sum_{z\in\mathcal{M}}}v(z)=U(\mathcal{M})$. The
Shapley values of all collaborators add up to the value of the grand
coalition, ensuring that the total benefit is shared among them.

  * •

Property 2. For a player $z$, if $U(\mathcal{L})=U(\mathcal{L}\cup{z})$ holds
for any alliance $\mathcal{L}$, then $v(z)=0$. This implies that if a model
that participates in the diffusion workflow leaves the generated image
unchanged from its absence, then this model does not affect the image
generation and deserves zero contribution.

  * •

Property 3. If player $z_{i}$ and player $z_{j}$ satisfy
$U(\mathcal{L}\cup{z_{i}})=U(\mathcal{L}\cup z_{j})$, for any alliance
$\mathcal{L}$ that does not include $z_{i}$ and $z_{j}$, then
$v(z_{i})=v(z_{j})$. If multiple models contribute equally, they should be
assigned the same contribution.

  * •

Property 4. For two value functions $U$ and $U^{\prime}$, we calculate the
Shapley value of a player $z$ based on $U$, $U^{\prime}$ and $U+U^{\prime}$,
denoted by $v(z)$, $v^{\prime}(z)$ and $v^{\prime\prime}(z)$, respectively.
Then, we have $v^{\prime\prime}(z)=v(z)+v^{\prime}(z)$. If we combine two
coalition games with value functions $U$ and $U^{\prime}$, then the Shapley
value of the combined coalition is equal to the sum of the Shapley values from
each individual coalition. In our scenario, the image generation tasks based
on different value functions are independent, and their contribution
evaluations satisfy such additivity property.

Based on the above analysis, we demonstrate that our scenario theoretically
fits the cooperative game model, and we can use the LOO and Shapley value
methods to evaluate the model contribution in the diffusion workflow of image
generation. We innovatively adopt the FID as the value function and propose
the FID-Shapley algorithm to measure the contribution of each model alliance
in AI image generation, as shown in Algorithm 1. The subsequent experiments
show that FID-Shapley can accurately reflect the contribution of each model to
the AI-generated images, and also agree with the human observation on the
infringement. Models with high FID-Shapley value have a more significant
impact on the AI-generated images and are more likely to cause infringement
issues, which provides guidance for AI-generated image users to pay extra
attention to these models with high FID-Shapley value in order to avoid
copyright infringement in real-world.

##  4\. Experiments

We conduct thorough experiments to ensure that the CopyScope framework can
effectively solve the copyright traceability problem of AI-generated images.

###  4.1. Experimental Setup

To investigate the specific infringement models involved in AI image
generation, we use a diffusion workflow to generate Mona Lisa paintings, one
of the classic artworks. We construct the AI image generation workflow using
the models in Table 2.1. We set SDv1-5 as the pipeline benchmark and assume
its contribution to the generated image is zero. Then, we introduce SDMv10 as
Base model, Depth as ControlNet model, Leonardo as Lora model, and ”Davinci”
and ”MonaLisa” as two Key Prompts, and thus we obtain the models set
$\mathcal{M}=\\{$ SDMv10, Depth, Davinci, ”MonaLisa”, ”Leonardo” $\\}$. We
keep the hyperparameters constant for all experiments, including image size,
scale, sample step, etc. We construct a total number of $30$ diffusion
workflows by different models alliances. With each diffusion workflow, we
generate $100$ images and evaluate their contribution using their average FID
value.

###  4.2. Quantitative Metrics for Model Performance

In this experiment, we compare different quantitative metrics for model
performance introduced in Section 2.2 and demonstrate that the FID is the most
effective metric for identifying the infringement models in AI image
generation. Table 3 presents the similarity values between the images
generated by various diffusion workflows and the original images under
different metrics. The Cosine and RGB-SSIM metrics give similar similarity
values for different workflows, which implies that they are not sensitive to
the model alliance. Thus, they are not informative for assessing the
contribution of various models in the workflow. As shown in Figure 4, the Hist
and DHash metrics, on the other hand, give similarity values that do not agree
with the human perception of infringement. The Hist metric only considers the
color distribution of the images, and the DHash metric only considers the hash
fingerprint of the images. The SSIM and FID metrics have better discrimination
ability on the similarity values between different generated images and
original images, where SSIM measures the structural similarity between images,
reflected in higher similarity values for images with similar outlines, and
FID measures the similarity from more comprehensive image features, including
outline, style, content, etc. As shown in Figure 5, under pixel-level
evaluation, high FID values correspond to overall similarity, while high SSIM
values correspond only to clear structural similarity. Therefore, FID is more
appropriate for tracing the infringement models in AI image generation.

###  4.3. LOO vs. Shapley: Contribution Evaluation Experiment

In this experiment, we compare the methods proposed in Section 3.3 for
contribution evaluation, FID-LOO and FID-Shapley value.

Figure 6. Comparison of contribution values under FID-LOO (left) and FID-
Shapley (right) methods.  Figure 7. Ablation experiment. Corresponding models
are eliminated in order, and the average FID value of all alliance solutions
composed of the remaining models is calculated. The greater the difference
between the remaining alliances’ FID and the all-model alliance’s FID, the
more influential the model is.

Figure 6 shows the normalized contribution of each model to the generation of
AI image generation, calculated by the two methods. We observe that there are
differences in the contribution evaluation between the two methods: under FID-
LOO, the contribution order is:
$SDMv10>$”Leonardo”$>Depth>$”MonaLisa”$>Davinci$, while under FID-Shapley
value, the contribution order is: $Davinci>Depth>$
”MonaLisa”$>$”Leonardo”$>SDMv10$. To verify which one of FID-LOO and FID-
Shapley values provides a more accurate contribution evaluation in AI image
generation, we further conduct ablation experiments. We drop out each model
separately and calculate the average FID value of the original image with the
images generated by the diffusion models compromised by all subsets composed
of the remaining models to measure the impact of the dropped model on the
generated AI image generation. We repeat generating images of $100$ batches
with regularized seed (reg seed: green line marked by triangles in Figure 7)
and random seed (ran seed: blue line marked by squares in Figure 7). The red
line marked by circles in Figure 7 shows the FID of the AI images generated by
the complete set of models and the original image. From Figure 7, we observe
that when dropping out Davinci, it causes the largest deviation of FID value,
meaning that the similarity between the generated image and the original image
significantly decreases after removing Davinci model, which also means that
Davinci model has the largest contribution to the copyright of the generated
AI image. Similarly, from Figure 7, we can observe that the following
important models are Depth, ”MonaLisa” and ”Leonardo”, SDMv10, respectively,
which is consistent with the contribution evaluation results of FID-Shapley
value. The experiment proves that the FID-Shapley value provides a more
accurate and realistic contribution evaluation in AI image generation than
FID-LOO.

##  5\. Related Work

Data Attribution: Some previous works traced copyright by evaluating the
relationship between training input and generated output. For example, Datta
et al. (2016) explored the impact of black-box machine learning model inputs
on the algorithm by proposing a transparent mechanism. Park et al. (2023)
introduced the method of using Tracing with the Randomly-projected After
Kernel (TRAK) to implement the data attribution problem of large-scale models.
Wang et al. (2023) evaluated data attribution for text-to-image models through
the “customization” method. However, such methods were based on the premise of
a known training dataset. As shown in Figure 2, in AI image generation, the
user can generate images by using the selected model with some prompts. The
specific images used to train the model were only known by the model provider.
Therefore, tracing copyright from the perspective of training images is
challenging to achieve in real-world practices.

Fingerprint Traceability Another sort of approach attempted to track the
models by adding a fingerprint to the model. Kim et al. (2023) modified the
model according to each user’s unique digital fingerprint, so each text-to-
image result was embedded with a unique fingerprint, and the model was traced
through the fingerprint. Marra et al. (2019) experimented with several popular
GAN architectures and datasets and demonstrated the existence of GAN
fingerprints and their value for reliable forensic analyses. Nie et al. (2023)
investigated the use of latent semantic dimensions as fingerprints, improved
fingerprinting methods exhibit a significant tradeoff between robust
attribution accuracy and generation quality, and enhanced the efficiency of
fingerprint identification methods. Fernandez et al. (2023) introduced an
active strategy combining image watermarking and Latent Diffusion Models (LDM)
by fine-tuning the decoder of LDM and embedding watermarks in all the
generated images. Although these work have studied various watermarking
methods to achieve model traceability, these methods can only trace back to a
chosen specific model in their experiments, without considering the interplay
among models in the complex AI image generation task.

##  6\. Conclusion

To address the challenge of potential copyright infringement in AI-generated
images, we have proposed a new framework called CopyScope that could identify
different copyright infringement sources at the model level in the AI image
generation process and evaluate their impact. We have proposed a FID-based
Shapley algorithm to assess the infringement contribution of each model in the
diffusion workflow. Extensive results have demonstrated that our proposed
CopyScope framework could effectively zoom in on the sources and quantify the
impact of infringement models in AI image generation. Our work offers a
promising solution for copyright traceability in AI image generation, which
could also promote the legally compliant use of AI-generated content.

## References

  * (1)
  * Mid (2022) 2022\.  _Midjourney_.  https://www.midjourney.com/
  * SDv (2022) 2022\.  _SDv1-5_.  https://huggingface.co/runwayml/stable-diffusion-v1-5
  * ext (2022) 2022\.  _Stable Diffusion Extensions_.  https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Extensions
  * any (2023) 2023\.  _Anythingv5/ink_.  https://civitai.com/models/9409/or-anything-v5ink
  * civ (2023) 2023\.  _Civitai_.  https://civitai.com.
  * Dep (2023) 2023\.  _ControlNet_.  https://huggingface.co/lllyasviel/ControlNet-v1-1
  * cos (2023) 2023\.  _Cosine similarity_.  https://en.wikipedia.org/wiki/Cosine_similarity
  * Dal (2023) 2023\.  _DALL·E2_.  https://openai.com/dall-e-2
  * dHa (2023) 2023\.  _DHash_.  https://github.com/benhoyt/dhash
  * dre (2023) 2023\.  _DreamShaper_.  https://civitai.com/models/4384/dreamshaper
  * Get (2023) 2023\.  _Getty Images lawsuit says Stability AI misused photos to train AI — Reuters_.  https://www.reuters.com/legal/getty-images-lawsuit-says-stability-ai-misused-photos-train-ai-2023-02-06/
  * gho (2023) 2023\.  _GhostMix_.  https://civitai.com/models/36520?modelVersionId=76907
  * his (2023) 2023\.  _Histogram similarity_.  https://docs.opencv.org/3.4/d8/dc8/tutorial_histogram_comparison.html
  * Leo (2023) 2023\.  _Leonardo da Vinci style_.  https://civitai.com/models/63331/leonardo-da-vinci-style
  * one (2023) 2023\.  _One Piece Style LoRA_.  https://civitai.com/models/4219?modelVersionId=6331
  * run (2023) 2023\.  _Runway_.  https://runwayml.com/
  * SD- (2023) 2023\.  _SD-XL by StabilityAI_.  https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0
  * SDM (2023) 2023\.  _SDMv10_.  https://civitai.com/models/12862/stablydiffuseds-magnum-opus
  * sta (2023a) 2023a.  _StabilityAI_.  https://stability.ai/
  * sta (2023b) 2023b.  _Stable Diffusion_.  https://stability.ai/
  * Ghi (2023) 2023\.  _Studio Ghibli Style LoRA_.  https://civitai.com/models/6526?modelVersionId=7657
  * Dai and Gifford (2023) Zheng Dai and David K Gifford. 2023.  Training Data Attribution for Diffusion Models.  _arXiv:2306.02174_ (2023).
  * Datta et al. (2016) Anupam Datta, Shayak Sen, and Yair Zick. 2016.  Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems. In _IEEE S &P)_.
  * Fernandez et al. (2023) Pierre Fernandez, Guillaume Couairon, Hervé Jégou, Matthijs Douze, and Teddy Furon. 2023.  The Stable Signature: Rooting Watermarks in Latent Diffusion Models. In _IEEE ICCV_.
  * Ghorbani and Zou (2019) Amirata Ghorbani and James Zou. 2019.  Data Shapley: Equitable Valuation of Data for Machine Learning. In _ICML_.
  * He et al. (2021) Qiangqiang He, Yu Qiao, Shang Yang, and Chongjun Wang. 2021.  Equitable Valuation of Crowdsensing for Machine Learning via Game Theory. In _Springer WASA_.
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017.  Gans trained by a two time-scale update rule converge to a local nash equilibrium. In _NeurIPS_.
  * Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.  _arXiv:2106.09685_ (2021).
  * Jia et al. (2021) Ruoxi Jia, Fan Wu, Xuehui Sun, Jiacen Xu, David Dao, Bhavya Kailkhura, Ce Zhang, Bo Li, and Dawn Song. 2021.  Scalability vs. utility: Do we have to sacrifice one for the other in data importance quantification?. In _IEEE CVPR_.
  * Kim et al. (2023) Changhoon Kim, Kyle Min, Maitreya Patel, Sheng Cheng, and Yezhou Yang. 2023.  WOUAF: Weight Modulation for User Attribution and Fingerprinting in Text-to-Image Diffusion Models.  _arXiv:2306.04744_ (2023).
  * Kwon and Zou (2021) Yongchan Kwon and James Zou. 2021.  Beta shapley: a unified and noise-reduced data valuation framework for machine learning.  _arXiv:2110.14049_ (2021).
  * Mahmood et al. (2022) Rafid Mahmood, James Lucas, David Acuna, Daiqing Li, Jonah Philion, Jose M. Alvarez, Zhiding Yu, Sanja Fidler, and Marc T. Law. 2022.  How Much More Data Do I Need? Estimating Requirements for Downstream Tasks. In _IEEE CVPR_.
  * Marra et al. (2019) Francesco Marra, Diego Gragnaniello, Luisa Verdoliva, and Giovanni Poggi. 2019.  Do GANs Leave Artificial Fingerprints?. In _IEEE MIPR_.
  * Nie et al. (2023) Guangyu Nie, Changhoon Kim, Yezhou Yang, and Yi Ren. 2023.  Attributing Image Generative Models Using Latent Fingerprints. In _ICML_.
  * Pandl et al. (2021) Konstantin D. Pandl, Fabian Feiland, Scott Thiebes, and Ali Sunyaev. 2021.  Trustworthy Machine Learning for Health Care: Scalable Data Valuation with the Shapley Value. In _ACM CHIL_.
  * Park et al. (2023) Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. 2023.  TRAK: Attributing Model Behavior at Scale. In _ICML_.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning Transferable Visual Models From Natural Language Supervision. In _ICML_.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022.  High-Resolution Image Synthesis with Latent Diffusion Models. In _IEEE CVPR_.
  * Rozemberczki et al. (2022) Benedek Rozemberczki, Lauren Watson, Péter Bayer, Hao-Tsung Yang, Olivér Kiss, Sebastian Nilsson, and Rik Sarkar. 2022.  The Shapley Value in Machine Learning. In _IJCAI_.
  * Sha et al. (2022) Zeyang Sha, Zheng Li, Ning Yu, and Yang Zhang. 2022.  De-fake: Detection and attribution of fake images generated by text-to-image diffusion models.  _arXiv:2210.06998_ (2022).
  * Shapley (1951) Lloyd S Shapley. 1951.  Notes on the n-person game—ii: The value of an n-person game.(1951).  _Lloyd S Shapley_ 7 (1951).
  * Sim et al. (2022) Rachael Hwee Ling Sim, Xinyi Xu, and Bryan Kian Hsiang Low. 2022.  Data valuation in machine learning:“ingredients”, strategies, and open challenges. In _IJCAI_.
  * Wang et al. (2023) Sheng-Yu Wang, Alexei A Efros, Jun-Yan Zhu, and Richard Zhang. 2023.  Evaluating Data Attribution for Text-to-Image Models.  _arXiv:2306.09345_ (2023).
  * Wang et al. (2004) Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004.  Image quality assessment: from error visibility to structural similarity.  _IEEE transactions on image processing_ 13, 4 (2004), 600–612.
  * Ying and Keong (2004) Zhao Ying and Kwoh Chee Keong. 2004.  Fast leave-one-out evaluation and improvement on inference for LS-SVMs. In _IEEE ICPR_.
  * Zhang et al. (2023) Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023.  Adding conditional control to text-to-image diffusion models. In _IEEE ICCV_.

##  Appendix A Appendix

No. | Alliances |  Cosine $\uparrow$ | Hist$\uparrow$ | DHash$\uparrow$ | SSIM$\uparrow$ |  RGB-SSIM $\uparrow$ | FID$\downarrow$
---|---|---|---|---|---|---|---
| SDv1-5 | 0.8802 | 0.1511 | 0.5468 | 0.2934 | 0.8424 | 310.18
| SDMv10 | 0.8759 | 0.0372 | 0.7031 | 0.2568 | 0.8837 | 320.48
1 |  SDv1-5+Depth | 0.8927 | 0.3518 | 0.5000 | 0.4541 | 0.9251 | 289.24
2 |  SDv1-5+Davinci | 0.9184 | 0.3734 | 0.4843 | 0.5275 | 0.9200 | 265.01
3 |  SDv1-5+MonaLisa | 0.8778 | 0.4417 | 0.3593 | 0.1982 | 0.8251 | 336.24
4 |  SDv1-5+Leonardo | 0.8705 | 0.0964 | 0.6718 | 0.2673 | 0.9148 | 307.06
5 |  SDMv10+Depth | 0.9063 | 0.4476 | 0.6250 | 0.5013 | 0.9876 | 292.57
6 |  SDMv10+Davinci | 0.8512 | 0.2867 | 0.6250 | 0.2314 | 0.8694 | 216.07
7 |  SDMv10+MonaLisa | 0.8902 | 0.5384 | 0.6718 | 0.3225 | 0.9650 | 322.85
8 |  SDMv10+Leonardo | 0.7085 | 0.5844 | 0.6562 | 0.3450 | 0.9557 | 240.05
9 |  SDv1-5+Depth+Davinci | 0.9817 | 0.5582 | 0.7656 | 0.9281 | 0.9971 | 239.17
10 |  SDv1-5+Depth+MonaLisa | 0.8865 | 0.1114 | 0.6406 | 0.2734 | 0.8473 | 262.23
11 |  SDv1-5+Depth+Leonardo | 0.7911 | 0.5074 | 0.5468 | 0.6238 | 0.9848 | 262.71
12 |  SDv1-5+Davinci+MonaLisa | 0.9328 | 0.4591 | 0.6562 | 0.7235 | 0.9420 | 241.18
13 |  SDv1-5+Davinci+Leonardo | 0.7543 | 0.4420 | 0.5156 | 0.8242 | 0.9864 | 260.68
14 |  SDv1-5+MonaLisa+Leonardo | 0.7730 | 0.5207 | 0.5000 | 0.2659 | 0.9279 | 332.45
15 |  SDMv10+Depth+Davinci | 0.8898 | 0.5227 | 0.7343 | 0.6214 | 0.9819 | 209.06
16 |  SDMv10+Depth+MonaLisa | 0.8798 | 0.5393 | 0.7343 | 0.4766 | 0.9910 | 246.01
17 |  SDMv10+Depth+Leonardo | 0.8489 | 0.5735 | 0.7031 | 0.6685 | 0.9245 | 240.16
18 |  SDMv10+Davinci+MonaLisa | 0.8566 | 0.1206 | 0.6718 | 0.3229 | 0.8826 | 230.70
19 |  SDMv10+Davinci+Leonardo | 0.7897 | 0.5634 | 0.5468 | 0.2048 | 0.9280 | 301.20
20 |  SDMv10+MonaLisa+Leonardo | 0.8248 | 0.6755 | 0.6406 | 0.1818 | 0.9254 | 310.06
21 |  SDv1-5+Depth+Davinci+MonaLisa | 0.9087 | 0.5789 | 0.7500 | 0.9684 | 0.9963 | 233.21
22 |  SDv1-5+Depth+Davinci+Leonardo | 0.8156 | 0.5702 | 0.7187 | 0.8251 | 0.9528 | 227.45
23 |  SDv1-5+Depth+MonaLisa+Leonardo | 0.7977 | 0.6190 | 0.7187 | 0.4265 | 0.9564 | 217.41
24 |  SDv1-5+Davinci+MonaLisa+Leonardo | 0.8232 | 0.3436 | 0.5468 | 0.4178 | 0.8974 | 238.63
25 |  SDMv10+Depth+Davinci+MonaLisa | 0.8590 | 0.5709 | 0.7968 | 0.4528 | 0.9775 | 219.01
26 |  SDMv10+Depth+Davinci+Leonardo | 0.8636 | 0.5636 | 0.8125 | 0.4807 | 0.9659 | 194.71
27 |  SDMv10+Depth+MonaLisa+Leonardo | 0.8142 | 0.5113 | 0.7500 | 0.4974 | 0.9315 | 240.71
28 |  SDMv10+Davinci+MonaLisa+Leonardo | 0.8081 | 0.6663 | 0.6250 | 0.2857 | 0.9558 | 248.54
29 |  SDv1-5+Depth+Davinci+MonaLisa+Leonardo | 0.8279 | 0.5356 | 0.7656 | 0.7463 | 0.9689 | 220.40
30 |  SDMv10+Depth+Davinci+MonaLisa+Leonardo | 0.8605 | 0.4096 | 0.8593 | 0.4481 | 0.9733 | 184.69
Table 4. A full quantitative results under different metrics.