* Alho  _et al._ (2021) A. Alho, J. Natário, P. Pani, and G. Raposo, (2021), arXiv:2107.12272 [gr-qc] .
  * Heeck  _et al._ (2021b) J. Heeck, A. Rajaraman, R. Riley, and C. B. Verhaaren, Phys. Rev. D103, 116004 (2021b), arXiv:2103.06905 [hep-th] .
  * Kling and Rajaraman (2017) F. Kling and A. Rajaraman, Phys. Rev. D 96, 044039 (2017), arXiv:1706.04272 [hep-th] .
  * Guzman and Urena-Lopez (2003) F. S. Guzman and L. A. Urena-Lopez, Phys. Rev. D68, 024023 (2003), arXiv:astro-ph/0303440 [astro-ph] .
  * Annulli  _et al._ (2020) L. Annulli, V. Cardoso, and R. Vicente, Phys. Rev. D 102, 063022 (2020), arXiv:2009.00012 [gr-qc] .
  * Kling and Rajaraman (2018) F. Kling and A. Rajaraman, Phys. Rev. D 97, 063012 (2018), arXiv:1712.06539 [hep-ph] .
  * Hui  _et al._ (2017) L. Hui, J. P. Ostriker, S. Tremaine, and E. Witten, Phys. Rev. D95, 043541 (2017), arXiv:1610.08297 [astro-ph.CO] .
  * Delgado  _et al._ (2020) J. F. M. Delgado, C. A. R. Herdeiro, and E. Radu, JCAP 2006, 037 (2020), arXiv:2005.05982 [gr-qc] .
  * Kusenko (1997a) A. Kusenko, Phys. Lett. B 405, 108 (1997a), arXiv:hep-ph/9704273 .
  * Kusenko (1997b) A. Kusenko, Phys. Lett. B 404, 285 (1997b), arXiv:hep-th/9704073 .
  * Yagi and Yunes (2013) K. Yagi and N. Yunes, Phys. Rev. D 88, 023009 (2013), arXiv:1303.1528 [gr-qc] .
  * Yagi and Yunes (2017) K. Yagi and N. Yunes, Phys. Rept. 681, 1 (2017), arXiv:1608.02582 [gr-qc] .
  * Chia and Edwards (2020) H. S. Chia and T. D. P. Edwards, JCAP 11, 033 (2020), arXiv:2004.06729 [astro-ph.HE] .
  * (126) W. R. Inc., “Mathematica, Version 12.3.1,” Champaign, IL, 2021\.


11institutetext: University of Maryland, College Park, USA
11email<EMAIL_ADDRESS>
# Known by the Company it Keeps: Proximity-Based Indexing for
Physical Content in Archival Repositories

Douglas W. Oard  0000-0002-1696-0407

###### Abstract

Despite the plethora of born-digital content, vast troves of important content
remain accessible only on physical media such as paper or microfilm. The
traditional approach to indexing undigitized content is using manually created
metadata that describes content at some level of aggregation (e.g., folder,
box, or collection). Searchers led in this way to some subset of the content
often must then manually examine substantial quantities of physical media to
find what they are looking for. This paper proposes a complementary approach,
in which selective digitization of a small portion of the content is used as a
basis for proximity-based indexing as a way of bringing the user closer to the
specific content for which they are looking. Experiments with 35 boxes of
partially digitized US State Department records indicate that box-level
indexes built in this way can provide a useful basis for search.

###### Keywords:

Proximity-based indexing Archival access Physical media

##  1 Introduction

The storyteller Aesop wrote (in Greek) what has been translated as “a man is
known by the company he keeps” [1]. In sociology, Aesop’s claim is reflected
the concept of homophily, a dictionary definition of which is “the tendency to
form strong social connections with people who share one’s defining
characteristics, as age, gender, ethnicity, socioeconomic status, personal
beliefs, etc.” [14]. Homophily arises in many contexts, including between
people in social network analysis, in organizational dynamics [8], and more
metaphorically in, for example, the meaning of terms in natural language
processing [3].

Our thesis in this paper is that a form of homophily is to be expected among
the content found in archival repositories. Our basis for this is that
archivists respect the original order of archival content when performing
arrangement and description [18]. In the arrangement task, archivists organize
physical materials, typically by placing those materials in folders, placing
those folders in boxes, and grouping those boxes into series. Archival
materials are generally the records of some organization or individual.
Respecting the original order of those records can help to preserve the
evidence of the creator’s activities that is implicit in that original order.
Because archivists consider the evidentiary value of records to be on par with
their informational value, this is an important consideration. A second
consideration that argues for respecting original order is that doing so makes
it possible to open collections for research use with a minimum of work on the
archivist’s part. Because that original order was useful to the organization
or individual that created the records, it is reasonable to assume that users
of an archive who put in the effort to learn how a particular collection is
ordered will find some value in that ordering [24].

It is this respect for original order in archival arrangement that produces
the homophily that we leverage in this paper. Specifically, we hypothesize
that if we know something about the content of some records in some archival
unit (e.g., folder, box, series, or repository) then we can make some
plausible inferences about where certain other records that we have not yet
seen might be found. However, it is one thing to reason from first principles
that such a claim might be true, and quite another thing to show that such a
claim actually is true. In this paper, we show that the claim is true in one
specific case, and thus that it could be broadly true, although we leave
investigation of the broader question of how widely applicable our claim is
for future work.

##  2 Related Work

Rapid growth in digital content over the last half century has resulted in the
initial trickle of digital content reaching archival repositories now becoming
a flood. A broad range of tools can be used to find born-digital content, and
there is currently a good deal of innovation in that space (e.g., [11, 16]).
Many such tools could also be used to find content digitized from physical
media, such as paper or microfilm, but problems of cost and scale limit the
scope of digitization efforts. For example, in the first five months of 2003
the National Archives and Records Administration (NARA) digitized 13 million
pages from their holdings of 11.7 billion pages [15]. Even at that impressive
rate, 121,000 pages per day, it would take 375 years to digitize the paper
holdings of that one repository. Clearly, the problem of finding things on
paper will not be going away any time soon.

The first problem faced by someone wishing to find materials on paper is
knowing where to look. Citations in the scholarly literature play a
particularly prominent role in this process. For example, Tibbo found that 98%
of historians followed leads in the published literature [22], and Marsh, et
al. found that for anthropologists 73% did so [13]. There are also tools that
support search of descriptions created by archivists across multiple
repositories, such as ArchiveGrid [9]. Once a user knows where to look, their
next challenge is to learn how to find what they want there. As Tibbo notes,
it is common for scholars to write to or call archivists before visiting a
repository [22]. Scholars also make use of finding aids that have been created
by archivists to describe (among other things) the nature of the content in a
collection, and how that content is arranged and described. Although full-text
search of finding aids, which for example ArchiveGrid provides, can be useful,
in recent years the use of a metadata format called “Describing Archives: a
Content Standard” (DACS) has emerged as a preferred basis for searching the
results of the descriptions that archivists create [21].

One limitation of these approaches for finding content on physical media is
that they depend entirely on descriptions that are created by archivists.
However, the same cost and scale pressures that limit digitization also limit
the creation of detailed descriptions [10, 23]. Marsh, for example, notes that
of 314 collections in the Smithsonian Institution’s National Anthropological
Archives, only 25% had an online finding aid online as of 2019 [12]. A second
limitation is that, as Cox has pointed out, these methods for helping people
find archival content were originally designed with scholars in mind, but the
general public also makes extensive use of resources found in archives (e.g.,
for genealogical research), and such users might well need different types of
support [5]. For both of these reasons, we see value in creating techniques to
guess where specific materials that have not yet been digitized (or otherwise
richly described at the level of individual items) might be found. That is the
focus of our work in this paper.

##  3 The “Subject-Numeric Files”

In the United States, the Department of State is responsible for management of
foreign relations. Between 1963 and 1973, the Department of State maintained
its records on paper as “Subject-Numeric Files” [7]. Simplifying somewhat, in
this filing system the top-level category is one of 56 three-letter “primary
subject” code (e.g., POL for Political Affairs & Relations), the next level
category indicates a Country (e.g., Brazil), and the third-level category is a
numeric code, the meaning of which is specific to each primary subject (e.g.,
for POL, numeric code 15-1 designates the executive branch of government, and
27-12 designates war crimes). The entire collection includes about 8.6 million
pages, all presently held by the United States National Archives and Records
Administration (NARA) in College Park, Maryland.

In recent years, Brown University has engaged in a large-scale digitization
effort focused on archival records that can shed light on Brazilian politics.
As one part of this project, Brown University arranged for about 14,000 items
in NARA’s Department of State Subject-Numeric Files to be digitized. These
digitized records were all from the POL-Brazil section of the NARA records.
They represent parts of the content of a total of 52 boxes.

NARA intends to make these records available online, although the links from
the NARA catalog to most of these records are not presently working.
Fortunately, the Brown University Library makes almost all of the digitized
content from 36 of those boxes
available,111https://library.brown.edu/create/openingthearchives/en/
importantly using the same box identifiers. We wrote a crawler to download up
to 100 of those records from each of 35 of those boxes (the 36th box had only
two digitized files, too few to be useful for our experiments). We also
crawled the Brown University’s title metadata for each downloaded
document.222Brown University’s title metadata is often more concise than
NARA’s title metadata for the same document, which for example sometimes also
indicates document type. The documents that we downloaded were the first (at
most) 100 that Brown showed on the results page for each box; this was more
than enough for our experiments, and thus there was no need to increase the
server load by crawling more documents. About 2% of the PDF files that we
downloaded were not actually documents but rather forms that indicated that a
document was not available for scanning; we manually removed all such
documents that could be identified (either by the word “Withdrawal” in the
title metadata, or by viewing PDF files that were small enough—less than
400kB—to suggest that they might be a single page).

This process resulted in a total of 3,205 PDF documents, organized by their
original location at NARA in one of 35 boxes. The smallest number of documents
per box was 22 (for box 1925); the largest number was 100 (for eleven of the
boxes). These 35 boxe numbers are grouped in 8 numeric sequences (1900–1908,
1925–1934, 1936–1938, 1941–1944, 2129, 2131–2132, 3832–3835, 3737–3838).

Boxes in the NARA’s Department of State Subject-Numeric Files have no
identifying metadata beyond the box number, but a box consists of folders,
with the folders holding the actual documents, and Brown University metadata
does include the folder label for the folder in which the document was found,
and we crawled that metadata as well. We can therefore describe a box by the
union of the folder labels for the folders in that box. There were typically
three to six file units per box. For example, box 1902 contains folders with
the following labels:

    POL 2-3 BRAZ 01/01/1967 [2-3 indicates economic reports]
    POL 5 BRAZ 01/01/1967  [5 indicates law]
    POL 6 BRAZ 01/01/1967  [6 indicates people]

##  4 The “BoxFinder” Experiments

The PDF files created by Brown University are searchable, which means that
finding a digitized document can be done with any full-text search system, and
Brown University provides such a service on their website. The situation is
quite different, however, for content from those same boxes that has not yet
been digitized. When finding undigitized content is the goal, as is the case
in this paper, all that a user of NARA’s archive would have available is the
folder label. An archives user would need to request every box containing any
folder labeled with with a subject-numeric code and date that might be related
to their search goal. This is a slow process, since it takes NARA staff
several hours to deliver a requested box to a user of the archives in the
reading room, and it can easily take hours to examine the records in just one
box.

Our ultimate goal is to accelerate this process by recommending to a user of
the archive what box they should look in. We imagine they might use what we
will build one of two ways. In the first, they use it like Google—they type in
a query, and we recommend a box. In the second, they are looking at some
documents, and we recommend a box that we expect contains similar documents.

Whichever type of query we get, we built the box index in the same way. We
pick a few digitized documents from each box, then use the OCR text from those
documents to create an index that can be used to search for a box. The way we
do this is straightforward – we take all the OCR words from some number of
pages, starting at the front (e.g., just the first page, or the first two
pages) from some number of PDF documents (e.g., 3 documents) that we know
actually are in each box. This gives us an index in which there are 35 items
that can be found (the 35 boxes), each of which is represented by a single
long string of words. If we can use this index to find other documents that
are in that same box, then we will have shown that homophily is a useful basis
for search, and that a document in this collection can to some extent be
“known by the company it keeps.”

    Attached note from the Brazilian Embassy
    Background checks
    Biographic Information: Vistor Jose Faccione, State Deputy (ARENA) in
        Rio Grande do Sul
    Biographic Reporting
    Brazil Opts to Counter Poor Image Abroad
    Brazilian Laws
    Brazilian Leaders Publicly Downplay Brazil’s Alleged Desire for
        Hegemony
    Church Newspaper Calls for Democracy
    Closure of Brazilian Radio Stations
    Daily Media Reaction Report to ITT-Chile Situation
    Detention of Fernand Legros
    Disability and Death of a President
    Ernesto Geisel in Rio
    Ester Ferraz
    Foreign Minister Gibson Speaks on Brazil’s Foreign Policy
    GOB Announces Development Goals and Programs for 1970-73
    Ineligibilities Again
    Itamaraty’s Role in Formulation of Brazilian Foreign Policy
    Law on Party Loyalty Expected Soon
    Monthly Trends Report - December 1971

Figure 1: Some examples of short queries built from title metadata.

In our experiments, we don’t have a real user, so we simulate two search
scenarios. To simulate a “type a query” approach, we search using Brown
University’s metadata for the title field of some document (one that we
actually do have an image of, but one the system had not indexed). Figure 1
shows some examples. If the system can guess which box actually contains the
document from which we got the title, then we expect that it could also do
well if a real searcher ever typed a query like that. Of course, real
searchers might type in a query that’s better or worse than the document title
that we chose as a query, but at least this will indicate whether our
homophily-based approach can work when it gets a query like the one we gave
it.

    Department o
    PAGE 01
    47
    ACTION SS 7*
    INFO CIAE 00,/070 W RIO DE 01693 06I913Z LIMITED OFFICIAL USE
    I I 9070
    R 06I6T0Z MAR 69
    FM AMEMPASSY RIO HE JANFIRO
    TO SECSTATE WASHDC 7226
    LIMITED OFFICIAL USE RIO DE JANEIRO t693
    L I MO IS
    FOR VAKY
    SUBJECTi DELIVERY OF ROCKEFELLER LETTER
    I, IN RFSPONSE TO REQUEST FOR AUDIENCE WITH PRES WE WERE INFORMED
    THAT IT NOT CUSTOMARY HERE FOR PRES TO PECEIVF CHARGE D’AFFAIRES
    AND PERHAPS I COULD DELIVER IT VIA FONMIN. IT WAS STATED* HOWEVER.
    THAT EXCEPTION WOULD BE MADE IN CASE OF U.S. CHARGE IF I So
    REQUESTED.
    p. IN VIEW FACT I HAD NO OTHER MAJOR SUBSTANTIVE BUSINESS TO DIS-
    CUSS AND NOTHING T0 ADD TO CONTENTS OF LETTER, I ELFCTED Tn USE
    FONMIN CHANNEL FOR DELIVERY* ON THEORY THERE MIGHT SOON BE SIG-
    NIFICANT ITEMS i SHOULD TA<E UP DIRECTLY WITH PRES AND IT ADVIS-
    ABLE TO SAVr TICKET FOR SUCH AN OCCASION. LETTER WAS DELIVFPED TO
    FONMIN THIS MORNING* BELTON
    LIMITED OFFICIAL USE

Figure 2: Example of OCR text for the first page of a document.

To simulate the “see a document” query, we use the OCR text from the document
that we picked as the basis for the query, and see if the system can guess
which box it was from. Figure 2 shows an example of the OCR text from the
first page of a document. We call this the query-by-example condition. Note
that this results in a rather long query, usually a few hundred words.333For
efficiency reasons, in an actual search system we would want to do some query
term selection (e.g., [17]). To limit the complexity of our result
presentation, we consistently report results for some number of pages of OCR
text that are used both to form the query-by-example and to build the index
(e.g., for a box index built from the first page of several documents, our
query is built using only the first page of the query-by-example document),
although that need not be the case in practice.

Whichever type of query we get, we then use bag of words retrieval, ranked
with Okapi BM-25 term weights [20] (with $k_{1}=1.2$ and $b=0.75$), with the
Porter stemmer, to create a ranked list of the thirty five boxes, hopefully
with the correct box at or near the top.444We used our own implementation of
BM-25 ranking, which is available in the code distributed with our data. As
our measure of success, we count how many times our system (which we call
“BoxFinder”) guesses the correct box (i.e. the box that actually contains the
document that the query was made from). To easily compute a percentage, we
choose 100 query documents then then report how many of that 100 the system
got right. We can do this using any number of documents to describe each box,
for any number of pages from those documents, and for either way of making a
query. For example, as Table 2 shows, if we make the query from the OCR words
on the first page of the PDF file, and we use OCR words from the first page of
three (other) PDF files to describe each box, then the system is right 27.9%
of the time. We call this way of measuring BoxFinder’s results Top-1 (since it
is the percent of the time that BoxFinder places the correct answer at rank
1). If it were just guessing randomly, with no real idea which box to look in,
it would only be right at Top-1 2.9% of the time (i.e., once in every 35
tries).

From this we can conclude that BoxFinder is well named – it can indeed find
boxes. Of course, 27.9% is a long way from perfect, but BoxFinder need not be
perfect to be useful. Indeed, even when BoxFinder’s top result isn’t quite
right, the right answer is “close” in some sense more often than chance would
predict. For example, if we look at when BoxFinder’s first or second guess is
perfect (Top-2), one of those is right 40.4% of the time when we have three
training samples per box. Moreover, if we ask how often BoxFinder’s Top-1
result is within one box (e.g., ranking box 1903, 1904 or 1905 first when it
should have found box 1904), that happens 36.8% of the time with three
training samples per box (and we note that this additional benefit from
looking ”nearby” was obtained despite there being some gaps in the box
numbering in the test collection we have used).

Of course, guessing randomly is a low baseline. We can instead index a set of
terms generated from the labels on the folders in each box. To do this, we
must decode the subject-numeric codes. The State Department developed a
classification guide to help staff assign subject-numeric codes consistently,
and that guide is available from NARA [6, 7]. It is straightforward to replace
each code (e.g., POL 12-6) with the corresponding labels (in that case,
”POLITICAL PARTIES: Membership. Leaders.”).555The 1963 handbook [6] was
updated in 1965 [7]. A few codes had different labels in the two; in such
cases, we combined terms for that code from both handbooks. That’s a
combination of the label for POL 12 (“POLITICAL PARTIES”) and the label for
POL 12-6 (”Membership. Leaders.”) because the State Department classification
guide subheadings (in this case, POL 12-6) are meant to be interpreted in the
context of the corresponding main heading (in this case, POL 12).

Subject-Numeric codes sometimes also include abbreviations of the names of
countries (e.g., PAR for Paraguay, USSR for the Soviet Union, and US for the
United States of America), so we also extract and expand those names to a
single standard form (e.g., we do not also expand USSR to the Union of Soviet
Socialist Republics). However, we do not extract or expand “BRAZ” (Brazil),
which appears in every folder label in our collection, since its presence
everywhere would result in it having no beneficial effect on the ranking of
boxes. The folder labels also include dates, from which we extract and include
the year (e.g., 1964) with the metadata that we index. We elected not to
extract and include the month or day from the date because the distribution of
dates that we observe makes it clear that these are start dates for a folder,
but that a folder can contain documents from several months. We also elected
not to index the subject-numeric code itself, since we did not expect the
queries used in our experiments to contain such codes (although we note that
in a practical application, expert searchers might indeed understand and use
subject-numeric codes).

We can also optionally include any scope note text. For example The scope note
for POL 12-6 states “Includes party elections, purges of party, etc. Subdivide
by name of leader if volume warrants." Scope notes contain both on-topic terms
(such as “party elections”) and off-topic terms (e.g., “subdivide”), and scope
notes can even include negated terms (e.g., the scope note for POL-12 includes
“Exclude: Materials on … legislative matters, for which SEE: POL 15 -2”).
Experience from the Text Retrieval Conference (TREC) suggests removing negated
terms may have little effect on average, since although retaining negated
terms is sometimes harmful, they can also sometimes be helpful (because
negated terms often have some relation to the topic) [19].666The TREC
experiments compared retention to removal of negated query terms, but because
ranking relies on term matching we would expect similar results for retention
or removal of content terms from the items being indexed. Scope notes can also
contain guidance not related specifically to the topic (e.g., “if volume
warrants”), but because such guidance is found in many scope notes, those
common terms will have little effect on the way boxes are ranked. For these
reasons, when we include scope notes, we use the full scope note, with no
human editing. Unlike the labels, where we combine the labels for broad topics
(e.g., POL 12) and subtopics (POL 12-6), we use only one scope note (in this
case, for POL 12-6) because some topic-level scope notes indicate when the
topic-level category should be assigned in preference to a subcategory.

How well the system ranks boxes depends on which documents describe each box,
and which documents we pick as queries. We pick documents to represent each
box randomly from within each box, without replacement. To select query
documents, we first randomly select the box the query document will be found
in. We do this 100 times, with replacement. Then for each of those 100 choices
of boxes, we randomly select a query document from that box, being careful not
to choose documents that were used to describe that box. This can choose the
same query document twice in a set of 100, but such cases are rare. Because of
these random choices, BoxFinder system scores will vary depending on the
choices that we made, so we run the whole experiment 100 times (randomly
choosing the documents to represent each box again, and randomly choosing
query documents again), averaging those results to get the percentages we
report.

Code and data for the experiments is at https://github.com/oard/BoxFinder.

##  5 Results

With that background, we can now look at our full set of results. Table 1
summarizes the results for title metadata queries. As we might expect, having
more randomly sampled documents (“samples”) to represent a box yields better
results. Surprisingly perhaps, it’s generally better to use only the first
page of each document. One reason for this might be that some documents are
very short—many have only a single page—so we only get more pages from the few
documents that are longer.

| First Page |  $\leq$ 2 Pages |  $\leq$ 3 Pages |  $\leq$ 4 Pages |  $\leq$ 100 Pages
---|---|---|---|---|---
Samples | Top-1 | Top-2 | Top-1 | Top-2 | Top-1 | Top-2 | Top-1 | Top-2 | Top-1 | Top-2
1 | 10.7 | 16.9 | 10.5 | 17.1 | 9.5 | 15.5 | 9.3 | 15.5 | 9.2 | 15.4
2 | 13.0 | 21.1 | 12.2 | 19.4 | 10.8 | 18.1 | 11.6 | 18.6 | 10.9 | 18.0
3 | 14.6 | 22.1 | 14.1 | 21.5 | 12.9 | 20.5 | 13.0 | 20.2 | 11.7 | 19.2
4 | 15.6 | 23.6 | 14.9 | 23.2 | 13.8 | 21.0 | 13.6 | 21.4 | 12.9 | 20.3
6 | 16.9 | 25.4 | 15.8 | 24.3 | 13.7 | 22.4 | 13.8 | 21.9 | 12.2 | 20.6
8 | 16.6 | 25.0 | 16.0 | 24.5 | 14.8 | 23.1 | 14.4 | 22.6 | 12.9 | 21.3
10 | 18.1 | 27.1 | 15.6 | 24.5 | 15.0 | 23.8 | 15.0 | 23.5 | 13.3 | 22.2
Table 1: Results using short title metadata queries to search OCR text from sampled documents. Top-1: percent in exact box at rank 1. Top-2: percent in exact box at rank 1 or 2. | First Page |  $\leq$ 2 Pages |  $\leq$ 3 Pages |  $\leq$ 4 Pages | All Pages
---|---|---|---|---|---
Samples | Top-1 | Top-2 | Top-1 | Top-2 | Top-1 | Top-2 | Top-1 | Top-2 | Top-1 | Top-2
1 | 16.5 | 24.5 | 15.2 | 22.7 | 15.2 | 22.1 | 14.3 | 21.5 | 10.4 | 16.9
2 | 23.4 | 34.4 | 21.1 | 29.3 | 18.4 | 27.7 | 18.6 | 26.9 | 13.0 | 19.4
3 | 27.9 | 40.4 | 25.6 | 34.9 | 21.8 | 32,0 | 22.2 | 30.5 | 17.2 | 25.5
4 | 31.5 | 43.7 | 25.0 | 36.2 | 24.1 | 35,8 | 24.6 | 34.8 | 20.0 | 28.1
6 | 34.1 | 47.4 | 29.7 | 42.3 | 28.6 | 40.0 | 27,2 | 39.2 | 12.5 | 32.4
8 | 35.0 | 49.0 | 33.6 | 46.7 | 31.3 | 44.1 | 29.4 | 40.5 | 26.8 | 38.6
10 | 39.2 | 53.5 | 34.7 | 47.2 | 33.0 | 46.0 | 32.1 | 43.9 | 27.4 | 38.8
Table 2: Results using OCR query-by-example queries to search OCR text from
sampled documents.

Table 2 summarizes results for queries built using OCR text from the query
document, and Figure 3 illustrates those results for the Top-2 condition. As
can easily be seen, BoxFinder does better with these longer queries. One
reason for this is that with short queries (like the ones from title metadata)
BoxFinder sometimes finds no matching terms at all, but that doesn’t happen
very often with the longer queries based on full-text OCR. Of course, these
longer queries have all kinds of strange things in them (from letterhead,
message addresses, OCR errors, handwriting that gets misrecognized, etc.), and
the document representations suffer from the same problem. Nonetheless, there
clearly is a lot of signal here in the midst of all that noise, since
BoxFinder is doing better with these longer queries than with the title
metadata queries, and it is doing way better than random guessing with either
of them.

Figure 3: Learning rate for correct box in Top-2 as number of scanned training documents per box grows, scanned query-by-example condition. Each line represents a different maximum number of per-document scanned pages. Scanning just the first page is best. | Folder Labels | +Scope Notes
---|---|---
Queries | Repetitions | Top-1 | Top-2 | Top-1 | Top-2
Short titles | 4,200 | 12.4 | 17.1 | 12.4 | 18.6
First-page OCR | 100 | 5.0 | 9.5 | 5.3 | 9.0
Table 3: Results when searching terms generated from the folder labels, optionally including scope notes. | First Page |  $\leq$ 2 Pages |  $\leq$ 3 Pages |  $\leq$ 4 Pages |  $\leq$ 100 Pages
---|---|---|---|---|---
Samples | Top-1 | Top-2 | Top-1 | Top-2 | Top-1 | Top-2 | Top-1 | Top-2 | Top-1 | Top-2
1 | 14.0 | 21.5 | 14.0 | 21.8 | 13.3 | 21.0 | 13.1 | 20.5 | 13.3 | 20.3
2 | 15.5 | 23.4 | 14.8 | 22.7 | 13.9 | 21.8 | 14.8 | 22.6 | 14.5 | 23.0
3 | 15.8 | 23.5 | 15.5 | 23.7 | 15.1 | 23.4 | 15.1 | 23.0 | 14.5 | 23.0
4 | 16.6 | 25.2 | 16.8 | 24.5 | 16.2 | 24.1 | 16.5 | 24.0 | 15.0 | 23.1
6 | 17.6 | 25.7 | 17.2 | 25.1 | 15.8 | 23.9 | 16.1 | 24.1 | 15.1 | 23.5
8 | 17.7 | 25.9 | 17.3 | 25.3 | 17.7 | 25.6 | 16.6 | 23.8 | 15.4 | 23.5
10 | 19.1 | 27.5 | 17.4 | 25.7 | 17.1 | 25.1 | 16.7 | 25.1 | 16.1 | 24.6
Table 4: Reciprocal rank fusion results with short title metadata queries,
merging results from searching terms generated from folder labels (without
scope notes) and results from searching OCR text.

Table 3 shows that when short (title metadata) queries are used to search
documents that are represented only using terms generated from the folder
titles, results comparable to those obtained by using the same queries to
search a single page of OCR-generated text. The Top-1 results for searching
terms generated from folder labels using short queries was 12.4%, and at Top-2
that same condition had the correct answer in rank 1 or rank 2 17.1% of the
time. We note that only about 70% of the short queries include a term that
matches some term that results from expanding the subject-numeric codes found
in the folder labels (without also indexing the scope notes), but short
queries still did far better than the longer OCR-based queries when searching
document representations that are based solely on terms generated from folder
labels.

Interestingly, the pattern observed in Tables 1 and 2, where query-by-example
using OCR text was markedly better than using the short title metadata as a
query, is reversed here. Essentially the pattern that we see in these results
is that a matched condition (using OCR to search OCR, or using metadata to
search metadata) is consistently outperforming an unmatched condition. This
might be explained by systematic errors in the OCR or by systematic
differences in the way language is used in the documents and in the metadata.
We see a benefit at rank 2 from the inclusion of scope notes when generating
terms from folder titles to represent each box, but no benefit at rank 1.777To
measure the benefit of sope notes with short titles more accurately, we
average over 4,200 repetitions for our experiments with short queries in Table
3.

Of course, we need not use terms generated from the folder titles in
isolation—we can instead use the folder titles together with the OCR text from
the sampled documents. We could do this in one of two ways, either by
concatenating the two representations of each box, or by performing two
searches, one using each representation, and then performing result fusion to
create a single ranked list of boxes. That second approach, result fusion, is
the one we would expect to work better in this case because of the mismatched
document lengths, so that’s the approach we tried.

Table 4 shows reciprocal rank fusion [4] results when the (short) title
metadata is used as the query.888Cormack’s implementation has a parameter that
controls the rank discount rate. Cormack recommends a value of 60, and we used
that value in our experiments. As with Tables 1 and 2, these are averages over
100 repetitions. With these short queries, it is sometimes the case that no
terms match in the query, resulting in no ranking of the boxes by one of the
systems. In such cases, we retain the other ranking unchanged.999When neither
approach has a term match, we generate an empty list. As can be seen by
comparing Tables 1 and 4, this rank fusion approach results a in an
improvement over what we achieved using metadata alone. This improvement is
both substantial (for example, compare 15.8 at Top-1 for Reciprocal Rank
Fusion with the first page from each of 3 samples in Table 4 to to 12.4 for
folder metadata alone in Table 3, a 27% relative improvement; the relative
improvement at Top-2 is 38%) and statistically significant (the standard
deviation over 42 100-sample averages when searching terms generated from
folder labels is 0.33 at both Top-1 and Top-2).

##  6 Conclusion and Future Work

We close by observing that we have shown that the homophily between digitized
and undigitized content that we expected to find in an archival collection
does indeed provide a useful signal that can help to improve search for
content that has not yet been digitized. There are several ways in which we
might push this work further. One thing to try would be to be selective about
which parts of a document image to index. For that, we could pay attention not
just to the OCR, but to cues from the layout of the words on the page. For
example, we might pay particular attention to who sent or received a document,
or to the date of the document. We could also use layout analysis to determine
what type of document we are looking at (e.g., telegram, letter, memo, or
form), and then apply type-specific information extraction, and possibly even
type-specific ranking. Speaking of ranking, there’s no reason why we need to
glom the OCR text from different documents together to make a single
representation for each box. Instead, we could make multiple representations,
one per document, and then let those representations vote on which box should
be chosen. That approach has, for example, been shown to work well for blog
search [2].

There is actually nothing in BoxFinder that is specific to boxes except the
way we tested it; the same ideas could work for folders, series, collections,
and even entire repositories (e.g., if the Massachusetts Historical Society
has one letter from Paul Revere, perhaps it also has other things involving
Paul Revere). Of course, some of the tuning decisions (e.g., how many
digitized documents are needed to represent a folder?) will likely differ when
applying these ideas at different scales. But tuning would not be hard if we
had a collection to tune on. So one key to making BoxFinder into FolderFinder
(or SeriesFinder, or …) is to assemble appropriate collections on which we can
train and test. For our experiments for this paper, we assembled a single
collection and then used it to see how well our approach of representing a box
using OCR from randomly selected documents would do. But when tuning a large
number of system details, we’ll want training, devtest and evaluation
partitions, so we’ll need larger collections. Fortunately, the complete NARA
Department of State Subject-Numeric files are indeed much larger than the part
of that collection that we have used so far, so there is at least one good
source for such a collection. But if we want to know how well these ideas work
in general, we’re going to need several collections, from a variety of
sources. So assembling several such collections is a natural next step.

Finally, searching based on digitized documents will only get us so far; to go
further we’ll need other sources of evidence for where to look. Archival
metadata can sometimes help with that, as in the example we provided using
folder labels. So another next step would be to perform experiments with that
metadata at different scales (e.g, collection, box, folder, and, where
available, item).

## Acknowledgements

The author appreciates comments from Katrina Fenlon, Emi Ishita, Diana Marsh,
Toshiyuki Shimizu, Tokinori Suzuki and Victoria Van Hyning on earlier drafts.

## References

  * [1] Aesop: The Complete Fables. Penguin Classics (1998)
  * [2] Balog, K., Azzopardi, L., de Rijke, M.: Formal models for expert finding in enterprise corpora. In: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 43–50 (2006)
  * [3] Brunila, M., LaViolette, J.: What company do words keep? revisiting the distributional semantics of JR Firth & Zellig Harris. In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 4403–4417 (2022)
  * [4] Cormack, G.V., Clarke, C.L., Buettcher, S.: Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In: Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 758–759 (2009)
  * [5] Cox, R.J., et al.: Machines in the archives: Technology and the coming transformation of archival reference. First Monday (2007)
  * [6] Department of State: Records classification handbook. Available at https://www.archives.gov/files/research/foreign-policy/state-dept/finding-aids/records-classification-handbook-1963.pdf, visited May 10, 2023 (1963)
  * [7] Department of State: Records classification handbook. Available at https://www.archives.gov/files/research/foreign-policy/state-dept/finding-aids/dos-records-classification-handbook-1965-1973.pdf, visited February 8, 2023 (1965)
  * [8] Ertug, G., Brennecke, J., Kovacs, B., Zou, T.: What does homophily do? a review of the consequences of homophily. Academy of Management Annals 16(1), 38–69 (2022)
  * [9] Falk, P.K.: Archivegrid. Technical Services Quarterly 34(2), 218–219 (2017)
  * [10] Greene, M., Meissner, D.: More product, less process: Revamping traditional archival processing. The American Archivist 68(2), 208–263 (2005)
  * [11] Kiusals, D., Leetaru, K.: The declassification engine: The sphere of influence. Website http://diploglobe.declassification-engine.org/, visited May 12, 2023 (2015)
  * [12] Marsh, D.E.: Research-driven approaches to improving archival discovery. IASSIST Quarterly 43(2), 1–9 (2019)
  * [13] Marsh, D.E., St. Andre, S., Wagner, T., Bell, J.A.: Attitudes and uses of archival materials among science-based anthropologists. Archival Science pp. 1–25 (2023)
  * [14] McPherson, M., Smith-Lovin, L., Cook, J.M.: Birds of a feather: Homophily in social networks. Annual Review of Sociology 27(1), 415–444 (2001). https://doi.org/10.1146/annurev.soc.27.1.415, http://arjournals.annualreviews.org/doi/abs/10.1146/annurev.soc.27.1.415
  * [15] National Archives and Records Administration: Record group explorer. Website https://www.archives.gov/findingaid/record-group-explorer, visited February 2, 2023 (Jan 2023)
  * [16] O’Connell, K.: The future of email archives: A report from the task force on technical approaches for email archives. The American Archivist 82(1), 214–217 (2019)
  * [17] Paik, J.H., Oard, D.W.: A fixed-point method for weighting terms in verbose informational queries. In: Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management. pp. 131–140 (2014)
  * [18] Prom, C., Frusciano, T., of American Archivists, S.: Archival Arrangement and Description. Trends in archives practice, Society of American Archivists (2013)
  * [19] Robertson, S.E., Walker, S.: Okapi/Keenbow at TREC-8. In: Proceedings of the Eighth Text Retrieval Conference. pp. 151–162 (1999)
  * [20] Robertson, S.E., Zaragoza, H.: The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval 3(4), 333–389 (2009). https://doi.org/10.1561/1500000019, https://doi.org/10.1561/1500000019
  * [21] Society of American Archivists: Describing archives: A content standard, version 2022. Website https://saa-ts-dacs.github.io/, visited May 12, 2023 (2022)
  * [22] Tibbo, H.: Primarily History in America: How U.S. Historians Search for Primary Materials at the Dawn of the Digital Age. The American Archivist 66(1), 9–50 (01 2003). https://doi.org/10.17723/aarc.66.1.b120370l1g718n74, https://doi.org/10.17723/aarc.66.1.b120370l1g718n74
  * [23] Trace, C.B.: Archival infrastructure and the information backlog. Archival Science 22(1), 75–93 (2022)
  * [24] Wiedeman, G.: The historical hazards of finding aids. The American Archivist 82(2), 381–420 (2019)


# LMPT: Prompt Tuning with Class-Specific Embedding Loss for Long-Tailed
Multi-Label Visual Recognition

Peng Xia1, 2, 4 Di Xu5 Lie Ju1, 2, 3,  Ming Hu1, 2, 3 Jun Chen6 Zongyuan Ge1,
2, 3

1 AIM Lab, Faculty of IT, Monash University, Australia
2 Airdoc-Monash Research, Airdoc, China
3 Faculty of Engineering, Monash University, Australia
4 School of Computer Science and Technology, Soochow University, China
5 Earth Science and Engineering, Imperial College London, UK
6 King Abdullah University of Science and Technology (KAUST), Saudi Arabia
richard.peng.xia@gmail.com,julie334600<EMAIL_ADDRESS>
https://www.monash.edu/mmai-group Corresponding author

###### Abstract

Long-tailed multi-label visual recognition (LTML) task is a highly challenging
task due to the label co-occurrence and imbalanced data distribution. In this
work, we propose a unified framework for LTML, namely prompt tuning with
class-specific embedding loss (LMPT), capturing the semantic feature
interactions between categories by combining text and image modality data and
improving the performance synchronously on both head and tail classes.
Specifically, LMPT introduces the embedding loss function with class-aware
soft margin and re-weighting to learn class-specific contexts with the benefit
of textual descriptions (captions), which could help establish semantic
relationships between classes, especially between the head and tail classes.
Furthermore, taking into account the class imbalance, the distribution-
balanced loss is adopted as the classification loss function to further
improve the performance on the tail classes without compromising head classes.
Extensive experiments are conducted on VOC-LT and COCO-LT datasets, which
demonstrates that the proposed method significantly surpasses the previous
state-of-the-art methods and zero-shot CLIP in LTML. Our codes are fully
available at https://github.com/richard-peng-xia/LMPT.

##  1 Introduction

Figure 1: The class distribution is long-tailed and the VL model compares
image embeddings$\star$ to text
embeddings$\bullet$$\blacksquare$$\blacktriangle$ of the category, which means
the closer the distance between the text embeddings and the image embeddings,
the higher the probability that the category of the text embeddings matches
the image. (a) Person and horse in the image belongs to the head classes and
the tail classes respectively. (b) Zero-Shot CLIP. (c) Exsiting Prompt Tuning
w/o CSE loss. (d) LMPT (Ours) w/ CSE loss.

Long-tailed multi-label visual recognition (LTML) [43, 11] is a common and
practical task owing to the highly imbalanced data distribution [50] and
diverse objects of real-world images [41, 17]. Compared with long-tailed
recognition and multi-label recognition task, LTML is more complex and
challenging, because it requires capturing multiple categories and the label
co-occurrence in individual images [3, 17], which needs to compensate for the
negative impacts caused by the long-tailed distribution (i.e., low performance
on the tail classes).

Several approaches have been proposed to address the LTML problem from
different perspectives, such as re-sampling [1, 7, 11], re-weighting [2, 43]
and modeling more powerful structures [3, 39, 41]. Despite their great
contributions, these works neglect to take into account two crucial aspects.
First of all, the importance of semantic feature interaction between classes
to capture label co-occurrence. However, these methods are limited to
balancing the distribution of categories from the perspective of samples,
without considering the feature correlation between different classes. Second,
synchronous improvements in head-to-tail category performance, while some of
these works improve the performance of tail classes at the expense of the head
classes.

Recently, graphic models have been introduced to model the semantic label
correlation in a few works [3, 39], whereas these works are complex and
modeling label dependencies mainly based on the image modality without
additional semantic information from other modal data. Vision-Language (VL)
models [27, 16, 37, 15] demonstrate the huge potential of text modality on
semantic context feature for downstream visual tasks, especially for the
prompt tuning methods [31, 33, 45], which provide an efficient way to transfer
pre-trained VL models to downstream tasks by learning the task-specific
prompts rather than finetuning the entire model. Nonetheless, the existing
prompt tuning methods [53, 52, 35] for visual recognition simply minimize
prediction errors using the classification loss (e.g., cross-entropy loss)
with respect to the learnable prompts, which may lead to learning general
embeddings or inaccurate class-related embeddings. For instance, as shown in
Fig. 1, when presented with an image that contains both a head class
$\left[\mathrm{person}\right]$ and a tail class $\left[\mathrm{horse}\right]$,
the zero-shot method (Fig. 1) relies solely on the rich knowledge of the pre-
trained VL model to assess the similarity between the image and the word
embeddings of the class names, while the existing prompt tuning method (Fig.
1) further learns more generalized prompt tokens to improve model performance.
However, these methods do not consider the inter-class relationships,
particularly between head and tail classes, which is a critical factor for
LTML. This underscores the need for approaches that incorporate such
relationships to improve performance in such scenarios.

Therefore, to address these issues, we present the class-specific embedding
loss for prompt tuning on long-tailed multi-label visual recognition, called
LMPT. The abundance of image-caption data facilitates prompt learning that
encompasses more nuanced and specific textual descriptions, as well as the
semantic inter-dependencies between categories (Fig. 1) that share
information, such as similar features or common descriptions. This attribute
is particularly critical in the identification of both head and tail classes.
More specifically, we propose the class-specific embedding loss to enhance the
inclusivity of class-related embeddings within prompts. By gradually
approaching the embeddings of the corresponding caption, our proposed approach
enables prompt tokens to effectively judge the association between different
classes with the aid of textual modality. Aiming for class imbalance and the
consistency improvements between head classes and tail classes, we integrate
class-aware soft margin and re-weighting into the class-specific embedding
loss, which serves to assign larger margins and more weights to tail classes.
Notably, for images containing both head and tail classes, our approach
outperforms visual models and current prompt tuning methods. Moreover, we
adopt the distribution-balanced loss [43] as the classification loss. To
verify the effectiveness of our method, we conduct extensive experiments on
the two benchmarks, including VOC-LT and COCO-LT [43, 21, 10]. Experimental
results show that our method surpasses zero-shot CLIP [27] and previous SOTA
[11] on the above benchmarks.

To sum up, the main contributions of this work include:

  * •

We propose a novel LMPT framework to adapt pre-trained VL models (e.g., CLIP)
to tackle long-tailed multi-label visual recognition, where text descriptions
(captions) are easily accessible from public image-caption datasets or
generated by powerful image-caption models [40].

  * •

We present the class-specific embedding loss with class-aware soft margin and
re-weighting to learn more fine-grained and class-related embeddings that
build semantic relationships across head and tail classes with shared semantic
information. Such design can benefit performance in tail classes and hard-to-
recognize classes with the help of text modality.

  * •

We verify the effectiveness of the proposed method on VOC-LT, and COCO-LT by
achieving new state-of-the-art results on two data sets, which outperform
previous SOTA [11] by 9 / 6 % and zero-shot CLIP by 6 / 2 % on VOC-LT / COCO-
LT.

Figure 2: Overview of the architecture of our proposed method.

##  2 Related Work

###  2.1 Long-Tailed Visual Recognition

Real-world training data usually exhibits long-tailed distribution [50], which
presents a challenge for traditional methods due to the imbalanced class
distribution. To address this problem, several approaches [5, 25, 26, 30] have
been proposed from different aspects. One common method is to directly re-
sample the training data to balance the class distribution [9, 1, 7], by
adjusting the sampling rate of head classes and tail classes, yet it might
lead to the overfitting of tail classes. A better solution is to design re-
weighted loss functions [19, 14, 2] that assign more weight to tail classes or
ignore negative gradients [36] for tail classes. In addition, researchers also
propose to use techniques such as transfer learning [23, 54] and self-
supervised learning [18, 49] to alleviate the class imbalance problem.
Recently, some studies [24, 37] also explore the possibility of text modality
by refining visual-language representations on the long-tailed recognition
tasks.

###  2.2 Multi-Label Visual Recognition

For multi-label visual recognition, some early methods include treating it as
multiple binary image classifications [38, 48] or finding k-nearest neighbors
[47]. To locate regions of interest, some researchers [39, 41] proposed to
introduce recurrent neural networks (e.g., RNN, LSTM) to learn a joint image-
label embedding. In addition, Chen et al. [3] proposed to model the label
correlations by constructing a graph based on the label co-occurrence and Ye
et al. [46] updated static graph to dynamic graph convolutional network (GCN)
for robust representation. Wu et al. [43] proposed a distribution-balanced
loss and Guo et al. [11] adopted collaborative training on the uniform and re-
balanced samplings to alleviate the class imbalanced problem. There is also a
popular trend to align between visual and textual features [44, 22, 15, 29]
for multi-label recognition.

###  2.3 Prompt Tuning for Vision-Language Models

Prompt tuning [31, 33, 45] is a parameter-efficient technique used to utilize
the representation ability of pre-trained vision-language models to achieve
better performance instead of fine-tuning the whole model on downstream tasks.
Meanwhile, large-scale vision-language models (e.g., CLIP [27], ALIGN [16])
have demonstrated impressive power to learn visual and textual features. CoOp
[53] learns soft prompts via minimizing the classification loss and CoCoOp
[52] further formulates the prompts in an image-conditional way to improve its
generalization to unseen classes. DualCoOp [35] firstly adapts CLIP to multi-
label image recognition by learning pairs of positive and negative prompts for
each class. Different from the above work, LMPT focuses on exploring the
transfer ability to address long-tailed multi-label visual recognition.

##  3 Methodology

In this section, we present our proposed prompting tuning method, i.e., LMPT,
for adapting pre-trained vision-language models for long-tailed multi-label
visual recognition.

###  3.1 Preliminaries

Consider $\mathcal{D}$ as the dataset we use, ${N}$ as the number of the
dataset, $C$ as the number of classes, and $L$ as the fixed length of contexts
for optimization. Then $(x^{k},y^{k},t^{k})\in\mathcal{D}_{train}$,
${k}\in\left\\{1,...,N\right\\}$, where $x^{k}$ is an input single image,
$y^{k}=\left[y_{1}^{k},...,y_{C}^{k}\right]\in{\left\\{0,1\right\\}}^{C}$ is
the multi-label ground-truth and $t^{k}=\left[t_{1}^{k},...,t_{L}^{k}\right]$
is the corresponding text embedding of text description (caption). But during
the test phase, only $(x^{k},y^{k})\in\mathcal{D}_{test}$. Let
$n_{i}=\sum_{k=1}^{N}y_{i}^{k}$ denote the number of training examples that
contain class $i$. Please note that labels for computing the class-specific
embedding loss need to be processed into
$\tilde{y}^{k}=\left[\tilde{y}_{1}^{k},...,\tilde{y}_{C}^{k}\right]=\left[2*y_{1}^{k}-1,...,2*y_{C}^{k}-1\right]\in{\left\\{-1,1\right\\}}^{C}$,
where $\left\\{-1,1\right\\}$ indicates negative and positive pairs
respectively.

###  3.2 Approach Overview

In order to make effective use of the linguistic modality in the long-tailed
multi-label visual recognition task, we propose a novel framework (i.e.,
LMPT), as depicted in Fig. 2. Text encoder from the pre-trained CLIP is used
to encode the prompts and text descriptions (captions) of images. Only the
parameters in the prompts are optimized, while the text encoder and image
encoder are both kept frozen. We introduce two sorts of trainable prompts to
obtain class embedding, which are jointly optimized by the classification loss
$\mathcal{L}_{cls}$ and class-specific embedding loss $\mathcal{L}_{cse}$.
Details of the aforementioned loss functions will be introduced in the later
sections.

###  3.3 Prompt Tuning

Formally, the vision-language model consists of an image encoder
$\boldsymbol{f}(\cdot)$ and a text encoder $\boldsymbol{g}(\cdot)$. Following
[52], a prompt is defined as:

$o_{i}|_{1}^{M}=\left[\mathrm{V}\right]_{1}\left[\mathrm{V}\right]_{2}...\left[\mathrm{V}\right]_{m}...\left[\mathrm{V}\right]_{M}\left[\mathrm{CLASS}\right],$
(1)

where $i\in\left\\{1,...,C\right\\}$, $m\in\left\\{1,...,M\right\\}$, the
$\left[\mathrm{CLASS}\right]$ token is replaced by the specific class name
(e.g., “cat,” “dog”, “car”), each $\left[\mathrm{V}\right]_{m}$ is a learnable
word embedding with the same dimension as normal word embeddings in the
vocabulary (i.e., 512 for CLIP), and $M$ is a hyper-parameter specifying the
number of context tokens. The prediction probability (classification output)
$z$ is then computed as:

$p(y=i\mid
x)=\frac{\exp\left(\cos\left(\boldsymbol{g}\left(o_{i}\right),\boldsymbol{f}\left(x\right)\right)/\tau\right)}{\sum_{j=1}^{C}\exp\left(\cos\left(\boldsymbol{g}\left(o_{j}\right),\boldsymbol{f}\left(x\right)\right)/\tau\right)},$
(2)

where $\tau$ is a temperature parameter learned by CLIP and $cos(\cdot,\cdot)$
represents cosine similarity.

###  3.4 Class-Specific Embedding Loss

Inspired by hinge embedding loss [34] which is widely exploited for measuring
whether two inputs are similar or not, we introduce the class-specific
embedding (CSE) loss to optimize the trainable fine-grained instance prompts
by learning from text embeddings of captions. It tries to minimize the cosine
distance of matching patches and to increase the cosine distance of non-
matching patches above the margin. Embedding loss is then computed as

$\displaystyle{\ell}_{ebd}$
$\displaystyle=\begin{cases}\Delta_{i}^{k},&\text{if}\quad\tilde{y}_{i}^{k}=1,\\\
\max\left(0,\mu-\Delta_{i}^{k}\right),&\text{if}\quad\tilde{y}_{i}^{k}=-1,\end{cases}$
(3) $\displaystyle\Delta_{i}^{k}=1-\cos\left(t_{i}^{k},o_{i}|_{m}^{M}\right),$

where $\mu$ is the margin factor. Intuitively the embedding loss penalizes
positive (i.e., prompts of matching classes) pairs that have large distances
and negative (i.e., prompts of non-matching classes) pairs that have small
distance (less than $\mu$).

Figure 3: The class margins (dotted lines) are enforced for generated samples
by updating the decision boundary with respect to class margins. Blue samples
(head classes) are classified incorrectly and the model update gradient is
shown with pointed arrows. Green samples (medium classes) are classified
correctly outside of the margin and the gradient is shown. Therefore, the
embedding loss does not give special consideration to the minority categories,
but with the help of class-aware soft margin, the trade-off of $\mu_{1}$ can
be optimized by shifting the decision boundary to encourage the tail classes
to have larger margins. So yellow samples (tail classes) are classified
correctly outside of the original margin but within the enlarged margin, and
the embedding loss has no gradient for these samples.

LDAM [2] has inspired the development of a decision boundary that is both
robust and generalizable, capable of accurately classifying features that vary
within a certain range. However, when applied to long-tailed datasets
characterized by a significant class imbalance, models tend to exhibit greater
sensitivity to more frequent classes. As a result, the performance of these
models in less frequent classes is often poor. To address this issue, as
illustrated in Fig. 3, CSE loss also incorporates the class-aware soft margin
strategy to encourage the model to have the optimal trade-off between per-
class margins by stimulating the minority classes to have larger margins,
which can be viewed as regularization [42]. Following the trade-off between
the class margins [2], we adopt a class-aware margin for multiple classes of
the form

$\widetilde{\mu_{i}}\propto n_{i}^{-1/4}=\frac{\eta}{n_{i}^{1/4}}.$ (4)

Here $\eta$ is a hyper-parameter to be tuned. Therefore, when $y_{i}^{k}=-1$,
the loss can be computed as
$\max\left\\{0,\widetilde{\mu_{i}}-\Delta_{i}^{k}\right\\}$.

Meanwhile, our loss can be combined with a re-weighting strategy to be more
efficient when it comes to long-tailed distribution data. We then define the
reference weight based on the empirical class frequencies
$\left\\{n_{1},...,n_{C}\right\\}$ on the training set:

$w_{i}=\frac{\left(1/n_{i}\right)^{\gamma}}{\sum_{i=1}^{C}\left(1/n_{i}\right)^{\gamma}},$
(5)

where $\gamma$ is a scale hyper-parameter to provide more flexibility. Hence,
the re-weighted class-specific embedding loss is defined as:

$\displaystyle{\ell}_{cse}$
$\displaystyle=\begin{cases}w_{i}\Delta_{i}^{k},&\text{if}\quad\tilde{y}_{i}^{k}=1,\\\
\max\left\\{0,w_{i}\left(\widetilde{\mu_{i}}-\Delta_{i}^{k}\right)\right\\},&\text{if}\quad\tilde{y}_{i}^{k}=-1,\end{cases}$
(6)

$\mathcal{L}_{cse}=\frac{\sum_{k=1}^{N}{\ell}_{cse}}{N}.$ (7)

###  3.5 Multi-Label Classification Loss

Our method can be easily combined with the existing multi-label classification
loss functions [28, 20, 6, 43], regardless of whether they are designed for
long-tailed distributions or not. By blending the classification loss
functions with our proposed CSE loss, our method facilitates prompt learning
of more refined class descriptions and semantic relationships between
categories, particularly between head and tail classes.

In this study, we introduce the distribution-balanced loss [43] as the
classification loss function, which can be formulated as:

$r=\alpha+\sigma\left(\beta\times\left(\frac{\frac{1}{n_{i}}}{\sum_{i=1}^{C}\frac{1}{n_{i}}}-\theta\right)\right),$
(8) $v_{i}=-\kappa\times-\log\left(\frac{1}{n_{i}/N}-1\right),$ (9)
$\displaystyle{\ell}_{cls}$
$\displaystyle=\begin{cases}-r\left(1-q_{i}^{k}\right)^{\gamma}\log\left(q_{i}^{k}\right),&\text{if}\quad
y_{i}^{k}=1,\\\
-\frac{r}{\zeta}\left(q_{i}^{k}\right)^{\gamma}\log\left(1-q_{i}^{k}\right),&\text{if}\quad
y_{i}^{k}=-1,\end{cases}$ (10)

where $q_{i}^{k}=\sigma\left(z_{i}^{k}-v_{i}\right)$ is for positive
instances, $q_{i}^{k}=\sigma\left(\zeta\left(z_{i}^{k}-v_{i}\right)\right)$ is
for negative ones and $\alpha,\beta,\theta,\kappa,\zeta$ are hyperparameters.
Then $\mathcal{L}_{cls}=\sum_{k=1}^{N}{\ell}_{cls}/N$.

Hence, the overall training loss can be written as:

$\mathcal{L}=\lambda\mathcal{L}_{cls}+\left(1-\lambda\mathcal{L}_{cse}\right),$
(11)

where $\lambda\in[0,1]$ is a hyperparameter to balance $\mathcal{L}_{cls}$ and
$\mathcal{L}_{cse}$.

Datasets | VOC-LT | COCO-LT
---|---|---
Methods | total | head | medium | tail | total | head | medium | tail
Backbone: RN-50 |  |  |  |  |  |  |  |
ERM | 70.86 | 68.91 | 80.20 | 65.31 | 41.27 | 48.48 | 49.06 | 24.25
RW | 74.70 | 67.58 | 82.81 | 73.96 | 42.27 | 48.62 | 45.80 | 32.02
Focal Loss [20] | 73.88 | 69.41 | 81.43 | 71.56 | 49.46 | 49.80 | 54.77 | 42.14
RS [32] | 75.38 | 70.95 | 82.94 | 73.05 | 46.97 | 47.58 | 50.55 | 41.70
ML-GCN [4] | 68.92 | 70.14 | 76.41 | 62.39 | 44.24 | 44.04 | 48.36 | 38.96
OLTR [23] | 71.02 | 70.31 | 79.80 | 64.95 | 45.83 | 47.45 | 50.63 | 38.05
LDAM [2] | 70.73 | 68.73 | 80.38 | 69.09 | 40.53 | 48.77 | 48.38 | 22.92
CB Focal [6] | 75.24 | 70.30 | 83.53 | 72.74 | 49.06 | 47.91 | 53.01 | 44.85
BBN [51] | 73.37 | 71.31 | 81.76 | 68.62 | 50.00 | 49.79 | 53.99 | 44.91
DB Focal [43] | 78.94 | 73.22 | 84.18 | 79.30 | 53.55 | 51.13 | 57.05 | 51.06
LTML [11] | 81.44 | 75.68 | 85.53 | 82.69 | 56.90 | 54.13 | 60.59 | 54.47
[1pt/1pt] Zero-Shot CLIP | 84.30 | 63.60 | 88.03 | 97.03 | 56.19 | 35.73 | 60.52 | 68.45
CoOp [53] | 81.34 | 65.10 | 81.54 | 93.37 | 54.94 | 38.06 | 56.67 | 67.51
CoCoOp [52] | 78.63 | 64.33 | 80.51 | 87.94 | 46.02 | 36.02 | 50.57 | 48.82
DualCoOp [35] | 81.03 | 66.45 | 80.53 | 92.33 | 53.11 | 40.48 | 55.20 | 62.11
LMPT (ours) | 85.44 | 66.62 | 88.11 | 97.86 | 58.97 | 41.87 | 61.60 | 69.60
Backbone: ViT-B/16 |  |  |  |  |  |  |  |
Zero-Shot CLIP | 85.77 | 66.52 | 88.93 | 97.83 | 60.17 | 38.52 | 65.06 | 72.28
CoOp [53] | 86.02 | 67.71 | 88.79 | 97.67 | 60.68 | 41.97 | 63.18 | 73.85
CoCoOp [52] | 84.47 | 64.58 | 87.82 | 96.88 | 61.49 | 39.81 | 64.63 | 76.42
LMPT (ours) | 87.88 | 72.10 | 89.26 | 98.49 | 66.19 | 44.89 | 69.80 | 79.08
Table 1: mAP performance of the proposed method and comparison methods. Above
the dotted line is the performance of visual models and below is that of
vision-language models.

##  4 Experiments

###  4.1 Datasets

Following [43], we conduct experiments on two datasets for long-tailed multi-
label visual recognition: VOC-LT and COCO-LT [43]. They are artificially
sampled from two multi-label recognition benchmarks, PascalVOC [10] and MS-
COCO [21], respectively.

VOC-LT is created from the 2012 train-val set of PascalVOC based on the
settings in [43]. The training set contains 1,142 images and 20 class labels,
and the number of images per class ranges from 4 to 775. All the classes are
split into three groups according to the number of training samples per class:
a head class has more than 100 samples, a medium class has 20 to 100 samples,
and a tail class has less than 20 samples. The ratio of head, medium, and tail
classes after splitting is 6:6:8. The text description of each image in the
training set is generated by OFA111We used the ofa_image_caption_coco_huge_en
as the image-caption model through ModelScope (https://modelscope.cn/). [40].
VOC2007 test set with 4,952 images is used to evaluate the performance as the
test set.

COCO-LT is constructed from MS-COCO 2017 in a similar way. The training set of
this dataset contains 1,909 images and 80 class labels, and the number of
images per class ranges from 6 to 1,128. The ratio of head, medium, and tail
classes is 22:33:25 in COCO-LT. We use captions as text descriptions of each
image in the training set from MS-COCO which is a public image caption dataset
as well. The test set is MS-COCO 2017 test set with 5,000 images.

###  4.2 Experimental Settings

Evaluation Metrics. As in [23], the classes are split into three groups by the
number of their training examples: head classes each contain over 100 samples,
medium classes each has between 20 and 100 samples, and tail classes with
under 20 samples each. We use mean average precision (mAP) to evaluate the
performance of long-tailed multi-label visual recognition for all the classes.

Implementation Details. We adopt CLIP ResNet-50 [13] or ViT-Base/16 [8] as the
visual encoder and use the corresponding CLIP Transformer as the text encoder.
During training, the parameters of both the two encoders are kept frozen, and
only learnable prompts are optimized. SGD optimizer is adopted to learn prompt
tokens, and the training epochs are set to 30. The learning rates for COCO-LT,
and VOC-LT are empirically initialized with 1e-4, 5e-4, and decay by the
cosine annealing rule during training. For loss functions, $\eta$ in Eq. 4,
$\gamma$ in Eq. 5 and $\lambda$ in Eq. 11 are set as 1.0, 1.0 and 0.5,
respectively. Other hyperparameters in DB loss are set as the same as [43].
All the experiments are implemented on PyTorch 1.13.0 using an NVIDIA GeForce
RTX 3090 GPU.

###  4.3 Comparison with Prior SOTA Works

To evaluate the effectiveness of the proposed method, firstly we compare it
with previous methods of visual models on the two long-tailed multi-label
datasets. The compared methods include Empirical Risk Minimization (ERM), a
smooth version of Re-Weighting (RW) using the inverse proportion to the square
root of class frequency, Re-Sampling (RS) [32], Focal Loss [20], ML-GCN [4],
OLTR [23], LDAM [2], Class-Balanced (CB) Focal [6], BBN [51], Distribution-
Balanced (DB) Focal [43] and LTML [11]. The mAP performance of different
methods is shown in Table 1. The prior best performance is achieved by LTML –
mAP of 81.44% over all classes on VOC-LT and 56.90% over all classes on COCO-
LT.

Furthermore, we compare zero-shot and few-shot methods based on CLIP on the
two benchmarks. The mAP performance of these methods is shown in Table 1 as
well. For a fair comparison, we initialize the prompt as the default hand-
crafted one “a photo of a” for all the methods. The results show that when
using ViT-Base/16 as the backbone, even the overall mAP performance of zero-
shot CLIP reaches 85.77% and 60.17%, which outperforms previous SOTA LTML by
4.33 points (85.77% vs.81.44%) and 3.27 points (60.17% vs.56.90%) on the two
datasets, respectively. Therefore, it is meaningful to explore how to use
prompt tuning based on CLIP effectively for better performance. From the
perspective of prompt tuning methods, when using ResNet-50 as the backbone,
the performance of our method on VOC-LT is more promising, which is 4.1
points, 6.81 points and 4.41 points better than CoOp [53], CoCoOp [52] and
DualCoOp [35], which are popular prompt learning methods for single-label and
multi-label recognition. And the performance on COCO-LT is similar to that on
VOC-LT, which is 4.03 points, 12.95 points, and 5.86 points better than CoOp,
CoCoOp and DualCoOp. When replacing the backbone with ViT-Base/16, the overall
mAP performance of our method can further boost up to 87.88% and 66.19% on
VOC-LT and COCO-LT, which is the current new state-of-the-art of the two
datasets.

Datasets | VOC-LT
---|---
Methods | total | head | medium | tail
BCE | 82.18 | 64.90 | 83.17 | 94.30
MLS | 84.30 | 64.31 | 84.82 | 97.47
Focal Loss [20] | 85.37 | 66.17 | 87.70 | 97.52
CB Loss [6] | 85.25 | 65.37 | 87.71 | 97.20
R-BCE-Focal [43] | 84.56 | 66.01 | 86.61 | 97.67
ASL [28] | 86.40 | 69.12 | 88.79 | 98.07
DB Focal [43] | 87.88 | 72.10 | 89.26 | 98.49

Datasets | COCO-LT
---|---
Methods | total | head | medium | tail
BCE | 58.04 | 41.79 | 58.86 | 73.90
MLS | 61.26 | 41.71 | 64.11 | 74.58
Focal Loss [20] | 54.40 | 37.60 | 59.36 | 62.33
CB Loss [6] | 56.45 | 34.61 | 58.77 | 74.52
R-BCE-Focal [43] | 60.13 | 38.11 | 64.87 | 72.79
ASL [28] | 64.89 | 43.18 | 68.22 | 78.43
DB Focal [43] | 66.19 | 44.89 | 69.80 | 79.08

Table 2: mAP performance of the proposed method with different multi-label
loss functions.

###  4.4 Ablation Analysis

Components Analysis. To further analyze which component makes our methods
performant for long-tailed multi-label recognition, we conduct a set of
ablation studies and report the results in Table 3. We first conduct
experiment with zero-shot CLIP and the achieved mAP performances are 85.77% on
VOC-LT, 60.17% on COCO-LT, which surprisingly outperforms the prior SOTA LTML.
It indicates that pre-trained VL models demonstrate a robust capability for
visual recognition, providing a solid foundation for our approach. However,
the mAP performance of the tail classes outperforms head classes by nearly 30
points on both VOC-LT and COCO-LT. Then CoOp benefited from soft prompts
(learnable prompts) and the mAP performance is improved to 86.02% on VOC-LT
and 60.68% on COCO-LT, with 0.25% and 0.51% increments. Besides, we design the
class-specific embedding (CSE) loss with class-aware soft margin (CASM) and
re-weighting (RW) to learn more fine-grained and class-related prompts that
build semantic relationships across different classes, especially for the tail
classes by encouraging those classes to have larger margins and weights. The
mAP performances of head, medium, and tail classes after adding the embedding
loss are all significantly improved and the overall mAP surpasses CoOp by
1.26% and 4.66% on VOC-LT and COCO-LT, which demonstrates our embedding loss
can help prompts learn fine-grained classes descriptions and semantic
relationships across the classes. Finally, the integration of CASM and RW
strategy further improves the mAP performance slightly, mainly for the tail
performance by 0.65% and 1.12% on VOC-LT and COCO-LT.

SP | EBD | CASM | RW | VOC-LT | avg.$\Delta$ | COCO-LT | avg.$\Delta$
---|---|---|---|---|---|---|---
total | head | medium | tail | total | head | medium | tail
|  |  |  | 85.77 | 66.52 | 88.93 | 97.83 |  | 60.17 | 38.52 | 65.06 | 72.28 |
✓ |  |  |  | 86.02 | 67.71 | 88.79 | 97.67 | +0.29 | 60.68 | 41.97 | 63.18 | 73.85 | +0.91
✓ | ✓ |  |  | 87.28 | 71.07 | 89.01 | 97.84 | +0.51 | 65.34 | 44.27 | 69.39 | 77.96 | +5.23
✓ | ✓ | ✓ |  | 87.62 | 72.01 | 89.26 | 98.13 | +1.99 | 65.81 | 44.90 | 69.71 | 78.76 | +5.79
✓ | ✓ | ✓ | ✓ | 87.88 | 72.10 | 89.26 | 98.49 | +2.17 | 66.19 | 44.89 | 69.80 | 79.08 | +5.98
Table 3: Ablation analysis on different components of the proposed method.
Here “SP” denotes Soft Prompt, “EBD” Embedding Loss, “CASM” Class-Aware Soft
Margin, “RW” Re-weighting, “avg.$\Delta$” average performance improvement.

Multi-Label Classification Loss Functions. Here, we compare a number of multi-
label classification loss functions, including Binary Cross-Entropy Loss
(BCE), Multi-Label Soft Margin Loss (MSL), Focal Loss [20], CB Loss [6],
R-BCE-Focal [43], Asymmetric Loss (ASL) [28] and DB Focal [43]. As illustrated
in Table 2, DB Focal loss that takes the co-occurrence of labels and the
dominance of negative labels into account works significantly better than
other multi-label classification loss for the long-tailed multi-label visual
recognition task.

Effectiveness of Text Supervision. We further compare our method with fine-
tuning CLIP’s image encoder when using ResNet-50 as the backbone to explore
whether the significant effect of our approach is due to text supervision or
simply because the CLIP’s image encoder is so powerful. In order to prevent
interference with the trained CLIP’s image encoder during the fine-tuning
phase, we only fine-tune a fully connected layer added at the end of the image
encoder. The results are shown in Fig. 4. Obviously, fine-tuning the image
encoder shows promising results, but still largely underperforms LMPT, which
suggests that the gradients that went through the text encoder provide more
useful information.

Figure 4: mAP performance of different methods w/o text supervision on two
datasets. (a) VOC-LT. (b) COCO-LT. Figure 5: Example decisions from our model,
CLIP and ResNet.

###  4.5 Qualitative Analysis

To better understand how our method deals with long-tailed multi-label data,
we performed qualitative experiments with ResNet222We used the open-source
pre-trained models trained on COCO-LT and VOC-LT by [43]., CLIP, and ours on
COCO-LT and VOC-LT. Fig. 5 shows several cases where the model justifies its
abilities for the prediction. For example, in the third column, ResNet only
recognizes $\left[\mathrm{person}\right]$ (belongs to head classes) and fails
to classify the image to $\left[\mathrm{train}\right]$ (belongs to tail
classes), which is a pervasive challenge encountered by visual models. The
emergence of CLIP is a great remedy for this issue, owing to its huge training
data and effective text supervision. Nevertheless, simple hand-crafted
templates as prompts still cannot accurately identify categories as they
cannot describe the characteristics of each category. Understanding the inter-
class relationships, particularly among head and tail categories, presents a
formidable challenge in multi-label visual recognition, which is essential for
achieving optimal performance in this domain. With the aid of our approach,
utilizing prompts that learn from a large corpus of image-caption data, it has
become feasible to discern the semantic relationships between categories and
accurately predict the relevant categories of simple objects, even in
challenging scenarios such as identifying $[\mathrm{stop}$ $\mathrm{sign}]$
from images. Therefore, our proposed method demonstrates significant
advantages in effectively addressing the intricate relationship among multiple
labels and the long-tailed problem with the aid of text supervision.

###  4.6 Limitations

Although the proposed method LMPT achieves good performance on two long-tailed
multi-label recognition benchmarks, it still has some imperfections. Firstly,
text descriptions in VOC-LT are generated by the open-source image-caption
model, which inevitably leads to some inaccurate texts. Secondly, although
captions and labels in COCO-LT are manually annotated, there are still many
errors (e.g., missing label words in the text, inconsistent labels), which
greatly affects the performance of the model. Moreover, VL models underperform
prior visual models overhead classes significantly, which demonstrates that
there is much room for improvement in the current VL models. However, we
believe that advancements in VL models and text data will lead to solutions
for these challenges in the future.

##  5 Conclusion

In this work, we propose a new view of prompt tuning for long-tailed multi-
label visual recognition by learning class-specific contexts from the
alignment of prompts and textual description (caption), which complements more
fine-grained features and builds semantic relationships across head and tail
classes. Considering the class imbalance, the class-specific embedding loss is
added with the class-aware soft margin and re-weighting strategy to encourage
the tail classes to be more generalized. Furthermore, we adopt distribution-
balanced loss as the classification loss function, which is more suitable for
our task compared with other loss functions. Our method demonstrates
substantial improvement over the previous SOTA and zero-shot CLIP on VOC-LT
and COCO-LT. We hope our approach will inspire future work in this field.

## References

  * [1] Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski.  A systematic study of the class imbalance problem in convolutional neural networks.  Neural networks, 106:249–259, 2018.
  * [2] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma.  Learning imbalanced datasets with label-distribution-aware margin loss.  Advances in neural information processing systems, 32, 2019.
  * [3] Tianshui Chen, Muxin Xu, Xiaolu Hui, Hefeng Wu, and Liang Lin.  Learning semantic-specific graph representation for multi-label image recognition.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 522–531, 2019.
  * [4] Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo.  Multi-label image recognition with graph convolutional networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5177–5186, 2019.
  * [5] Jiequan Cui, Shu Liu, Zhuotao Tian, Zhisheng Zhong, and Jiaya Jia.  Reslt: Residual learning for long-tailed recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * [6] Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie.  Class-balanced loss based on effective number of samples.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9268–9277, 2019.
  * [7] Qi Dong, Shaogang Gong, and Xiatian Zhu.  Class rectification hard mining for imbalanced deep learning.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1851–1860, 2017.
  * [8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [9] Chris Drummond, Robert C Holte, et al.  Class imbalance, and cost sensitivity: why under-sampling beats over-sampling.  In Workshop on learning from imbalanced datasets II, volume 11, pages 1–8, 2003.
  * [10] Mark Everingham, SM Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes challenge: A retrospective.  International journal of computer vision, 111(1):98–136, 2015.
  * [11] Hao Guo and Song Wang.  Long-tailed multi-label visual recognition by collaborative training on uniform and re-balanced samplings.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15089–15098, 2021.
  * [12] Zixian Guo, Bowen Dong, Zhilong Ji, Jinfeng Bai, Yiwen Guo, and Wangmeng Zuo.  Texts as images in prompt tuning for multi-label image recognition.  arXiv preprint arXiv:2211.12739, 2022.
  * [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [14] Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang.  Learning deep representation for imbalanced classification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5375–5384, 2016.
  * [15] Xinyu Huang, Youcai Zhang, Ying Cheng, Weiwei Tian, Ruiwei Zhao, Rui Feng, Yuejie Zhang, Yaqian Li, Yandong Guo, and Xiaobo Zhang.  Idea: Increasing text diversity via online multi-label recognition for vision-language pre-training.  In Proceedings of the 30th ACM International Conference on Multimedia, pages 4573–4583, 2022.
  * [16] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In International Conference on Machine Learning, pages 4904–4916. PMLR, 2021.
  * [17] Lie Ju, Xin Wang, Zhen Yu, Lin Wang, Xin Zhao, and Zongyuan Ge.  Long-tailed multi-label retinal diseases recognition using hierarchical information and hybrid knowledge distillation.  arXiv preprint arXiv:2111.08913, 2021.
  * [18] Bingyi Kang, Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng.  Exploring balanced feature spaces for representation learning.  In International Conference on Learning Representations, 2020.
  * [19] Salman H Khan, Munawar Hayat, Mohammed Bennamoun, Ferdous A Sohel, and Roberto Togneri.  Cost-sensitive learning of deep feature representations from imbalanced data.  IEEE transactions on neural networks and learning systems, 29(8):3573–3587, 2017.
  * [20] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017.
  * [21] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [22] Shilong Liu, Lei Zhang, Xiao Yang, Hang Su, and Jun Zhu.  Query2label: A simple transformer way to multi-label classification.  arXiv preprint arXiv:2107.10834, 2021.
  * [23] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu.  Large-scale long-tailed recognition in an open world.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2537–2546, 2019.
  * [24] Teli Ma, Shijie Geng, Mengmeng Wang, Jing Shao, Jiasen Lu, Hongsheng Li, Peng Gao, and Yu Qiao.  A simple long-tailed recognition baseline via vision-language model.  arXiv preprint arXiv:2111.14745, 2021.
  * [25] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar.  Long-tail learning via logit adjustment.  arXiv preprint arXiv:2007.07314, 2020.
  * [26] Wanli Ouyang, Xiaogang Wang, Cong Zhang, and Xiaokang Yang.  Factors in finetuning deep model for object detection with long-tail distribution.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 864–873, 2016.
  * [27] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [28] Tal Ridnik, Emanuel Ben-Baruch, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi Zelnik-Manor.  Asymmetric loss for multi-label classification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 82–91, 2021.
  * [29] Tal Ridnik, Gilad Sharir, Avi Ben-Cohen, Emanuel Ben-Baruch, and Asaf Noy.  Ml-decoder: Scalable and versatile classification head.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 32–41, 2023.
  * [30] Dvir Samuel and Gal Chechik.  Distributional robustness loss for long-tail learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9495–9504, 2021.
  * [31] Timo Schick and Hinrich Schütze.  Exploiting cloze-questions for few-shot text classification and natural language inference.  In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255–269, 2021\.
  * [32] Li Shen, Zhouchen Lin, and Qingming Huang.  Relay backpropagation for effective learning of deep convolutional neural networks.  In European conference on computer vision, pages 467–482. Springer, 2016.
  * [33] Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh.  Autoprompt: Eliciting knowledge from language models with automatically generated prompts.  arXiv preprint arXiv:2010.15980, 2020.
  * [34] Edgar Simo-Serra, Eduard Trulls, Luis Ferraz, Iasonas Kokkinos, Pascal Fua, and Francesc Moreno-Noguer.  Discriminative learning of deep convolutional feature point descriptors.  In Proceedings of the IEEE international conference on computer vision, pages 118–126, 2015.
  * [35] Ximeng Sun, Ping Hu, and Kate Saenko.  Dualcoop: Fast adaptation to multi-label recognition with limited annotations.  arXiv preprint arXiv:2206.09541, 2022.
  * [36] Jingru Tan, Changbao Wang, Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan.  Equalization loss for long-tailed object recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11662–11671, 2020.
  * [37] Changyao Tian, Wenhai Wang, Xizhou Zhu, Jifeng Dai, and Yu Qiao.  Vl-ltr: Learning class-wise visual-linguistic representation for long-tailed visual recognition.  In European Conference on Computer Vision, pages 73–91. Springer, 2022.
  * [38] Grigorios Tsoumakas and Ioannis Katakis.  Multi-label classification: An overview.  International Journal of Data Warehousing and Mining (IJDWM), 3(3):1–13, 2007.
  * [39] Jiang Wang, Yi Yang, Junhua Mao, Zhiheng Huang, Chang Huang, and Wei Xu.  Cnn-rnn: A unified framework for multi-label image classification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2285–2294, 2016.
  * [40] Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang.  Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework.  arXiv preprint arXiv:2202.03052, 2022.
  * [41] Zhouxia Wang, Tianshui Chen, Guanbin Li, Ruijia Xu, and Liang Lin.  Multi-label image recognition by recurrently discovering attentional regions.  In Proceedings of the IEEE international conference on computer vision, pages 464–472, 2017.
  * [42] Colin Wei, Jason Lee, Qiang Liu, and Tengyu Ma.  On the margin theory of feedforward neural networks.  2018\.
  * [43] Tong Wu, Qingqiu Huang, Ziwei Liu, Yu Wang, and Dahua Lin.  Distribution-balanced loss for multi-label classification in long-tailed datasets.  In European Conference on Computer Vision, pages 162–178. Springer, 2020.
  * [44] Shichao Xu, Yikang Li, Jenhao Hsiao, Chiuman Ho, and Zhu Qi.  A dual modality approach for (zero-shot) multi-label classification.  arXiv preprint arXiv:2208.09562, 2022.
  * [45] Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun.  Cpt: Colorful prompt tuning for pre-trained vision-language models.  arXiv preprint arXiv:2109.11797, 2021.
  * [46] Jin Ye, Junjun He, Xiaojiang Peng, Wenhao Wu, and Yu Qiao.  Attention-driven dynamic graph convolutional network for multi-label image recognition.  In European conference on computer vision, pages 649–665. Springer, 2020.
  * [47] Min-Ling Zhang and Zhi-Hua Zhou.  Ml-knn: A lazy learning approach to multi-label learning.  Pattern recognition, 40(7):2038–2048, 2007.
  * [48] Min-Ling Zhang and Zhi-Hua Zhou.  A review on multi-label learning algorithms.  IEEE transactions on knowledge and data engineering, 26(8):1819–1837, 2013.
  * [49] Yifan Zhang, Bryan Hooi, Lanqing Hong, and Jiashi Feng.  Test-agnostic long-tailed recognition by test-time aggregating diverse experts with self-supervision.  arXiv preprint arXiv:2107.09249, 2021.
  * [50] Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng.  Deep long-tailed learning: A survey.  arXiv preprint arXiv:2110.04596, 2021.
  * [51] Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen.  Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9719–9728, 2020.
  * [52] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816–16825, 2022.
  * [53] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  International Journal of Computer Vision, 130(9):2337–2348, 2022\.
  * [54] Linchao Zhu and Yi Yang.  Inflated episodic memory with region self-attention for long-tailed visual recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4344–4353, 2020.

Supplementary Document

##  6 Prompt Initialization

We compare random initialization with manual initialization. The latter uses
the embeddings of “a photo of a” to initialize the context vectors for the two
datasets. For a fair comparison, we also set the context length to 4 when
using random initialization. Table 4 suggests that in the case of a small-
scale training set (e.g., VOC-LT: 1.9k+ images) a good initialization has a
big impact. The model performance may be better if we use the initialization
words for a specific dataset for further optimization.

| Total | Head | Medium | Tail
---|---|---|---|---
$\left[\mathrm{V}\right]_{1}\left[\mathrm{V}\right]_{2}\left[\mathrm{V}\right]_{3}\left[\mathrm{V}\right]_{4}$ | 84.74 | 65.73 | 87.23 | 97.13
“a photo of a” | 87.88 | 72.10 | 89.26 | 98.49
Table 4: Random v.s. manual initialization.

##  7 Comparison with Prompt Ensembling

Radford et al. [27] proposed that ensembling over multiple zero-shot
classifiers generated using different hand-crafted prompts can improve
performance, such as “a photo of the large [CLASS].”, “a bad photo of the
[CLASS].”, which represent different scales and views for images. The 80
context prompts are selected for experiments, which have been demonstrated
their usefulness on ImageNet. The used templates are listed in
“prompt_template.py”. As shown in Table 5, there is a slight increase in model
performance over the single-template prompt method on certain datasets, but
justifies the superiority of LMPT.

Dataset | Method | mAP (%)
---|---|---
Total | Head | Medium | Tail
VOC-LT | Zero-Shot | 85.77 | 66.52 | 88.93 | 97.83
Ensembling | 85.17 | 65.57 | 88.36 | 97.45
LMPT | 87.88 | 72.10 | 89.26 | 98.49
COCO-LT | Zero-Shot | 60.17 | 38.52 | 65.06 | 72.28
Ensembling | 60.86 | 41.14 | 65.21 | 72.47
LMPT | 66.19 | 44.89 | 69.80 | 79.08
Table 5: Comparison with prompt engineering and prompt ensembling. The best
and second-best performance are indicated by bold and underline, respectively.

##  8 Prompt Length

Overall Prompt Length $M$. As shown in Fig. 6, we have provided the comparison
of the performance of LMPT with different prompt context lengths (i.e.,
$M=2,4,8,16,32,64$) in the two datasets. For long-tailed multi-label visual
recognition, we learn class-specific prompts and thus LMPT performs well when
$M$ is small, such as $2,4$.

Figure 6: Results (mAP %) with different prompt length $M$ for computing
classification loss. (a) VOC-LT. (b) COCO-LT.

Prompt Length $m$ for Computing CSE Loss. Moreover, we also compare with
different lengths $m$ of prompt context to compute the class-specific
embedding loss $\left(m\leq M\right)$. Table 6 shows that LMPT performs the
best when $m/M=25\%$ on the two datasets, which is quite reasonable. Only a
small fraction of tokens is required to learn the general prompts (e.g., “a
photo of a”), as they are not very complex and do not require many tokens to
represent. However, more tokens are needed to learn the class-specific
prompts, because more tokens are needed to describe the categories and the
semantic relationships between them. The prompts cannot fully capture the
class descriptions and semantic relationships if the number of tokens is
relatively small.

Dataset | $m/M$ | mAP (%)
---|---|---
Total | Head | Medium | Tail
VOC-LT | 12.5% | 82.35 | 63.08 | 82.83 | 96.43
25% | 83.56 | 64.06 | 84.22 | 97.59
50% | 82.43 | 62.52 | 83.86 | 96.30
75% | 82.01 | 61.46 | 83.85 | 96.06
100% | 83.32 | 65.34 | 83.60 | 96.59
COCO-LT | 12.5% | 51.89 | 26.61 | 56.01 | 68.71
25% | 56.16 | 31.29 | 60.46 | 72.38
50% | 52.15 | 26.56 | 56.09 | 69.47
75% | 54.14 | 29.34 | 57.28 | 71.81
100% | 53.55 | 27.71 | 58.67 | 69.53
Table 6: Performance with different settings of prompt length $m$. The
performance reported here is the average performance when $M=16$.

##  9 Algorithm of Class-Specific Embedding Loss

The overall process of class-specific embedding loss is outlined in Algorithm
1.

Input: Text embeddings of textual descriptions (captions) $t$, labels
$\widetilde{y}$, prompt $o$

Output: Class-Specific Embedding Loss $\mathcal{L}_{cse}$

1 for  _$k=1,2,...,N$_ do

2 $\ell_{cse}=0$;

3 for  _$i=1,2,...,C$_ do

4 Compute class-aware soft margin $\widetilde{\mu}_{i}$ using Eq. (4);

5 Compute weight $w_{i}$ using Eq. (5);

6 Compute $\Delta_{i}^{k}=1-\cos\left(t_{i}^{k},o_{i}|_{m}^{M}\right)$;

7 if  _$\widetilde{y}_{i}^{k}=1$_ then

8 $\ell_{cse}=w_{i}\Delta_{i}^{k}$;

9

10 else

11
$\ell_{cse}=\text{ReLU}\left(w_{i}\left(\widetilde{\mu}_{i}-\Delta_{i}^{k}\right)\right)$;

12

13 end if

14

15 end for

16

17 end for

Compute $\mathcal{L}_{cse}$ using Eq. (7).

Algorithm 1 Class-Specific Embedding Loss

##  10 Comparison with Fine-tuning CLIP

We try to fine-tune the entire CLIP model and find that in the first epoch of
fine-tuning, there is a slight improvement in model performance, with an
increase in the head classes but a decrease in the tail classes. However, as
shown in Table 7, in the following epochs, the performance gradually
deteriorates in the following epochs. Therefore, fine-tuning CLIP is far
inferior to LMPT, which can simultaneously improve the performance of both
head and tail classes.

Dataset | Method | mAP (%)
---|---|---
Total | Head | Medium | Tail
VOC-LT | Zero-Shot | 85.77 | 66.52 | 88.93 | 97.83
FT (Epoch 1) | 86.77 | 70.05 | 90.32 | 96.63
FT (Epoch 2) | 85.26 | 66.55 | 89.42 | 96.17
FT (Epoch 3) | 85.67 | 67.32 | 89.78 | 96.34
FT (Epoch 4) | 84.80 | 65.83 | 88.52 | 96.23
FT (Epoch 5) | 85.35 | 67.00 | 89.35 | 96.12
LMPT | 87.88 | 72.10 | 89.26 | 98.49
COCO-LT | Zero-Shot | 60.17 | 38.52 | 65.06 | 72.28
FT (Epoch 1) | 58.72 | 40.55 | 62.67 | 69.50
FT (Epoch 2) | 58.44 | 40.33 | 62.63 | 68.86
FT (Epoch 3) | 57.83 | 39.76 | 61.74 | 68.75
FT (Epoch 4) | 55.67 | 36.89 | 59.30 | 67.41
FT (Epoch 5) | 52.12 | 33.09 | 56.62 | 62.93
LMPT | 66.19 | 44.89 | 69.80 | 79.08
Table 7: Comparison with fine-tuning CLIP. “FT” denotes fine-tuning.

##  11 Analysis of Multi-Label Classification Loss Functions

Below we discuss the connections between some common multi-label
classification loss functions. The standard formula for the multi-label
classification loss function is as follows:

$\displaystyle{\ell}_{cls}$
$\displaystyle=\begin{cases}{\ell}_{cls+}=-w\left(1-p_{+}\right)^{\gamma}\log\left(p_{+}\right),\\\
{\ell}_{cls-}=-w\left(p_{-}\right)^{\gamma}\log\left(1-p_{-}\right),\end{cases}$
(12)

where $w$ denotes weight, $\gamma$: scale hyper-parameter, $p_{+},p_{-}$:
prediction probability for positive and negative instances. Detailed
comparison is reported in Table 8. Hence, we apply the distribution-balanced
loss [43] as the classification loss function, which integrates rebalanced
weighting and negative-tolerant regularization for long-tailed multi-label
recognition.

Methods | $\gamma$ | $w$ | $p_{+}$ | $p_{-}$
---|---|---|---|---
BCE | - | - | $\sigma\left(z\right)$ | $\sigma\left(z\right)$
FL [20] | $\gamma$ | - | $\sigma\left(z\right)$ | $\sigma\left(z\right)$
CB [6] | $\gamma$ | $\frac{1-\beta}{1-\beta^{n}}$ | $\sigma\left(z\right)$ | $\sigma\left(z\right)$
ASL [28] | $\gamma_{+},\gamma_{-}$ | - | $\sigma\left(z\right)$ | $\max\left(\sigma\left(z\right)-m,0\right)$
DB [43] | $\gamma$ | $\alpha+\sigma\left(\beta\frac{\frac{1}{n}}{\sum\frac{1}{n}}-\theta\right)$ | $\sigma\left(z-v\right)$ | $\sigma\left(\zeta\left(z-v\right)\right)$
Table 8: Comparison with the exsiting multi-label classification loss functions. $\alpha,\beta,m,\zeta,\kappa$: hypper-parameter, $v$: class-specific bias calculated by $\sigma\left(z\right)$ and $\kappa$, $\sigma$: sigmoid function. Dataset | Method | #P | mAP (%)
---|---|---|---
Total | Head | Medium | Tail
VOC-LT | Zero-Shot | - | 84.30 | 63.60 | 88.03 | 97.03
CoOp [52] | $P$ | 81.34 | 65.10 | 81.54 | 93.37
TaI-DPT [12] | $2P$ | 83.75 | 66.27 | 85.17 | 94.57
DualCoOp [35] | $2P$ | 81.03 | 66.45 | 80.53 | 92.33
LMPT | $P$ | 85.44 | 66.62 | 88.11 | 97.86
COCO-LT | Zero-Shot | - | 56.19 | 35.73 | 60.52 | 68.45
CoOp [52] | $P$ | 54.94 | 38.06 | 56.67 | 67.51
TaI-DPT [12] | $2P$ | 56.23 | 40.52 | 58.40 | 66.09
DualCoOp [35] | $2P$ | 53.11 | 40.48 | 55.20 | 62.11
LMPT | $P$ | 58.97 | 41.87 | 61.60 | 69.60
Table 9: Comparison with TaI-DPT and DualCoOp. “#P” denotes the number of
parameters that the model needs to update. The reported performances here all
use ResNet-50 as the backbone for a fair comparison.

##  12 Loss Weight $\lambda$

As shown in Fig. 7, we compare the model performance under different settings
of loss weight $\lambda$ in the two loss functions, and the results show that
when lambda is set to 0.5, the two loss functions can be better balanced,
resulting in the best model performance.

Figure 7: Results (mAP %) with different weight $\lambda$. (a) VOC-LT. (b)
COCO-LT.

##  13 Comparison with DualCoOp and TaI-DPT

In this section, we aim to analyze and compare two methods that adapt pre-
trained CLIP for multi-label recognition tasks. DualCoOp [35] proposes to use
a pair of positive and negative prompts to generate binary classification
probability for each class through contrastive learning. Table 9 suggests that
the negative prompt is not able to handle long-tailed multi-label visual
recognition effectively. While it can significantly improve the performance of
the head classes, it can cause a substantial drop in the performance of the
tail classes. In addition, TaI-DPT [12] suggests using double-grained prompts
to focus on global and local features separately, which has a certain effect
on multi-label visual recognition, but it cannot alleviate the long-tailed
problem, and the performance on the tail classes still decreases.

However, for our proposed prompt tuning with class-specific embedding (CSE)
loss, a single prompt is used to learn both class-related descriptions and
semantic relationships between categories, instead of using two prompts as
used in DualCoOp and TaI-DPT, which reduces the number of parameters the model
needs to train. The class-specific contexts with the benefit of textual
descriptions (captions) can help establish semantic relationships between
classes, especially between the head and tail classes. Furthermore, the CSE
loss, which incorporates class-aware soft margin and re-weighting, allows the
model to pay more attention to the tail classes without sacrificing the
performance on the head classes. As a result, we achieve state-of-the-art
performance on this task.

##  14 More Examples

In this section, we provide four sets of images containing two labels with
semantic relationships to further compare the performance of CLIP and LMPT
(ours). And the selected labels include not only the head classes but also the
tail classes, which comprehensively demonstrates the superiority of LMPT. As
shown in Fig. 8, whether it is two head classes with semantic relationship, or
one head class and one tail class, or one medium class and one tail class,
LMPT can cover more visual attributes than CLIP due to the advantages of CSE
loss.

Figure 8: More examples of our model and CLIP. The correctness of the
judgement of each image depends on whether the two labels with semantic
relationships are within the top 10 predicted labels of the model. The top
left is CLIP’s judgment, and the top right is ours. The color of the labels
corresponds to whether they belong to the head, medium, and tail classes.