# Dissecting the Discrepancies: Analyzing Pre-trained Deep Neural Network
Variations Across Model Hubs

Wenxin Jiang  0000-0003-2608-8576 Purdue UniversityWest Lafayette, IN, USA
<EMAIL_ADDRESS>,  Chingwo Cheung  0009-0001-0599-7909 Purdue
UniversityWest Lafayette, IN, USA<EMAIL_ADDRESS>,  George K.
Thiruvathukal  0000-0002-0452-5571 Loyola University ChicagoChicago, IL, USA
<EMAIL_ADDRESS>and  James C. Davis  0000-0003-2495-686X Purdue UniversityWest
Lafayette, IN, USA<EMAIL_ADDRESS>
# Analyzing Pre-trained Deep Neural Network Variations Across Model Hubs

Wenxin Jiang  0000-0003-2608-8576 Purdue UniversityWest Lafayette, IN, USA
<EMAIL_ADDRESS>,  Chingwo Cheung  0009-0001-0599-7909 Purdue
UniversityWest Lafayette, IN, USA<EMAIL_ADDRESS>,  George K.
Thiruvathukal  0000-0002-0452-5571 Loyola University ChicagoChicago, IL, USA
<EMAIL_ADDRESS>and  James C. Davis  0000-0003-2495-686X Purdue UniversityWest
Lafayette, IN, USA<EMAIL_ADDRESS>
# Assessing the Effects of Pre-trained Deep Neural Network Architecture
Variations Across Deep Learning Model Hubs

Wenxin Jiang  0000-0003-2608-8576 Purdue UniversityWest Lafayette, IN, USA
<EMAIL_ADDRESS>,  Chingwo Cheung  0009-0001-0599-7909 Purdue
UniversityWest Lafayette, IN, USA<EMAIL_ADDRESS>,  George K.
Thiruvathukal  0000-0002-0452-5571 Loyola University ChicagoChicago, IL, USA
<EMAIL_ADDRESS>and  James C. Davis  0000-0003-2495-686X Purdue UniversityWest
Lafayette, IN, USA<EMAIL_ADDRESS>
# Assessing Naming Conventions of Pre-trained Deep Learning Models Across Deep
Learning Model Hubs

Wenxin Jiang  0000-0003-2608-8576 Purdue UniversityWest Lafayette, IN, USA
<EMAIL_ADDRESS>,  Chingwo Cheung  0009-0001-0599-7909 Purdue
UniversityWest Lafayette, IN, USA<EMAIL_ADDRESS>,  George K.
Thiruvathukal  0000-0002-0452-5571 Loyola University ChicagoChicago, IL, USA
<EMAIL_ADDRESS>and  James C. Davis  0000-0003-2495-686X Purdue UniversityWest
Lafayette, IN, USA<EMAIL_ADDRESS>
# Assessing Naming Conventions of Pre-trained Deep Learning Models

Wenxin Jiang  0000-0003-2608-8576 Purdue UniversityWest Lafayette, IN, USA
<EMAIL_ADDRESS>,  Chingwo Cheung  0009-0001-0599-7909 Purdue
UniversityWest Lafayette, IN, USA<EMAIL_ADDRESS>,  George K.
Thiruvathukal  0000-0002-0452-5571 Loyola University ChicagoChicago, IL, USA
<EMAIL_ADDRESS>and  James C. Davis  0000-0003-2495-686X Purdue UniversityWest
Lafayette, IN, USA<EMAIL_ADDRESS>
# Exploring Naming Conventions of Pre-trained Deep Learning Model: Analyzing
the Impact on Reuse

Wenxin Jiang  0000-0003-2608-8576 Purdue UniversityWest Lafayette, IN, USA
<EMAIL_ADDRESS>,  Chingwo Cheung  0009-0001-0599-7909 Purdue
UniversityWest Lafayette, IN, USA<EMAIL_ADDRESS>,  George K.
Thiruvathukal  0000-0002-0452-5571 Loyola University ChicagoChicago, IL, USA
<EMAIL_ADDRESS>and  James C. Davis  0000-0003-2495-686X Purdue UniversityWest
Lafayette, IN, USA<EMAIL_ADDRESS>
# Exploring Naming Conventions and Naming Defects in Pre-trained Deep Learning
Models

Wenxin Jiang  0000-0003-2608-8576 Purdue UniversityWest Lafayette, IN, USA
<EMAIL_ADDRESS>,  Chingwo Cheung  0009-0001-0599-7909 Purdue
UniversityWest Lafayette, IN, USA<EMAIL_ADDRESS>,  George K.
Thiruvathukal  0000-0002-0452-5571 Loyola University ChicagoChicago, IL, USA
<EMAIL_ADDRESS>and  James C. Davis  0000-0003-2495-686X Purdue UniversityWest
Lafayette, IN, USA<EMAIL_ADDRESS>
# Naming Conventions and Defects in Pre-trained Models

Wenxin Jiang  0000-0003-2608-8576 Purdue UniversityWest Lafayette, IN, USA
<EMAIL_ADDRESS>,  Chingwo Cheung  0009-0001-0599-7909 Purdue
UniversityWest Lafayette, IN, USA<EMAIL_ADDRESS>,  George K.
Thiruvathukal  0000-0002-0452-5571 Loyola University ChicagoChicago, IL, USA
<EMAIL_ADDRESS>and  James C. Davis  0000-0003-2495-686X Purdue UniversityWest
Lafayette, IN, USA<EMAIL_ADDRESS>
# Exploring Naming Conventions (and Defects) of Pre-trained Deep Learning
Models in Hugging Face and Other Model Hubs

Wenxin Jiang  0000-0003-2608-8576 Purdue UniversityWest Lafayette, IN, USA
<EMAIL_ADDRESS>,  Chingwo Cheung  0009-0001-0599-7909 Purdue
UniversityWest Lafayette, IN, USA<EMAIL_ADDRESS>,  George K.
Thiruvathukal  0000-0002-0452-5571 Loyola University ChicagoChicago, IL, USA
<EMAIL_ADDRESS>and  James C. Davis  0000-0003-2495-686X Purdue UniversityWest
Lafayette, IN, USA<EMAIL_ADDRESS>
###### Abstract.

As innovation in deep learning continues, many engineers want to adopt Pre-
Trained deep learning Models (PTMs) as components in computer systems. PTMs
are part of a research-to-practice pipeline: researchers publish PTMs, which
engineers adapt for quality or performance and then deploy. If PTM authors
choose appropriate names for their PTMs, it could facilitate model discovery
and reuse. However, prior research has reported that model names are not
always well chosen, and are sometimes erroneous. The naming conventions and
naming defects for PTM packages have not been systematically studied —
understanding them will add to our knowledge of how the research-to-practice
process works for PTM packages.

In this paper, we report the first study of PTM naming conventions and the
associated PTM naming defects. We define the components of a PTM package name,
comprising the package name and claimed architecture from the metadata. We
present the first study focused on characterizing the nature of naming in PTM
ecosystem. To this end, we developed a novel automated naming assessment
technique that can automatically extract the semantic and syntactic patterns.
To identify potential naming defects, we developed a novel algorithm,
automated DNN ARchitecture Assessment pipeline (DARA), to cluster PTMs based
on architectural differences. Our main findings are: (1) Architecture is the
basic semantic component of a PTM package name; (2) The most common naming
defects are incorrect identifier, uninformative identifier, and incorrect
architecture label; (3) PTM package names from gated model hubs are more
standardized than open model hubs. We identified 1144 possible naming defects.
Our study suggests the naming conventions for PTMs, and frames the naming
conventions as signal of the research-to-practice relationships in the PTM
ecosystem. We envision future works on further empirical study on leveraging
meta-features of PTMs to support model search and reuse.

Software reuse, Machine learning, Deep neural networks, Model zoos, Empirical
software engineering

##  1\. Introduction

The naming conventions of Deep Neural Networks (DNNs) serve as the entry point
for engineers and researchers exploring or reusing these models. DNNs are
being incorporated into many computing systems, including autonomous vehicles
(Garcia et al., 2020), marketing (Tkáč and Verner, 2016), and healthcare
(Jiang et al., 2017). However, DNNs are costly to develop and optimize,
particularly when built from scratch (Patterson et al., 2021). Engineers
therefore tend to reuse Pre-Trained Models (PTMs) and adapt them to downstream
tasks (Jiang et al., 2023a, b). In the context of deep learning, “model zoos”
or “model hubs” have emerged as specialized “model registries” that store PTM
packages. These repositories serve as one-stop-shops for engineers seeking
optimal solutions for various downstream tasks, democratizing access to state-
of-the-art architectures and fostering a culture of model reuse (Jiang et al.,
2023a, 2022). While these repositories have significantly accelerated the
deployment of deep learning solutions across different domains, they also
present certain challenges. Due to the non-standardized nature of the open-
source PTM package ecosystem, engineers often encounter issues such as missing
attributes and incomparable packages (Jiang et al., 2023b). Despite these
challenges, the role of naming conventions in PTM packages remains under-
explored. Specifically, it is unclear how naming conventions contribute to
these reuse challenges and what naming defects might exacerbate them.

Previous work has emphasized the importance of naming conventions within the
software ecosystem. Naming conventions serve critical roles in code
readability, maintainability, and collaboration (Lawrie et al., 2007b; Butler
et al., 2010), and can also facilitate software reuse activities (Wang et al.,
2023). However, much of this research has been focused on non-reusable
components such as functions and variables. The naming conventions for
reusable components — like those found in the Java or C++ standard libraries
or packages available on GitHub or npm — are comparatively under-explored,
including in the context of PTM packages. Additionally, prior studies have
often focused either on the syntax or the semantics of method names,
neglecting the intricate relationship between syntax format and semantic
meaning. This oversight is particularly crucial in the context of software
reuse, where the fork or inheritance nature of packages necessitates a deeper
understanding of naming components.

Our goal is to understand the naming patterns and identify potential naming
defects of PTM packages. To achieve this, we collected 8744 PTM packages from
Hugging Face and 368 from 4 other model hubs. We designed and implemented an
automated naming assessment pipeline that is integrated with our tool for Deep
Neural Network Architecture Assessment (DARA). We designed a specialized
prompt for extracting naming components from PTM package names and
automatically identified potential naming defects. The validity of our
findings is further confirmed and checked by multiple researchers.

Our result shows the most common naming patterns and defect types. We also
find that PTM naming patterns have a strong implication for model inheritance.
Our assessment pipeline identified 1144 real naming defects. Drawing upon
these findings, we propose effective naming conventions aimed at mitigating
the current non-standardization in the open-source PTM ecosystem, with the
ultimate goal of aiding comprehension and reuse of PTM packages. We also
compare our fidings on PTM naming conventions with traditional software method
naming practices to elucidate key differences. Our findings envision
opportunities for further empirical studies on naming conventions, the
development of automated tool for PTM categorization, and the creation of a
naming recommendation system." Our contributions are:

  * •

We define the nature of naming in the PTM ecosystem using the framing of the
Research-To-Practice pipeline (section 3).

  * •

We developed a novel GPT4-based technique to automatically extract the
semantic and syntactic patterns from PTM package names (section 6.1.1). Using
this technique, we identified PTM naming patterns in open-source model
registries (section 6).

  * •

We developed a novel algorithm, DNN Architecture Assessment (DARA), to cluster
PTMs based on architectural differences. With DARA, we analyzed 8744 open and
368 gated PTM packages and identified four types of naming defects (8).

Significance of this work to software engineering: Pre-trained models are an
emerging component in computing systems. The names of PTMs are a major means
for discovering PTMs to re-use, and previous work shows that these names are
flawed in ways that impact re-use (engineering efficiency). Our study frames
this problem in terms of the research-to-practice pipeline, describes naming
conventions, and provides a methodology and automated tooling to discover
different classes of naming defects. Our study works towards making PTMs
easier to re-use, helping software engineers improve the performance of their
systems.

##  2\. Background and Related Work

###  2.1. Software Naming Conventions

As Knuth writes, “The main idea is to treat a program as a piece of
literature, addressed to human beings rather than to a computer” (Knuth,
1984). Most software is natural because it is created by humans (Hindle et
al., 2016). Naming conventions in software engineering have been a subject of
research for a while, serving a critical role in code readability,
maintainability, and collaboration (Lawrie et al., 2007b; Butler et al.,
2010). In the domain of computer programming, naming conventions comprise a
set of rules for selecting character sequences that denote variables, types,
functions, and other code entities (Wikipedia, 2023). Previous studies reveal
that most professional developers generally adhere to these established naming
guidelines (Alsuhaibani et al., 2021). Nevertheless, naming conventions in
open-source PTM registries remain inconsistent due to the absence of
standardized practices. However, naming conventions in open-source model
registries remain inconsistent due to the absence of standardized practices.
According to a prior interview study (Jiang et al., 2023b), engineers often
rely on the names of PTM packages for information, as model cards (README) and
metadata provided by these registries are often insufficient. Such missing
details and inconsistencies can pose challenges in PTM reuse, including
problems with understanding the model’s capabilities and limitations, as well
as issues related to code maintenance and collaboration (Butler et al., 2010).

The significance of naming conventions extends beyond just readability and
maintenance (Knuth, 1984). Prior work shows that inconsistencies in software
engineering data, often arising from informal naming conventions, can
introduce noise into datasets (Nguyen et al., 2023). Moreover, these
variations in naming can pose security risks. For example, “typosquatting” is
a tactic where an adversary registers a package name that is a slight
misspelling of a well-known package, thereby exploiting these naming variances
(Ohm et al., 2020; Zimmermann et al., 2019). Prior work has also shown the
importance of package names in software registries like npm and PyPI (Guo et
al., 2023; Gu et al., 2023). Such practices create confusion between malicious
and legitimate software packages and have been exploited in software supply
chain attacks (Ladisa et al., 2023; Okafor et al., 2022). Jiang et al. show
that PTM package registries, such as Hugging Face, are comparable to
traditional software packages registries. Therefore, these risks can also
potentially impact PTM reuse. Our study aims to address these naming
discrepancies by identifying potential naming issues, and proposing a robust
naming standard tailored for PTMs. Standardizing naming conventions can not
only enhance readability and facilitate collaboration but also bolster the
trustworthiness of PTM naming.

In the rapidly evolving landscape of PTMs, naming conventions are vital for
ensuring both consistent communication among stakeholders and compatibility
across different models (Jiang et al., 2023b). We define PTM naming
conventions as the structured methodology engineers employ, integrating
architecture details with a unique fork label, to unambiguously identify a
given model. Existing literature notes that the prevalent approach to PTM
reuse is fork-based, incorporating both architectural modifications and
additional training (Jiang et al., 2023a; Bhatia et al., 2023). Prior studies
have often focused either on the syntax or the semantics of method names,
neglecting the intricate relationship between syntax format and semantic
meaning (Lawrie et al., 2007b; Butler et al., 2010). This oversight is crucial
in the context of software reuse, particularly PTM reuse, where the fork or
inheritance nature of packages necessitates a deeper understanding of naming
components. However, the lack of standardized naming guidelines can result in
manually crafted names that introduce errors and confusion. This paper aims to
address this gap by presenting an in-depth study on the patterns and defects
in naming conventions within open-source PTM registries.

###  2.2. Deep Learning Research-to-practice (RTP) Pipeline

Figure 1.  Research-to-practice (RTP) process in the PTM ecosystem. The PTM
ecosystem is defined as interaction of a set of actors on top of the model hub
that results in a number of PTM solutions or services (Jiang et al., 2022).

Knowledge and technology transfer are foundational processes in scientific
research, often referred to as the Research-to-Practice (RTP) process (Nidhra
et al., 2013; Bozeman, 2000; Grima-Farrell, 2017). The RTP process also plays
a critical role in deep learning software ecosystem (Chen et al., 2015).
Developing DNNs from scratch usually requires large-scape computational
resources (Strubell et al., 2019; Patterson et al., 2021), such as
supercomputing clusters. To save the computational resources and time,
engineers usually apply knowledge transfer approaches to deep learning
research prototypes (Gopalakrishna et al., 2022; Jiang et al., 2023b). In
recent times, there has been a surge in the involvement of practitioners in
the RTP process pertaining to DNN techniques and prototypes. The swift
evolution of large-language models (LLMs) exemplifies this trend (Bubeck et
al., 2023; Brown et al., 2020).

Figure 1 illustrates how PTM naming is involved in the research-to-practice
pipeline in the Hugging Face PTM ecosystem (Davis et al., 2023). Hugging Face
is an open-gated model registry which means anyone can contribute models and
package names to it (Jiang et al., 2022). This is in contrast to gated and
closed registries, where PTM packages are reviewed by registry staff or can
only be submitted by authorized users such as employees. We highlight three
kinds of users in the RTP pipeline in PTM ecosystem: researcher, reengineer,
and adopter. These users are key components of the PTM supply chain (Jiang et
al., 2022). The researchers are not only the new model contributors, but also
the “name producer”. To support rapid model development and training,
engineers (annotated as “reengineer” in Figure 1) can adapt existing deep
learning models on downstream tasks or datasets through different
reengineering approaches, including transfer learning, fine-tuning, and
knowledge distillation (Davis et al., 2023; Jiang et al., 2023a; Han et al.,
2021). For example, an engineer can adapt the model head, i.e., the fully
connected output layer, to the target problem through model reengineering (Qi
et al., 2023). As the capability of PTMs keep growing in LLMs, researchers
have identified more approaches to reuse these models, including zero/few-shot
learning, prompt tuning (Brown et al., 2020; Radford et al., 2019; Lester et
al., 2021). PTM package registries offer a convenient method for PTM reuse
(Jiang et al., 2022). Real practitioners (e.g., reengineers and adopters) can
access PTM packages from Hugging Face by specifying the package names and
invoking the API within their system. These practitioners need to search and
compare different PTMs so they are also the “name consumer”.

###  2.3. Challenges in the RTP pipeline

Though the RTP pipeline can help reduce the engineering cost, it also comes
with many challenges Prior work conducted a large-scale vulnerability
assessment of GitHub repositories that are linked to models from Hugging Face,
and provided evidence for the existence of vulnerabilities in fundamental AI
repositories (Kathikar et al., 2023). Furthermore, the open-source nature of
PTMs can also lead to privacy and security concerns (Liu et al., 2022; Shen et
al., 2021). Ensuring the trustworthiness of open-source PTMs becomes more
complicated due to these discrepancies, making it challenging to evaluate
their performance and reliability (Jiang et al., 2023b, c).

Prior work also shows that inconsistencies in PTM packages could make the
understanding and utilization more complicated and costly. This complexity
could be further aggravated by organizational constraints, such as an
insufficient infrastructure to accommodate different PTMs or a lack of
necessary skills and training among staff to interpret the discrepancies in
the PTMs (Jiang et al., 2023a; Gopalakrishna et al., 2022). Prior work also
identified discrepancies of PTM architectures across multiple model hubs,
which poses a new threat to model zoo reliability (Montes et al., 2022). Open-
source model registries offer PTMs implemented in various frameworks. Due to
their "open-gated" contribution workflow, these model hubs are more
susceptible to discrepancies and inconsistencies (Jiang et al., 2022) For PTMs
implemented in different deep learning frameworks, some behaviors can be
different. For example, variations in SGD optimizers between Pytorch and
TensorFlow can lead to behavior differences (Banna et al., 2021). Prior defect
study shows that these inconsistencies make reengineering deep learning models
a challenging task (Jiang et al., 2023a).

Sculley et al. introduce the concept of “abstraction debt” as a common problem
in ML systems (Sculley et al., 2015). Abstraction debt is the lack of strong
abstractions to effectively manage ML systems, instead building ML systems up
in a haphazard organic way. _In this work, we consider the name of PTM package
as a kind of abstraction debt_. A name is an abstraction of the PTM package,
condensing information about the model architecture, dataset, etc. Prior
research has highlighted the importance of naming conventions in the PTM
ecosystem (Montes et al., 2022; Jiang et al., 2023b). Different from GitHub
packages, the naming of PTM packages is a significant part of the RTP
pipeline. Researchers and reengineers often have divergent opinions about the
naming conventions of PTM packages, making it challenging to fully automate a
large-scale measurement to help the community characterize the nature of
naming in the PTM ecosystem.

To summarize, existing work has shown concerns about potential risks in deep
learning model architectures, emphasizing the need for accuracy, transparency,
and compatibility. These risks, possibly due to mislabeling or variances
across model sources, can influence users’ trust and the models’ performance
and reliability. In response, our study takes a closer look at these
challenges by focusing on PTM naming conventions. We quantitatively analyze
the frequency and impact of various naming patterns and defects, thereby
providing insights that can enhance engineers’ understanding of the PTM
ecosystem. Our work aims to mitigate trust issues and ultimately facilitate a
more efficient RTP process for PTM reuse.

##  3\. The Context of PTM Package Naming

In this section, we provide a novel interpretation of PTM package naming
conventions.

Figure 2.  Hierarchy of model categories in Hugging Face. The blue categories
are Hugging Face model categorization system, while the red categories are
model family categorization. The center role (bold face) comprises the package
name (PTM and model type architecture) — this is the focus of our study.

###  3.1. Definition 1: PTM Package Naming

Hugging Face employs a multi-tiered categorization system, spanning diverse
domains, tasks, PTMs, and configurations (Face, 2023): A domain is the general
field or area of application where a set of models operate, such as Natural
Language Processing or Computer Vision. Within a domain, tasks are specific
problems that models aim to solve, like Sentiment Analysis or Named Entity
Recognition in the Natural Language Understanding domain. A PTM is an
individual instance of a trained model for a specific task, complete with
weights, parameters, and configuration metadata. A PTM package’s config.json
file details the model’s architecture and specifies the model type and the
task it is designed for, e.g., Albert for the model type and MaskedLM for the
task.

We mapped the categories from the Hugging Face naming system to PTM package
naming:

  * •

Domain and task are high-level metadata that are part of the package type.

  * •

We define a package name as the combination of a package identifier (e.g.,
albert-base-v2) and the model type or architecture indicated in the metadata
(e.g., AlbertForMaskedLM).

  * •

Package content: low-level files included in a package, such as weights and
training scripts.

In the PTM ecosystem, both the package type and its content are objective
factors. On the other hand, the package name is a subjective element, often
produced manually by the PTM contributors themselves. This subjectivity makes
the naming process a critical point where inadequacies or ambiguities can
occur. It’s essential to recognize that both objective and subjective
components of a package are susceptible to varying degrees of judgement, which
can impact how reengineers and adopters search for models (Figure 1).
According to prior research, the engineers involved in PTM ecosystem commonly
rely on either package identifiers or architectural labels when searching for
PTM packages appropriate for their specific tasks.

###  3.2. Definition 2: Naming Patterns and Defects

####  3.2.1. Naming Patterns

Figure 3.  Example of a PTM naming defect from Hugging Face. Top part: PTM
identifier. Bottom part: PTM architecture and model types. The identifier
incorrectly says “deberta” — this isa bert model.

Semantic Patterns refers to the constituent elements that form a model’s name.
These semantic patterns are different categories or attributes that are
combined to form the full model name. For example, "A-S-C" could stand for
"Architecture-Size-Dataset Characteristic," indicating that the model name is
composed of information about its architecture (deberta), size (base), and
dataset characteristic (uncased).

Syntactic Patterns are the syntactical styles and conventions used in naming
the models. Following prior works, our work includes four different naming
patterns: kebab-case, camelCase, snake_case, mix_ed-case (Zhang et al., 2021;
Binkley et al., 2013). These patterns reflect how words and phrases are
concatenated and capitalized in the names of the models. For instance, “kebab-
case” uses hyphens to separate words, "snake_case" uses underscores, and
"mix_ed-case" includes multiple patterns.

####  3.2.2. Naming Defects

Naming Defects are semantic inconsistencies or inaccuracies in the model names
that may lead to confusion or implementation errors. Figure 3 shows an example
PTM naming defect: there is a mismatch between the model identifier (deberta)
and the model architecture metadata (bert). It’s important to note that the
appropriateness of a PTM name can be subjective; currently, no standard or
guidelines exist to assess this in the PTM ecosystem.

Table 1.  Proposed taxonomy of naming defects, deduced from Cartesian product of PTM name elements: the identifier and/or the architecture can be incorrect. See Table 4 for the details induced from our dataset.  Naming defect type |  Definition
---|---
Incorrect identifier components |  Components in identifier do not match actual components.
Uninformative package identifier |  The identifier fails to provide useful information to discern the actual architecture of the model.
Incorrect architecture label |  The identifier correctly describes the actual architecture of the model, but the architecture labeled in metadata is incorrect.

##  4\. Research Questions

To summarize the literature: previous studies have focused on the naming
conventions associated with traditional software packages (Zimmermann et al.,
2019; Ladisa et al., 2022). However, PTM package presents its own set of
naming challenges that have yet to be discovered. This study seeks to
empirically assess the current naming conventions employed by engineers in PTM
ecosystem. To address this gap and identify the potential risks of naming in
the PTM ecosystem, we formulate the following research questions:

  * RQ1

What are the common naming patterns and corresponding frequencies in Hugging
Face?

  * RQ2

What are the common naming defects and corresponding frequencies in Hugging
Face?

  * RQ3

Do findings on naming patterns and defects in Hugging Face generalize to other
PTM hubs?

RQ1-2 focus on the Hugging Face ecosystem, investigating the nuances and
potential defects of naming conventions in PTM packages. RQ3 examines the
generalizability of the findings from Hugging Face to other model hubs.
Together, these questions provide a comprehensive view of the state, quality,
and scale of naming conventions within the Hugging Face repository and beyond.

##  5\. Methodology

###  5.1. Overview

Figure 4. Overview of our methodology and its relationship to research
questions. The list of models were passed into our naming assessment pipeline
which includes the usage of our architecture assessment tool (DARA, more
details are described in section 7) and GPT-4 to extract the naming patterns
and defects.

Figure 4 presents the relationship between our methods and the RQs. More
details about our methods are described in the corresponding sections. We
present how we extract and analyze the semantic and syntactic naming patterns
from PTM names in section 6.1 (RQ1). To better analyze naming defects (RQ3),
we developed a tool to automatically measure the naming patterns section 6.1
and defects section 8.1 of open-source PTMs. Our tool (section 7)
quantitatively identifies the number of models with naming defects. Finally,
we use the gated PTM packages we collected from section 5.2. Finally, we
compare our findings from PTM packages collected from Hugging Face to those
collected from other model hubs, and discussed about the generalizability of
our findings.

In section 5.2, we shows how we collected both open and gated PTM packages for
further analysis in .

###  5.2. PTM Collection

Figure 5.  Overview of the PTM collection process. We collect a list of models
from Hugging Face and other model hubs. Models that do not meet our predefined
criteria are subsequently filtered out.

Figure 5 shows the overview process of our PTM collection. To characterize the
nature of PTM naming, we first need to collect a list of PTM packages,
including their package content and names. Jiang et al. introduced multiple
types of model hubs, including open, gated, and commercial hubs (Jiang et al.,
2022). In this study, we collect PTM packages from two sources: (1) open PTM
packages from Hugging Face; (2) gated PTM packages provided by major
frameworks.

In our study, we refer to the data from Hugging Face as “open” PTM packages,
while PTMs connected from other model hubs are termed “gated” packages.

####  5.2.1. Open PTM Collection

Due to the rate limitation of Hugging Face APIs (Face, 2023), we used the
PTMTorrent dataset which is a snapshot of the 12,401 PTM packages available
from Hugging Face as of January 2023 (Jiang et al., 2023c). We apply four
filters to these PTM packages: (1) The model should be accessible through
Hugging Face API; (2) The model should have at least 10 monthly downloads; (3)
The model architecture (i.e., model$\\_$type in metadata) should exist; (4)
There should be at least two models per architecture. Filters (1) and (2)
ensure that the collected models are actually being used. Filters (3) and (4)
ensure that the models can be compared even if some information is missing.
After applying the filters to all Hugging Face models, we got a set of 8744
from 185 architectures, where for each model type, there are 2–1212 PTMs.

####  5.2.2. Gated PTM Collection

To collect gated PTM packages, we have to look at multiple model hubs.
Following prior work (Montes et al., 2022), we chose to look at gated model
hubs provided by each deep learning framework, including ONNX Model Zoo (ONNX,
2023), KerasAPP/NLP (Ker, [n. d.]), torchvision (Meta, 2022), and TensorFlow
Model Garden (Google, 2022). Though KerasCV was also an option, we chose to
omit it from our study as their models are already encompassed within
KerasApp, thereby avoiding redundancy. It’s worth noting that TensorFlow Model
Garden was not included in our dataset. This omission is primarily because
TensorFlow Model Garden lacks a centralized API for automated model retrieval,
and its models are dispersed across multiple directories, thereby complicating
automated collection efforts. Due to the variety of different model hubs, and
limited numbers, we manually apply three filters to these models: (1) The
model type or architecture should be explicit provided; (2) The pre-trained
weights should be available; (3) There should be at least two models per model
type; After applying the filters to all models from 4 model hubs, we got a set
of 368 from 56 architectures, where for each model type, there are 2–49 DNNs.

##  6\. RQ1: What are the naming patterns and frequencies in PTM Registries?

###  6.1. Method

To gain a comprehensive understanding of the naming characteristics of PTM
packages, and to ensure comparability with traditional software packages and
method naming conventions, we employed a dual-pronged analytical approach. To
assess the naming patterns of PTM packages, we initially designed a prompt
capable of extracting naming components, leveraging the capabilities of GPT-4
(section 6.1.1). Subsequently, we engineered an automated tool to identify
potential naming inconsistencies or defects. These were manually verified and
annotated (section 8.1).

####  6.1.1. Semantic Pattern Analysis

We first focused on the semantic sequences of PTM identifiers. For example, by
the sequence A-S-V, which stands for Architecture-ModelSize-Version. To
address our first research question (RQ1), we computed the frequency of each
naming pattern.

Figure 6.  Example of input/output of GPT-4. Input is model names. Output is
semantic patterns and confidence level.

Preliminary Analysis We conducted a preliminary analysis on 20 models to
understand the naming conventions of PTMs packages. Based on those results, we
identified 8 naming component, including architecture, model size, dataset,
model versioning, language, task, and other. Corresponding definitions and
examples are in Table 2.

These identified components are very common in PTM package names. However,
there is no standard or guidelines for these models. Due to the open-
contribution nature of Hugging Face model registry, we found that these model
names are very non-standardized.

Table 2.  Definition and examples of PTM naming components. Components above the horizontal line were identified before our pilot study. Others were identified based on the results of our pilot study.  Semantic naming component |  Definition |  Example
---|---|---
Architecture [A] |  The foundational structure or design of the deep learning model. |  bert, albert, resnet
Model size [S] |  The complexity of the model, often in terms of the number of parameters or architectural units. |  50, 101, base, large, xxlarge
Dataset [D] |  The specific dataset used for training or evaluation of the model. |  squad, imagenet
Model versioning [V] |  The version of the model. |  v1, v2
Language [L] |  Specifies the natural language for which the model is trained on. |  english, chinese, arabic
Task [T] |  The particular problem or application the model is designed to address. |  qa (question Answering), cls (classification)
Reuse method [F] |  The approach employed to adapt a PTM for downstream tasks. |  fine-tune, distill, few-shot
Other [O] |  A portion of the model name cannot be classified into the above categories. |  demo, test
Training process [R] |  The algorithm or approach used for training the model. |  pretrain, sparse
Number of layers [L] |  The depth of the neural network in terms of layers. |  L-12, L-24
Number of heads [L] |  The number of attention heads in models like transformers. |  H-12, H-256
Number of parameters [P] |  The total number of trainable parameters in the model. |  100M, 8B
Dataset characteristics [C] |  Specific traits or features of the dataset, such as text casing or image dimensions. |  case, uncased, 1024-1024, 244

Prompt Design Recently, large language models have demonstrated significant
capabilities in natural language understanding (Bubeck et al., 2023). We chose
to utilize the ChatGPT API111https://openai.com/gpt-4 to develop a pipeline
for assessing non-standardized naming conventions in PTMs.

Initially, we crafted a textual input (i.e., prompt) and conducted a
preliminary analysis on 20 randomly selected models. This process underwent
iterative refinement to achieve two primary objectives: (1) Ensuring the
generated output adheres to a well-defined, easily parsable format; (2)
Optimizing the performance of the output, evaluated through manual inspection.
In our initial approach, we utilized semantic patterns identified from the
preliminary analysis to establish our taxonomy. During the pipeline
development, we observed that several patterns consistently appeared as others
in our results. To address this, we systematically analyzed these recurring
patterns and incorporated them into our existing taxonomy. Concurrently, we
updated our prompt to include these additional patterns, accompanied by their
respective definitions and examples.

We then conducted a pilot study of 20 PTMs for 10 iterations. We observed that
no new patterns were classified under others, suggesting that our taxonomy had
reached saturation (Charmaz, 2006). By the end of the pilot study, our
taxonomy had expanded to include five additional patterns: training method,
number of layers, number of heads, number of parameters, and dataset
characteristics. More details of our taxonomy can be found in Table 2.

To standardize the semantic naming patterns, we enhanced the prompt by adding
definitions and providing examples for each pre-defined component. To control
the output format, we employed few-shot learning techniques, which involved
supplying example inputs and their corresponding expected outputs. To further
optimize the output accuracy, we incorporated an additional requirement: each
output segment should include the top three most likely categories,
accompanied by their respective confidence values.

Evaluation We randomly sampled 50 models from the model list. One researcher
manually labeled the semantic patterns of their names. We compared our manual
labels to the output from the pipeline, and calculated the Cohen’s Kappa Value
(0.68). This indicate that our results have our results from the pipeline are
reliable.

####  6.1.2. Syntactic Pattern Analysis

We also observed that the naming styles of PTM packages exhibit strong
similarities to traditional method naming conventions. Specifically, both
share common stylistic patterns such as camelCase, kebab-case, and snake_case
(Alsuhaibani et al., 2021; Lawrie et al., 2007a). In our study, snake_case and
kebab-case can contain capital letters. We define mix_ed-case as a combination
of kebab-case and snake_case. The regex patterns used are presented in Table
3. The frequency of each pattern was measured in the complete names of the PTM
identifier under consideration. Additionally, we examined the relationship
between semantic and syntactic patterns in PTM naming.

Table 3.  Regex patterns for different naming conventions using Python notation.  Syntactic Patterns | Regex Pattern
---|---
camelCase | ^[a-z][a-zA-Z0-9]*$
PascalCase | ^[A-Z][a-zA-Z0-9]*$
snake_case | ^(?=.*_)[a-zA-Z0-9]+(_[a-zA-Z0-9]+)*$
kebab-case | ^(?=.*-)[a-zA-Z0-9]+(-[a-zA-Z0-9]+)*$
mix_ed-case | ^(?=.*-)(?=.*_)[a-zA-Z0-9]+([-_][a-zA-Z0-9]+)*$

###  6.2. Results

Finding 1: (Semantic patterns) Architecture is the basic semantic component of
a PTM package name. Most of the model names include architecture size,
dataset, and task. Finding 2: (Syntactic patterns) The most common syntactic
pattern is kebab-case (85.10%). There are some mix_ed-case (7.67%) and
snake_case (5.36%). We did identify PascalCase and camelCase in the syntactic
naming patterns of open PTM packages.

####  6.2.1. Semantic Patterns

Figure 7(a) is the distribution of the top-20 semantic naming patterns of PTM
packages from Hugging Face. The common semantic patterns are A-S-C-F-D and
A-S-C-F-T. This suggests that for many PTM packages, the architecture, model
size, dataset characteristics, reuse method, and either the dataset or the
task are the key considerations and many model names are well structured.
These specific patterns likely are models fine-tuned and customized for
particular applications or datasets. Simpler patterns like A or A-S may
represent foundational models or those with other unspecified characteristics,
respectively. A-S-L and A-S-D point to a focus on architecture, size, and
language or dataset, which might be commonly seen in natural language
processing or multi-lingual applications. The appearance of patterns like A-A-
S-F-D-L suggests that there exist complex models with multiple layers of
specification, perhaps hinting at high customization or particular complexity
in those cases. Overall, the diversity in semantic components reflected in
these patterns indicates a wide range of use-cases and customizability in PTM
packages.

Figure 7(b) shows the frequency of each naming category in open PTMs. The most
common semantic naming categories are architecture (A), size (S), and dataset
(D). The results indicate that

(a)

(b)

Figure 7.  Frequency of semantic naming (a) patterns and (b) categories (open
PTM packages). Axis labels per Table 2, e.g., a naming pattern is “A-S-C-F-D”
or “Arch-Size-dataset Characteristic-Finetuning-Dataset”.

####  6.2.2. Syntactic Patterns

Figure 8. Frequency of PTM package syntactic naming patterns, for open and
gated PTMs. We discuss this gated PTM data in section 9.

The frequency of syntactic naming patterns in PTM packages is illustrated in
Figure 8. In the case of PTM packages from Hugging Face, the three most common
syntactic naming patterns are kebab-case, mix_ed-case, and snake_case. These
patterns account for 85.10% (6528/8744), 7.67% (588/8744), and 5.36%
(411/8744) of the cases, respectively. Notably, PascalCase and camelCase are
not identified in PTM package names.

The popularity of kebab-case has several potential factors. One explanation
for the prevalence of kebab-case is that it is often preferred for web URLs,
as it is SEO-friendly and easily readable by both humans and machines. In
Hugging Face, PTM names are used in both APIs and page URLs.

This limited usage of PascalCase and camelCase could also be attributed to
some factors. Firstly, the semantic naming components in PTM package names
often encapsulate a multitude of information (Table 2). When this variety of
information is combined into a single string without clear delimiters like
hyphens or underscores, the result can be challenging to read. PascalCase and
camelCase, by their nature, do not provide these delimiters, making the names
harder to parse at a glance. Secondly, the lack of visual breaks in these
naming patterns could make them less suitable for machine parsing, which is
often an important consideration in PTM packages. This is especially critical
when these names are used in contexts like URLs, configuration files, or
database entries where readability and ease of parsing are crucial.

##  7\. DNN Architecture Assessment Pipeline (DARA)

Figure 9.  To identify naming defects, we propose an algorithm of DNN
ARchitecture Assessment pipeline (DARA). Our goal is to automatically identify
potential naming defects, including mislabeled and under-labeled names. To
achieve this goal, we automatically identify the architectural variations
among multiple PTMs, subsequently classifying them into distinct clusters and
identifying outliers.

section 7 presents an overview of $\boldsymbol{D}$NN
$\boldsymbol{AR}$chitecture $\boldsymbol{A}$ssessment ($\boldsymbol{DARA}$).
DARA first converts a PTM to abstract architecture, then extracts features for
clustering and outlier detection.

###  7.1. Pipeline Overview

Figure 9 delineates the workflow of the DARA pipeline. The process commences
with open-source pre-trained weights. Utilizing Torchview (Tor, 2023), an
open-source tool, we transform DNNs into computational graphs. It is important
to note that some DNNs may have identical architectural designs but could
differ in their implementations, leading to slightly different computational
graphs. To address this, we introduce a layer sorting algorithm, facilitating
the extraction of a shared abstract architecture from these isomorphic models.
Using our layer sorting algorithm, we were able to avoid the isomorphisms when
converting the models to computational graphs.

Furthering our methodology, we apply a 2-gram feature extraction technique,
extracting the abstract architecture into feature vectors. Conclusively, a
model clustering algorithm is executed on these vectors. This not only
automates the categorization of models into distinct groups but also help
identify outliers. Our model clustering algorithm discerns architectural
differences, enabling differentiation even between PTMs in the same family,
like ResNet-50 and ResNet-101.

###  7.2. Feature Extraction

####  7.2.1. Layer Sorting Algorithm

Our proposed algorithm provides a novel way to serialize PTMs, addressing the
complexities of varying branch orders in the architectures. Our layer sorting
algorithm takes a pre-trained weight as input and outputs a serialized list of
its layers. The algorithm traverses the model layer-by-layer, assigning a
unique sorting hash id to each layer based on its connections. This sorting
hash id is computed recursively to ensure that connected layers are
appropriately ordered, ultimately leading to a sorted list of layers in the
model architecture. This algorithm can convert a model architecture into a
list of layers following a specific order, ensuring that equivalent models
with swapped branches yield identical ordered lists.

Our method treats all neural network graphs as Directed Acyclic Graphs (DAGs),
breaking cycles in models like LSTM (Hochreiter and Schmidhuber, 1997) by
removing redundant edges. This turns every network into a DAG with “complete
reachability”, where each node exists along a path between an input and an
output. This ensures that our algorithm captures all layers that contribute to
the network’s function, making it both comprehensive and efficient.

Our depth-first search (DFS)-based topological sorting algorithm uniquely
identifies each node using a hash generated from its own information and that
of its child nodes. This ensures consistent node priority across different
neural network graphs and allows for deterministic ordering of PTMs. The
unique identifier makes our sorting method robust and capable of meaningfully
comparing different PTM architectures.

####  7.2.2. Abstract Architecture

Prior work proposed abstract neural network (ANN) (Nguyen et al., 2022). We
proposed a similar concept, abstract PTM architecture, which not only includes
layer information, parameters, input/output dimensions, but also has the
connection information between each node. Our abstract architecture stores the
key information of each PTM architecture. Different to ANN introduced by prior
work, our abstract PTM architecture not only contains layer and parameter
information, but also additional information about the layer connections which
are not directly available from API calls (Nguyen et al., 2022). We publish
our abstract architecture as a dataset for future use.

####  7.2.3. Vectorization

It is insufficient to identify naming defects solely based on the
indentifiers. In this study, we propose a method for vectorizing PTMs using
three fundamental components that define their architecture: layer
connections, and layer parameters. Layer connections represent the structural
configuration of the PTM checkpoint (e.g., (Conv2D, BatchNormalization)) The
parameters can encapsulate the network’s hyperparameters (e.g., <kernel_size:
[1, 1]> in a Conv2D layer). By representing these three aspects as vectors
using a frequency-based approach, we transform each DNN into a pair of
vectors, and then concatenate them together.

Prior work shows that different frameworks can have different computational
graphs (Zhang et al., 2019). For example, TensorFlow use static computational
graph, while PyTorch is dynamic. To directly compare the model architectures,
we therefore converted the models from different frameworks to ONNX
representations. For the model converters, we used keras2onnx and torch2onnx
which are both gated model converters for each framework. Jajal et al. note
that converters can add error (Jajal et al., 2023), so we tested each
converted model to verify the conversion was correct.

The primary objective of this process is to enable comparison of DNN
architectures. By observing the variation in these vector representations, we
can intuitively cluster DNN architectures, offering insights into the
influence of structural and hyperparameter choices on network behavior.

###  7.3. Model Clustering

We first apply PCA to reduce the dimension of our feature vectors, then adapt
the DBSCAN clustering algorithm to them. In DBSCAN, the parameter $\epsilon$
is used to determine the radius of the neighborhood around each data point,
effectively influencing the size and structure of each cluster. In DBSCAN, a
cluster is a dense region of points, which is separated by regions of lower
point density from other dense regions. Outliers are points that do not belong
to any cluster and are far from any dense regions. Therefore, the choice of
$\epsilon$ plays a significant role in determining the granularity of clusters
and the identification of outliers.

To select an appropriate $\epsilon$ value for the DBSCAN algorithm, we
conducted a grid search and analyzed how varying $\epsilon$ affects the
clustering outcomes. We calculated the gradient of the sum of the rates of
change in the number of clusters and the number of outliers. The gradient
effectively encapsulates how sensitive the clustering results are to changes
in $\epsilon$. The script then identifies the $\epsilon$ value where the
absolute value of this sum of gradients is minimized. This selection indicate
that: (1) we tried to maximize the number of outliers and the corresponding
false positives, which we can manually filter out in further analysis; (2)
beyond this selected $\epsilon$ value, the clustering algorithm becomes less
stable. In such cases, models that should be clustered together are at a
greater risk of being identified as outliers. We consider this value optimal
$\epsilon$, as it minimizes the rate of change in both the number of clusters
and outliers, thereby providing a more stable clustering solution. Finally, we
automatically select $\epsilon$ as 0.027 for open models and 0.01 for gated
models.

###  7.4. Naming Space and Weight Space

DARA provides informative information from the pre-trained weights which are
not directly available in model names or metadata. We leverage this
information to identify potential naming defects. The clustering results from
DARA output can be considered as an “weight space” where the information is
provided by information from the pre-trained weights. On the other hand, the
outputs we get from GPT-4 provide a space for PTM package names, i.e., “naming
space”. To effectively identify potential naming defects, we can map the
semantic naming patterns from the naming space to the clustered groups in the
architecture space.

##  8\. RQ2: What are the naming defects and frequencies in Hugging Face?

###  8.1. Method

####  8.1.1. Stratified Sampling

We employ a stratified sampling method based on semantic naming patterns. From
our preliminary analysis (section 6.1.1), we created a list of common semantic
name elements, such as “Architecture” and “Dataset” (Table 2). In principle,
any of these elements could be incorrect, yielding naming defects. To examine
the distribution of these defects, we utilize stratified sampling. For each
semantic naming element, we enumerate all model names containing that element,
and randomly select 50 model names for further analysis.

####  8.1.2. Architecture analysis

The output of DARA includes a list of outliers whose architecture are likely
to be incorrectly labeled. These architectures might be wrong or incorrectly
implemented, therefore they do not match the actual architecture of the model,
and were clustered as outliers.

To analyze outliers, two researchers independently categorized the root causes
with our taxonomy. We measured inter-rater agreement with Cohen’s Kappa
(Cohen, 1960). The value prior to settling disagreement was 0.65 for open
PTMs, and 0.64 for gated PTMs (“moderate” agreement).

####  8.1.3. Semantic Defect analysis

As discussed in section 7.4, we can manually map a PTM package name to the
“weight space” generated using DARA. This weight space offers additional
insights into layer connections and parameters of the model.

There are connections between the naming space and the weight space: The layer
connection features relate to the model’s architecture, size, versioning, and
language. The parameter features are relevant to the model size, language, and
dataset characteristics. Leveraging the insights from DARA, we manually map
the names to the clusters to identify potential naming defects.

####  8.1.4. Automated filter

During out defect analysis, we identified nuance distinctions of wrong
architecture in identifiers. We search if any architecture identifiers exist
in the PTM identifier, if not, the model is categorized as a mislabeled model.
The mislabeled model is classified by using the semantic pattern generated by
GPT-4. Any model name without an Architecture naming component is marked
uninformative model. Any model that the Architecture naming component that is
not in the Hugging Face model type is classified as wrong architecture in the
classifier.

###  8.2. Results

Finding 3: Using stratified sampling methods, we discovered that 6% of the
architectures and 4% of the model sizes in the identifiers were incorrect. Our
automated filtering system effectively identified these potentially erroneous
architectures, accounting for 20% of the post-filter samples. Finding 4: Among
all model names, we identified 12.8% uninformative identifier and 1.1% wrong
architecture in metadata.

Incorrect Identifiers Table 4 presents the defects we have identified.
Utilizing data from DARA, we successfully detected incorrect architectures in
6% of the identifiers. Our automated filter effectively elevates this rate,
capturing 20% of defects in the post-filter samples.

We also flagged 4% of the identifiers for having incorrect model sizes. For
instance, the model albert-large-urdu is clustered with albert-base models,
not the albert-large models — this model uses the wrong size (large vs. base)
in its name. With the architectural data supplied by DARA, we were able to
detect such anomalies.

One limitation of the DARA algorithm is its focus solely on layer connections
and parameters for the clustering algorithm, neglecting the significance of
input and output values which are more indicative of the actual weights.
Consequently, we could not observe other types of defects that are connected
to these weights, such as those related to dataset, language, or fine-tuning
methodology.

Uninformative Identifier Our automated filtering mechanism successfully
identified a significant portion of models with uninformative identifiers,
accounting for 12.8% of the total. These identifiers, such as test-v1 or eval-
base, lack explicit information about the architecture, making it challenging
to glean any meaningful insights solely based on their names. This issue is
non-trivial within the PTM ecosystem, not only do these ambiguously-named
models exist, but they are also actively utilized. Our data reveals that each
of these models has garnered a minimum of 10 downloads, suggesting that their
lack of informative identifiers has not deterred usage. The prevalence and
active use of such models highlight the need for more transparent and
descriptive naming conventions to enhance understandability and effective
deployment within the community.

Incorrect Architecture Label Our in-depth architecture analysis, detailed in
Section 8.1.2, successfully identifies instances of incorrectly labeled
architectures. These are detected as outliers by DARA. Among the 149 outliers
identified from open PTMs, we discovered 4 mislabeled and 6 underlabeled
architectures. These PTMs have a significant rate of re-use, with monthly
downloads ranging from 10 to 59,981. The mislabeled architectures have
metadata that does not align with their actual architecture. On the other
hand, underlabeled architectures are mostly accurate but are built upon
different, more complex architectures than indicated in their metadata. We
have opened issues regarding these defects and received one reply from Meta.
Their response indicates that the current infrastructure was not able to keep
up with the rapid development of PTMs, illustrating deficiencies in the RTP
within the Hugging Face ecosystem.

Such mislabeling presents a significant risk to the integrity of the PTM
ecosystem. Unscrupulous contributors could potentially upload models with
embedded adversarial attacks or malware, falsely claiming to utilize well-
known, trusted architectures. This deceptive practice can critically
compromise security. When engineers or data scientists rely on architecture
labels to search for and implement these models, their systems become
vulnerable to these concealed risks. Our findings demonstrate that our tool is
effective in proactively identifying such inaccuracies and potential security
risks, thereby contributing to the defense of the overall PTM ecosystem.

Table 4.  Detailed taxonomy and frequency of naming defects induced from our dataset, refined from our original taxonomy in Table 1). The defect types in first group are all refined from the category “Incorrect identifier component” in the original taxonomy.  Defect types | Frequencies | Proportion (%)
---|---|---
Incorrect architecture in identifier | 3/50, 10/50 (after filter) | 6%, 20% (after filter)
Incorrect size in identifier | 2/50 | 4%
Incorrect dataset in identifier | 0/50 | 0%
Incorrect versioning in identifier | 0/50 | 0%
Incorrect language in identifier | 0/50 | 0%
Incorrect fine-tuning method in identifier | 0/50 | 0%
Uninformative identifier | 1119/8744 | 12.8%
Mislabeled architecture | 4/8744 | 0.5%
Underlabeled architecture | 6/8744 | 0.7%

##  9\. RQ3: Do our findings on naming patterns and defects generalize to
other model hubs?

Finding 5: Gated PTM package names are more standardized. Almost all of them
(97.55%) include architectures. Gated model hubs have a better versioning
system where 15.76% of packages are versioned, double the rate in open model
hubs (7.21%). Finding 6: The top-3 syntactic patterns in open and gated model
hubs are the same, but mix_ed-Case are more widely used in gated model hubs.
Finding 7: The naming defect rates of open PTM packages is at least 20%, while
the rate of gated PTM packages is 0%.

###  9.1. Method

To assess the generalizability of our findings discussed in section 6 and
section 8, we extend our analysis to include additional model hubs. As
outlined in section 5.2, our data collection encompasses both open and gated
Pre-Trained Model (PTM) packages. We apply the methodologies detailed in
section 6.1 and section 8.1 to the PTMs in the gated model hubs, and then
compare results between open and gated model hubs.

###  9.2. Results

####  9.2.1. Naming Patterns

The semantic and syntactic naming patterns and defects of gated PTM packages
are shown in Figure 10 and Figure 8 correspondingly.

The top-3 semantic patterns in gated PTM packages are A, A-S, and A-V-S
(Figure 10(a)). In contrast, the top-3 patterns in open PTM packages are A-S-
C-F-D, A-S-C-F-T, and A-S-L (Figure 7(a)). Unlike open PTMs, which are often
fine-tuned downstream models for multiple datasets and tasks, gated PTMs tend
to be more foundational. Their identifiers mostly contain information about
the architecture, size, or version. In the context of the RTP pipeline, gated
PTMs are generally closer to research prototypes, while open PTMs are closer
to deployment or practical use.

Our results also indicate that naming conventions are more standardized among
gated PTM packages. They tend to use less distributed semantic naming patterns
(Figure 10(b)), making them more directly comparable to each other. In
comparison to open PTMs, different semantic categories appear to be more
evenly distributed (Figure 7(a)).

Concerning syntactic naming patterns (Figure 8), gated model hubs tend to use
more mix_ed-case compared to open model hubs. We noticed that mix_ed-case is
often employed across different semantic components of the identifier.

####  9.2.2. Naming Defects

We applied the same defect analysis methodology to gated PTM package names.
However, after manually all the 87 outliers we identified from gated model
hubs, we found no naming issues. A possible reason is because of the limited
number of gated PTM packages per architecture we collected, thus some of the
PTMs with a different model size were clustered as outliers in our algorithm.
In contrast, the defect analysis of open PTM packages revealed at least 23%
defects. This suggests that gated model hubs are generally more reliable and
trustworthy compared to their open counterparts.

(a)

(b)

Figure 10.  The frequency of (a) semantic naming patterns (b) semantic naming
categories in gated PTM packages. Axis labels follow Table 2, e.g., a common
naming pattern is “A-V-S” or “Architecture-Version-Size”.

##  10\. Discussion and Future Work

###  10.1. A Naming Convention for PTMs

Although there is no official naming standard for PTMs, our data allow us to
issue a recommendation based on popular practice. Specifically, to follow
community norms, a PTM name should:

  * •

Semantically, prefer longer names incorporating more details of the PTM. Favor
A-S-C-F (Architecture, model Size, dataset Characteristic, and Finetuned reuse
method) and consider including the Dataset name and/or the Task. (Finding 1).

  * •

Syntactically, use kebab-case as $\sim$80% of PTMs do. (Finding 2).

This approach will follow norms in both open and gated model hubs (Finding 5).

This naming convention is derived from community practice, but that does not
mean it is ideal. Empirical research can be used to refine it and explore
alternatives. Such work could build on that of Alsuhaibani et al., who
examined the quality and practicality of naming conventions of
methods/functions through a survey study on professional developers
(Alsuhaibani et al., 2021).

###  10.2. Naming Conventions as Signal of RTP Relationships

Expanding on our exploration of naming conventions, we further examine the
less-studied aspects of their relevance to the RTP pipeline. While traditional
package registries and GitHub usually make RTP relationships straightforward —
often indicated by “forking” — PTM packages present a more complex scenario.
These complexities include the variability in semantic naming patterns, which
may involve terms specific to model architectures or data preprocessing
techniques. This makes it challenging to straightforwardly infer upstream or
downstream dependencies. Furthermore, the lack of standardized metadata in PTM
model hubs complicates efforts to understand relationships like code
reusability, versioning. Nonetheless, our research indicates that the semantic
naming patterns of PTM packages can act as useful markers for their role
within the RTP pipeline. For instance, the presence of terms like “finetuning”
and “dataset” in a model name suggests likely dependencies on upstream models.
This nuanced understanding of PTM naming allows downstream users to more
strategically select packages aligned with their specific requirements.

Our work is grounded in the Research-To-Practice pipeline, but many
implications of that pipeline still remain unexamined. One natural question is
whether package names can be used to infer upstream/downstream relationships
between packages. The existing metadata provided by model hubs makes this
inference difficult, with implications in areas such as PTM re-use, patch
propagation, and vulnerability mitigation. As a potential direction, we
observe that in gated models — where researchers often introduce models — the
most common naming pattern was simply “A”, the Architecture (Figure 10(a)). In
contrast, in open models — where re-engineers adapt models — the most common
naming patterns include 3-5 additional semantic features, implying greater and
greater degrees of specialization. To what extent do PTMs that share prefixes
also share heritage?

###  10.3. Leveraging Meta-features of PTMs

To Improve DARA Although DARA can identify real naming defects, the features
used by the DARA algorithm are weak. N-grams discard much information about
the PTM architecture. In future work, we propose to develop stronger feature
extraction methods by converting the architecture to embeddings. Contrastive
learning (Ni et al., 2021) could also mitigate some of the weaknesses of the
DBSCAN clustering algorithm. Recent advancements have also unveiled techniques
for encoding directed graphs directly using transformers (Geisler et al.,
2023). Applying these feature extraction techniques could improve DARA’s
ability to detect naming defects.

To Support Model Search and Reuse Our dataset of abstract PTM architectures
includes meta-features suitable for various tasks, such as identifying
potentially mislabeled architectures. Nguyen et al. demonstrated the utility
of meta-features from PTM implementations on GitHub for enhancing AutoML
performance (Nguyen et al., 2022). Studies have also explored effective
support and reuse of PTMs through novel model selection methods (You et al.,
2021a, b; Lu et al., 2019). We propose applying PTMTorrent and DARA to extract
these meta-features and devise techniques to support large-scale AutoML and
efficient model selection, ultimately simplifying PTM adoption.

To Recommend PTM Names Given established naming standards, Alsuhaibani et al.
suggested building a tool to automatically assess the quality of method names
(Alsuhaibani et al., 2021). Gao et al. defined the method naming convention
problem as learning the conventional mapping from natural language to concise
and meaningful methods, and showed the capability of neural models to generate
method names based on function description (Gao et al., 2019). Similarly,
leveraging neural models, such as natural language processing tools, the
research community can further develop a naming recommendation system for
method names. In a similar vein, our work introduces a naming standard in
section 10.1. We posit that a comparable tool or system can be developed for
PTM naming. Such a tool would promote a standardized PTM supply chain and
potentially enhance its trustworthiness.

##  11\. Threats to Validity

Construct We operationalize the concept of PTM names in terms of the
identifier and architecture. Alternative operationalizations might incorporate
package type or package contents. However, per our domain model (Figure 2), we
believe our operationalization covers the elements that involve judgment on
the part of the namer.

Internal We made no claims of cause and effect. Our conclusions might be
influenced by the subjective (manual) element of our evaluation — we used
inter-rater agreement and reported Kappa scores where appropriate, and these
scores were reasonable.

External We acknowledge two threats to generalizability. (1) We used GPT-4 to
analyze model names, which would affect performance of our system on models
introduced after the GPT-4 training date. As mitigation, it happens that all
the PTMs we used from the PTMTorrent dataset were published before the release
of GPT-4. (2) The PTMTorrent dataset reflects HuggingFace models as of January
2023. This predates the growth of large language models (LLMs). LLMs may have
changed naming practices due to the influx of new users and new model types.

##  12\. Conclusion

Pre-Trained Models (PTMs) are a new unit of re-use in software engineering. We
defined the nature of naming in the PTM ecosystem via the lens of the PTM
Research-To-Practice pipeline. We conducted the first study of PTM naming
conventions. We taxonomized possible PTM naming defects and developed a
methodology and algorithm to detect them. We found a low, but consequential,
rate of naming defects in the open HuggingFace model hub. These defects add
friction to PTM re-use, slowing the progress of the software engineering
community in adopting this emerging technology. Our algorithm provides a means
to automatically detect these defects.

To conclude: Pre-Trained Models are a new unit of re-use. Like previous units
of re-use, engineers struggle to name PTMs well. There are many opportunities
for future work to support engineers: in best practices for naming, in model
search, and in improved automated naming error detection.

## Acknowledgments

This work was supported by gifts from Google and Cisco and by NSF awards
#2107230, #2229703, #2107020, and #2104319. A. Srivastava and S. Vummaji
helped with the collection of syntactic naming patterns and defect analysis.
We thank A. Kazerouni and members of the Purdue Duality Lab for their feedback
on the manuscript.

## References

  * (1)
  * Ker ([n. d.]) [n. d.].  Keras: the Python deep learning API.  https://keras.io/
  * Tor (2023) 2023\.  TorchView: Visualize Pytorch Model.  https://torchview.dev/
  * Alsuhaibani et al. (2021) Reem Alsuhaibani, Christian Newman, Michael Decker, Michael Collard, and Jonathan Maletic. 2021\.  On the Naming of Methods: A Survey of Professional Developers. In _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_. 587–599.  https://doi.org/10.1109/ICSE43902.2021.00061
  * Banna et al. (2021) Vishnu Banna, Akhil Chinnakotla, Zhengxin Yan, Ani Vegesana, Naveen Vivek, Kruthi Krishnappa, Wenxin Jiang, Yung-Hsiang Lu, George K. Thiruvathukal, and James C. Davis. 2021.  An experience report on machine learning reproducibility: Guidance for practitioners and TensorFlow model garden contributors.  https://arxiv.org/abs/2107.00821
  * Bhatia et al. (2023) Aaditya Bhatia, Ellis E. Eghan, Manel Grichi, William G. Cavanagh, Zhen Ming Jiang, and Bram Adams. 2023.  Towards a change taxonomy for machine learning pipelines: Empirical study of ML pipelines and forks related to academic publications.  _Empirical Software Engineering_ 28, 3 (May 2023), 60.  https://doi.org/10.1007/s10664-022-10282-8
  * Binkley et al. (2013) Dave Binkley, Marcia Davis, Dawn Lawrie, Jonathan I Maletic, Christopher Morrell, and Bonita Sharif. 2013.  The impact of identifier style on effort and comprehension.  _Empirical software engineering_ 18 (2013), 219–276.
  * Bozeman (2000) Barry Bozeman. 2000\.  Technology transfer and public policy: a review of research and theory.  _Research Policy_ 29, 4-5 (April 2000), 627–655.  https://doi.org/10.1016/S0048-7333(99)00093-1
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  _Language Models are Few-Shot Learners_.  Technical Report arXiv:2005.14165. arXiv.  http://arxiv.org/abs/2005.14165
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023\.  Sparks of Artificial General Intelligence: Early experiments with GPT-4.  http://arxiv.org/abs/2303.12712
  * Butler et al. (2010) Simon Butler, Michel Wermelinger, Yijun Yu, and Helen Sharp. 2010.  Exploring the Influence of Identifier Names on Code Quality: An Empirical Study. In _2010 14th European Conference on Software Maintenance and Reengineering_. 156–165.  https://doi.org/10.1109/CSMR.2010.27
  * Charmaz (2006) Kathy Charmaz. 2006\.  _Constructing grounded theory: A practical guide through qualitative analysis_.  sage.
  * Chen et al. (2015) Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. 2015\.  Net2net: Accelerating learning via knowledge transfer.  _arXiv preprint arXiv:1511.05641_ (2015).
  * Cohen (1960) Jacob Cohen. 1960\.  A coefficient of agreement for nominal scales.  _Educational and psychological measurement_ 20, 1 (1960), 37–46.
  * Davis et al. (2023) James C. Davis, Purvish Jajal, Wenxin Jiang, Taylor R. Schorlemmer, Nicholas Synovic, and George K. Thiruvathukal. 2023.  Reusing Deep Learning Models: Challenges and Directions in Software Engineering. In _Proceedings of the IEEE John Vincent Atanasoff Symposium on Modern Computing (JVA’23)_.
  * Face (2023) Hugging Face. 2023\.  Hugging Face Documentations.  https://huggingface.co/docs
  * Gao et al. (2019) Sa Gao, Chunyang Chen, Zhenchang Xing, Yukun Ma, Wen Song, and Shang-Wei Lin. 2019\.  A neural model for method name generation from functional description. In _International Conference on Software Analysis, Evolution and Reengineering (SANER’19)_. IEEE, 414–421.
  * Garcia et al. (2020) Joshua Garcia, Yang Feng, Junjie Shen, Sumaya Almanee, Yuan Xia, and Qi Alfred Chen. 2020\.  A comprehensive study of autonomous vehicle bugs. In _International Conference on Software Engineering (ICSE)_.  https://dl.acm.org/doi/10.1145/3377811.3380397
  * Geisler et al. (2023) Simon Geisler, Yujia Li, Daniel J Mankowitz, Ali Taylan Cemgil, Stephan Günnemann, and Cosmin Paduraru. 2023.  Transformers meet directed graphs. In _International Conference on Machine Learning (ICML’23)_. PMLR, 11144–11172.
  * Google (2022) Google. 2022.  TensorFlow Model Garden.  https://github.com/tensorflow/models
  * Gopalakrishna et al. (2022) Nikhil Krishna Gopalakrishna, Dharun Anandayuvaraj, Annan Detti, Forrest Lee Bland, Sazzadur Rahaman, and James C Davis. 2022.  “If security is required”: Engineering and Security Practices for Machine Learning-based IoT Devices. In _International Workshop on Software Engineering Research & Practices for the Internet of Things (SERP4IoT)_.
  * Grima-Farrell (2017) Christine Grima-Farrell. 2017\.  The RTP Model: An Interactive Research to Practice Framework.  _What Matters in a Research to Practice Cycle? Teachers as Researchers_ (2017), 237–250.
  * Gu et al. (2023) Yacong Gu, Lingyun Ying, Yingyuan Pu, Xiao Hu, Huajun Chai, Ruimin Wang, Xing Gao, and Haixin Duan. 2023\.  Investigating package related security threats in software registries. In _2023 IEEE Symposium on Security and Privacy (SP)_. IEEE, 1578–1595.
  * Guo et al. (2023) Wenbo Guo, Zhengzi Xu, Chengwei Liu, Cheng Huang, Yong Fang, and Yang Liu. 2023\.  An Empirical Study of Malicious Code In PyPI Ecosystem. In _Internaltional Conference on Automated Software Engineering (ASE’23)_.
  * Han et al. (2021) Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang, Liang Zhang, Wentao Han, Minlie Huang, Qin Jin, Yanyan Lan, Yang Liu, Zhiyuan Liu, Zhiwu Lu, Xipeng Qiu, Ruihua Song, Jie Tang, Ji-Rong Wen, Jinhui Yuan, Wayne Xin Zhao, and Jun Zhu. 2021\.  Pre-trained models: Past, present and future.  _AI Open_ 2 (2021), 225–250.
  * Hindle et al. (2016) Abram Hindle, Earl T Barr, Mark Gabel, Zhendong Su, and Premkumar Devanbu. 2016.  On the naturalness of software.  _Commun. ACM_ 59, 5 (2016), 122–131.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997), 1735–1780.
  * Jajal et al. (2023) Purvish Jajal, Wenxin Jiang, Arav Tewari, Joseph Woo, Yung-Hsiang Lu, George K Thiruvathukal, and James C Davis. 2023.  Analysis of Failures and Risks in Deep Learning Model Converters: A Case Study in the ONNX Ecosystem.  _arXiv_ (2023).  https://arxiv.org/abs/2303.17708
  * Jiang et al. (2017) Fei Jiang, Yong Jiang, Hui Zhi, Yi Dong, Hao Li, Sufeng Ma, Yilong Wang, Qiang Dong, Haipeng Shen, and Yongjun Wang. 2017.  Artificial intelligence in healthcare: past, present and future.  _Stroke and vascular neurology_ 2, 4 (2017).
  * Jiang et al. (2023a) Wenxin Jiang, Vishnu Banna, Naveen Vivek, Abhinav Goel, Nicholas Synovic, George K Thiruvathukal, and James C Davis. 2023a.  Challenges and practices of deep learning model reengineering: A case study on computer vision.  _arXiv preprint arXiv:2303.07476_ (2023).
  * Jiang et al. (2023b) Wenxin Jiang, Nicholas Synovic, Matt Hyatt, Taylor R. Schorlemmer, Rohan Sethi, Yung-Hsiang Lu, George K. Thiruvathukal, and James C. Davis. 2023b.  An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry. In _IEEE/ACM 45th International Conference on Software Engineering (ICSE’23)_ (Melbourne, Australia).
  * Jiang et al. (2023c) Wenxin Jiang, Nicholas Synovic, Purvish Jajal, Taylor R Schorlemmer, Arav Tewari, Bhavesh Pareek, George K Thiruvathukal, and James C Davis. 2023c.  PTMTorrent: A Dataset for Mining Open-source Pre-trained Model Packages.  _Proceedings of the 20th International Conference on Mining Software Repositories (MSR’23)_ (2023).
  * Jiang et al. (2022) Wenxin Jiang, Nicholas Synovic, and Rohan Sethi. 2022\.  An Empirical Study of Artifacts and Security Risks in the Pre-trained Model Supply Chain.  _Los Angeles_ (2022), 10\.
  * Kathikar et al. (2023) Adhishree Kathikar, Aishwarya Nair, Ben Lazarine, Agrim Sachdeva, and Sagar Samtani. 2023\.  Assessing the Vulnerabilities of the Open-Source Artificial Intelligence (AI) Landscape: A Large-Scale Analysis of the Hugging Face Platform.  (2023).
  * Knuth (1984) Donald Ervin Knuth. 1984\.  Literate programming.  _The computer journal_ 27, 2 (1984), 97–111.
  * Ladisa et al. (2022) Piergiorgio Ladisa, Henrik Plate, Matias Martinez, and Olivier Barais. 2022.  Taxonomy of Attacks on Open-Source Software Supply Chains.  http://arxiv.org/abs/2204.04008
  * Ladisa et al. (2023) Piergiorgio Ladisa, Henrik Plate, Matias Martinez, and Olivier Barais. 2023.  SoK: Taxonomy of Attacks on Open-Source Software Supply Chains. In _2023 IEEE Symposium on Security and Privacy (SP)_. 1509–1526.  https://doi.org/10.1109/SP46215.2023.10179304
  * Lawrie et al. (2007a) Dawn Lawrie, Henry Feild, and David Binkley. 2007a.  An empirical study of rules for well-formed identifiers.  _Journal of Software Maintenance and Evolution: Research and Practice_ 19, 4 (2007), 205–229.  https://doi.org/10.1002/smr.350
  * Lawrie et al. (2007b) Dawn Lawrie, Christopher Morrell, Henry Feild, and David Binkley. 2007b.  Effective identifier names for comprehension and memory.  _Innovations in Systems and Software Engineering_ 3, 4 (Nov. 2007), 303–318.  https://doi.org/10.1007/s11334-007-0031-2
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021\.  The Power of Scale for Parameter-Efficient Prompt Tuning.  http://arxiv.org/abs/2104.08691
  * Liu et al. (2022) Bo Liu, Ming Ding, Sina Shaham, Wenny Rahayu, Farhad Farokhi, and Zihuai Lin. 2022\.  When Machine Learning Meets Privacy: A Survey and Outlook.  _Comput. Surveys_ (2022).
  * Lu et al. (2019) Bingqian Lu, Jianyi Yang, Lydia Y. Chen, and Shaolei Ren. 2019\.  Automating Deep Neural Network Model Selection for Edge Inference. In _2019 IEEE First International Conference on Cognitive Machine Intelligence (CogMI)_. 184–193.  https://doi.org/10.1109/CogMI48466.2019.00035
  * Meta (2022) Meta. 2022.  Torchvision.  https://github.com/pytorch/vision
  * Montes et al. (2022) Diego Montes, Pongpatapee Peerapatanapokin, Jeff Schultz, Chengjun Guo, Wenxin Jiang, and James C Davis. 2022\.  Discrepancies among pre-trained deep neural networks: a new threat to model zoo reliability. In _European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE-IVR track)._
  * Nguyen et al. (2022) Giang Nguyen, Md Johirul Islam, Rangeet Pan, and Hridesh Rajan. 2022.  Manas: mining software repositories to assist AutoML. In _International Conference on Software Engineering (ICSE)_. ACM, Pittsburgh Pennsylvania, 1368–1380.  https://doi.org/10.1145/3510003.3510052
  * Nguyen et al. (2023) Tai Nguyen, Yifeng Di, Joohan Lee, Muhao Chen, and Tianyi Zhang. 2023.  Software Entity Recognition with Noise-Robust Learning. In _International Conference on Automated Software Engineering (ASE’23)_. arXiv.  http://arxiv.org/abs/2308.10564
  * Ni et al. (2021) Renkun Ni, Manli Shu, Hossein Souri, Micah Goldblum, and Tom Goldstein. 2021.  The close relationship between contrastive learning and meta-learning. In _International Conference on Learning Representations (ICLR’21)_.
  * Nidhra et al. (2013) Srinivas Nidhra, Muralidhar Yanamadala, Wasif Afzal, and Richard Torkar. 2013.  Knowledge transfer challenges and mitigation strategies in global software development—A systematic literature review and industrial validation.  _International Journal of Information Management_ 33, 2 (April 2013), 333–355.  https://doi.org/10.1016/j.ijinfomgt.2012.11.004
  * Ohm et al. (2020) Marc Ohm, Henrik Plate, Arnold Sykosch, and Michael Meier. 2020\.  Backstabber’s Knife Collection: A Review of Open Source Software Supply Chain Attacks.  In _Detection of Intrusions and Malware, and Vulnerability Assessment_ , Clémentine Maurice, Leyla Bilge, Gianluca Stringhini, and Nuno Neves (Eds.). Vol. 12223. Springer International Publishing, Cham, 23–43.  http://link.springer.com/10.1007/978-3-030-52683-2_2
  * Okafor et al. (2022) Chinenye Okafor, Taylor R. Schorlemmer, Santiago Torres-Arias, and James C. Davis. 2022.  SoK: Analysis of Software Supply Chain Security by Establishing Secure Design Properties. In _Proceedings of the 2022 ACM Workshop on Software Supply Chain Offensive Research and Ecosystem Defenses_. ACM, Los Angeles CA USA, 15–24.  https://doi.org/10.1145/3560835.3564556
  * ONNX (2023) ONNX. 2023.  ONNX Model Zoo.  https://github.com/onnx/models.
  * Patterson et al. (2021) David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. 2021.  Carbon Emissions and Large Neural Network Training.  https://doi.org/10.48550/arXiv.2104.10350
  * Qi et al. (2023) Binhang Qi, Hailong Sun, Xiang Gao, Hongyu Zhang, Zhaotian Li, and Xudong Liu. 2023\.  Reusing Deep Neural Network Models through Model Re-engineering. In _International Conference on Software Engineering (ICSE’23)_.  http://arxiv.org/abs/2304.00245
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019\.  Language Models are Unsupervised Multitask Learners.  _OpenAI blog_ 1, 8 (2019), 9.
  * Sculley et al. (2015) D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-François Crespo, and Dan Dennison. 2015.  Hidden Technical Debt in Machine Learning Systems. In _Advances in Neural Information Processing Systems_ , Vol. 28. Curran Associates, Inc.  https://proceedings.neurips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
  * Shen et al. (2021) Lujia Shen, Shouling Ji, Xuhong Zhang, Jinfeng Li, Jing Chen, Jie Shi, Chengfang Fang, Jianwei Yin, and Ting Wang. 2021.  Backdoor Pre-trained Models Can Transfer to All. In _Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security_. 3141–3158.  https://doi.org/10.1145/3460120.3485370
  * Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019\.  Energy and Policy Considerations for Deep Learning in NLP.  http://arxiv.org/abs/1906.02243
  * Tkáč and Verner (2016) Michal Tkáč and Robert Verner. 2016.  Artificial neural networks in business: Two decades of research.  _Applied Soft Computing_ 38 (2016), 788–804.
  * Wang et al. (2023) Shangwen Wang, Ming Wen, Bo Lin, Yepang Liu, Tegawendé F. Bissyandé, and Xiaoguang Mao. 2023.  Pre-Implementation Method Name Prediction for Object-Oriented Programming.  _ACM Transactions on Software Engineering and Methodology (TOSEM)_ (2023).  https://doi.org/10.1145/3597203
  * Wikipedia (2023) Wikipedia. 2023\.  Naming convention (programming).  https://en.wikipedia.org/wiki/Naming_convention_(programming)
  * You et al. (2021b) Kaichao You, Yong Liu, Jianmin Wang, Michael I. Jordan, and Mingsheng Long. 2021b.  Ranking and Tuning Pre-trained Models: A New Paradigm of Exploiting Model Hubs.  _The Journal of Machine Learning Research (JMLR)_ 23, 1 (Oct. 2021), 9400–9446.  http://arxiv.org/abs/2110.10545
  * You et al. (2021a) Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. 2021a.  LogME: Practical Assessment of Pre-trained Models for Transfer Learning. In _International Conference on Machine Learning (ICML)_. PMLR, 12133–12143.  https://proceedings.mlr.press/v139/you21b.html
  * Zhang et al. (2021) Jingxuan Zhang, Weiqin Zou, and Zhiqiu Huang. 2021.  An Empirical Study on the Usage and Evolution of Identifier Styles in Practice. In _Asia-Pacific Software Engineering Conference (APSEC)_. IEEE, 171–180.
  * Zhang et al. (2019) Tianyi Zhang, Cuiyun Gao, Lei Ma, Michael Lyu, and Miryung Kim. 2019.  An Empirical Study of Common Challenges in Developing Deep Learning Applications. In _International Symposium on Software Reliability Engineering (ISSRE)_.
  * Zimmermann et al. (2019) Markus Zimmermann, Cristian-Alexandru Staicu, and Michael Pradel. 2019.  Small World with High Risks: A Study of Security Threats in the npm Ecosystem. In _USENIX Security Symposium_.  https://doi.org/10.5555/3361338.3361407