# Capital as Artificial Intelligence

Cesare Carissimo1,∗ Marcin Korecki1,∗,
1ETH Zurich, Switzerland
∗Equal Contribution
<EMAIL_ADDRESS>
###### Abstract

We gather many perspectives on Capital and synthesize their commonalities. We
provide a characterization of Capital as a historical agential system and
propose a model of Capital using tools from computer science. Our model
consists of propositions which, if satisfied by a specific grounding,
constitute a valid model of Capital. We clarify the manners in which Capital
can evolve. We claim that, when its evolution is driven by quantitative
optimization processes, Capital can possess qualities of Artificial
Intelligence. We find that Capital may not uniquely represent meaning, in the
same way that optimization is not intentionally meaningful. We find that
Artificial Intelligences like modern day Large Language Models are a part of
Capital. We link our readers to a web-interface where they can interact with a
part of Capital.

In this paper we wish to study a component of the economic system which most
of us are a part of and which most of us reinforce. In so far as it represents
a system much larger than any individual, we wish to study it for its emergent
properties in a holistic manner. It possesses qualities of the artificial as
it lives in quantities, and its creation is inextricable from the work of man.
It also possesses qualities of the emergent which produce behaviour that can
appear agential and intelligent. Alongside current reductionist perspectives,
we believe we must also study this system as if it were an organism, a sort of
artificial life.

The digital revolution of the 20th and 21st centuries has brought a great
wealth of novel computational methodologies. Many of these methods yield
themselves to applications beyond their initial computational domain. They can
be readily used to study systems which are quantifiable. At the same time, a
risk of unlimited application of computationalism to subjects that are not
readily quantifiable emerges. Indeed, it might be argued that some systems may
not be quantifiable without a significant loss, and perhaps more dangerously
whose quantification might yield an illusion of understanding. Economics,
however, is a field that has always dealt with quantities. More so its main
subject might even be said to arrive already quantified111 _‘Every non-barter
commodity exchange, or non-financial transaction, catalyzes a flow, by pricing
it. The ‘real economy’ is thus automatically captured, by a monetary code,
even when – as is overwhelmingly typical – this quantification through signs
is never aggregated. Since money systems install an intrinsic economic
intelligence (preceding all reflective theorization), economics – uniquely
among the sciences – inherits a field of objectivity whose arithmetization has
already taken place.’_ (Land,, 2018). While economics might not limit itself
to the realm of quantities (by also studying the social aspects) its focus is
on studying a system whose inherent feature is that of quantification.. There
are many names that have been given to this subject but we will refer to it as
Capital, and it is a large, spreading, and complex ‘beast’.

While the word Capital brings an intuitive association with the economic
processes its precise definition is elusive at best. Capital appears to have
many facets and according to some theories can itself be broken down into
subcategories. The challenges of defining the term, which appears at the core
of any economic endeavor, have by no means been lost on its researchers (Hayek
and White,, 2019). In classical economics Capital is divided into fixed and
circulating capital. The former constituting the goods that are not consumed
in the production process and the latter being those goods that are consumed
in production (Adam,, 2016). According to the neoclassical synthesis, that
combines the Keynesian economics with the neoclassical economics, Capital
refers to those durable produced goods that have potential to be used for
further production (Samuelson and Nordhaus,, 2012). The Austrian School, on
the other hand, as exemplified by the work of F.A. Hayek, would assign the
name Capital to ‘the total stock of the non-permanent factors of production’
(Hayek and White,, 2019). According to Marx, Capital can be viewed as always
seeking to create value and surplus-value. As this surplus-value is created
via labour, Capital takes a more abstract form than the previously mentioned
definitions222 _‘Capital is dead labour, that, vampire-like, only lives by
sucking living labour, and lives the more, the more labour it sucks.’_ and
_‘The circulation of money as capital is, on the contrary, an end in itself,
for the expansion of value takes place only within this constantly renewed
movement. The circulation of capital has therefore no limits’(Marx,, 2004)_
present the Marxist sentiment towards Capital. Note the highly anthropomorphic
language and an implicit, albeit perhaps metaphorical, endowing of Capital
with agency.. Finally, more provocative definitions of Capital exist and are
exemplified by, among others, Nick Land, who sees Capital as limitless
dehumanizing consumption of life and intelligence that creates new forms of
these and is essentially an auto-promoting feedback loop. Land’s perspective
would often seemingly equate Capital with life333 _’As capital ’evolves’, the
increasingly absurd rationalization of production-for-profit peels away like a
cheap veneer from the positive-feedback detonation of production-for-
production’_ (Land,, 1988) and _‘Life continues, and capitalism does life in a
way it has never been done before’_ (Land,, 2011) are some of Land’s positions
on the topic, which take the organic analogies of Marx even further.. Further,
he characterises Capital as an end in itself rather than as means to an end.

A seemingly common property of most pertinent definitions is that Capital
represents value. In particular Capital is that which has a potential to
produce more Capital444The autopoietic undertones should not be missed here
(Maturana and Varela,, 1991; Luhmann,, 2012).. Thus, Capital could be summed
up as that value which might produce more value. In this work we refer to
Capital with uppercase ‘C’ to differentiate it from the many definitions of
capital such as intellectual capital or natural capital known in modern
economics. On the other hand, in referring to the features of our model we
will use a lowercase ‘c’. At the end of this section we will attempt a
synthesizing of the above definitions taken from traditions between which
considerable tensions have accumulated.

As an abstract value that breeds abstract value, Capital and its medium of
quantification have grown within, or maybe over, the social landscape. Whether
it serves humanity or subjugates it has not been agreed on and both
perspectives have been argued for many times. Nevertheless, in manufacturing
pure value Capital is driven by a vast number of humans, their technique and
labour. As such it has been interpreted as an expression of a spontaneous
order that emerges from all the aims of individuals that take part in it
(Hayek,, 2011). Instead of adopting this perspective uncritically we will
modify it while accepting that Capital is indeed best understood as a complex
system.

While Capital, as arising from the social, is at its core the emergent of a
complex system, it is not directly clear that it embodies the perfect
aggregation of preferences of all individuals that take part in it. As it is,
any complex system is essentially more than the sum of the constituting parts
(Heylighen et al.,, 2006). This manifests in practice in the growing
frustration of realising that as individuals we have limited control over
Capital. Naturally, the fact that a single individual is not able to influence
the entire system does not imply that the system does not take into account
that individual’s aims. However, within Capital we seem to observe situations,
where a vast majority of individuals seem to agree on certain abstract goals
and yet these goals do not appear to be readily reflected in the emergent
goals of Capital. The environmental crisis or the persistent continuation of
the military industrial complex exemplify this property. The majority of
people seem to agree that the environment should be protected and that wars
should not be fought and yet Capital continues to grow by ravaging nature and
dealing in death. And perhaps that is not at all surprising as the
uncontrollable character of Capital arises from its identity as an emergent.
Indeed Capital can also grow through investments that improve nature, and
trade — and thus capital growth — flourishes more in peacetime than wartime.
Thus, something akin to a will of its own arises in Capital that can be
neither ruled nor explained in terms of its constituents555It is this
independence, among other things, that breeds the curious form of resentment,
disengagement and alienation characteristic of our modern days. Within Capital
it is not only the individual who is disempowered but also aggregations of
individuals..

The complexity perspective, arising from cybernetics, offers a valuable view
on Capital, which through its lens can be understood as a positive
reinforcement feedback loop (Mirowski,, 2002). As such Capital is that, which
in its continuation constantly yields more of itself. This agrees with the
traditional definitions that would see Capital as the value that has a
potential to produce more of itself. Indeed, many economic processes have
already been identified as being highly self-reinforcing Arthur, (2018). In
understanding Capital as a system we can extend the function of its feedback
character to include the curious property, analogous to autopoeisis, that
allows it to produce and maintain its own structure666 _”Capitalism” […],
designates the sovereign self-escalation of an innovative entity, defined only
by the practical relation of auto-promotion it establishes with – and through
– itself. What it is, in itself, is more than itself. Growth is its essence.’_
(Land,, 2018). This self-productive maintenance, however, might also, at
times, fail leading to a degeneration of the system in the form of an economic
crisis. The Capital then could be seen as undergoing phases akin to growing
and declining or even living and dying. An aversion towards letting the
Capital fail has remained a popular sentiment in many circles (Mirowski,,
2014) but it could be argued that letting the system ‘die’ could allow it to
learn and improve (Korecki et al.,, 2023). Moreover, these common analogies of
living and dying further support the potential of an organic perspective on
Capital.

Historically, the emergence of surplus value might be associated with the
beginning of civilisation (10000BC), where a surplus of goods would be created
by applying a novel agrarian technique777We use technique in the sense
proposed by Jacques Ellul. _‘Technique is the totality of methods, rationally
arrived at and having absolute efficiency (for a given stage of development)
in every field of human activity’_ (Ellul,, 2021).. The pre-agrarian hunter-
gatherer societies did not have a concept of wage labour because they did not
have currency. Labour might have been rewarded with concrete services or goods
and commodities may have been exchanged directly. Nevertheless, the lack of a
universal medium of commensurabilisation made it impossible to either
accumulate or consistently quantify value. But through currency the surplus
goods could be converted and stored. Assuming the price of producing the goods
was lower than the selling price, surplus value would be created. As such the
historical emergence of Capital appears intrinsically linked both to the
surplus goods being created and the advent of currency as an accepted medium
of exchange.

It appears, though, that Capital has changed significantly since those bygone
days of its infancy. As time passed, it has become more and more abstracted
away from the direct realities of human experience (financial instruments such
as derivatives or complex insurance plans serve as examples of the growing
abstraction). It has also expanded greatly to include ever more things in its
network of commensurability. The leaps in ‘territory’ that Capital covers
appear correlated with the paradigm shifts in technique888In the later parts
of this work we argue for an analogy between Capital and autonomous technique
which might perhaps add to explaining this apparent historical correlation.,
such as the already mentioned agrarian revolution or more recently the
industrial revolution (Braudel,, 1992).

The definition of Capital that we will use in what follows does not aim to add
yet another formulation to the already crowded landscape. We posit a minimal
and functional definition that encapsulates the characteristics of Capital as
identified above, taking into account its historical context and the way it is
experienced by individuals.

######  Definition 1.

Capital is a historical agential system that arises and continues embedded in
the complexity of a social sphere. As a system it is embodied in quantified
value and that which pursues its maximization.

## Capital as a Historical Agential System

To construct our model of capital we inherit useful definitions from Continual
Reinforcement Learning (Abel et al.,, 2024). Capital then extends the
definitions to suit the needs of Capital: a system which at its core extends
as it is extended.

We start with an agent-environment interface (Abel et al.,, 2024) defined by
the tuple $(\mathcal{A},\mathcal{O})$ of actions and observations. The set of
all possible histories given $(\mathcal{A},\mathcal{O})$ is defined as:

$\mathcal{H}=\bigcup_{t=0}^{\infty}(\mathcal{A}\times\mathcal{O})^{t}$

A single history $h$ is a sequence of observation action tuples $(o,a)$, and
is an element of $\mathcal{H}$. The environment is defined as a function which
maps histories and actions to a probability distribution over observations:
$e:\mathcal{H}\times\mathcal{A}\rightarrow\Delta(\mathcal{O})$. We define a
reward function which maps an observation and an action to a real number:
$r:\mathcal{O}\times\mathcal{A}\rightarrow\mathbb{R}$. We denote agents by
$\lambda$, and agents map histories to a probability distribution over
actions: $\lambda:\mathcal{H}\rightarrow\Delta(\mathcal{A})$.

The realizable histories (Abel et al.,, 2024) of a given agent-environment
pair $(\lambda,e)$, define the set of histories of any length that can occur
with non-zero probability from the interaction of $\lambda$ and $e$,

$\mathcal{H}^{\lambda,e}=\bigcup_{t=0}^{\infty}\bigg{\\{}h_{t}\in\mathbf{H}_{t}\>:\>\prod_{k=0}^{t-1}e(o_{k+1}|h_{k},a_{k})\lambda(a_{k}|h_{k})>0\bigg{\\}}.$

The realizable history suffixes (Abel et al.,, 2024) of a given $(\lambda,e)$,
relative to a history prefix $h\in\mathcal{H}^{\lambda,e}$, define the set of
histories that, when concatenated with prefix $h$, remain realizable,

$\mathcal{H}^{\lambda,e}_{h}=\bar{\mathcal{H}}_{h}=\\{h^{\prime}\in\mathcal{H}\>:\>hh^{\prime}\in\mathcal{H}^{\lambda,e}\\}.$

######  Proposition 1.

Capital is a historical agential system.

We take the perspective of Capital as a historical process in that it is
instantiated in time and shaped by its sequential history. We characterize its
history as the past observations and the actions that were taken at those
observations. In this manner, a minimal instance of Capital is: an agent
$\lambda$ and an environment $e$ which together establish the set of
realizable histories $\mathcal{H}^{\lambda,e}_{h}$, which are analogously the
future possibilities of its becoming. The actions $a$ and observations $o$ are
afforded to the agent by Capital so that the interactions between the agent
and the environment are always mediated by Capital.

The environment represents the entire space in which Capital is embedded (the
Earth, the Universe). Across that environment Capital extends its own medium
through which the observations are accessed. Capital perceives the environment
through its observations, and the two need not be in a one-to-one
relationship.

######  Proposition 2.

Capital is fundamentally discretized (digital) into units of capital.

The medium of Capital is quantified value. In the history of societies this
quantification has come in the form of money and currency. By quantifying the
value of things we embed them in a space where they can be compared999These
things, whose value we quantify, might have subjective or qualitative value to
us but as part of Capital they must exist as quantities.. Thus the universal
medium of commensurability is established, in which Capital operates.

Historically all currencies have quantified value in a countable way.
Intervals on uncountable infinities all contain the same infinity, and thus
are not meaningfully commensurable, they all are the same size infinity. Since
commensurable quantification can only be done on countable sets the units of
quantified value (such as currency) are discrete. The digitalness follows from
the ability to represent all countable sets as 0’s and 1’s.

Therefore, we define a smallest unit of capital, $0<\mbox{\textcent}<<1$, much
like there is a smallest unit of Euro, Dollars and Bitcoin. Every capital
reward $r$ is expressable as a multiple of ¢:
$r=k\mbox{\textcent},k\in\mathrm{N}$. Accumulated capital in time can be
expressed as the sum over all the capital rewards:

$G_{t}=\sum_{i=0}^{t}r_{i}$

######  Proposition 3.

Each unit of capital affords observations and actions.

In its fully discretized form, we can understand the units of capital as
deployed in the environment where they afford observations of the environment
and possible actions to the agents. At every timestep $t$, an observation for
the system is constituted of an observation for each unit of capital,

$\bar{o}=\\{o_{1},o_{2},\dots,o_{m}\\}$

and similarly each unit of capital affords actions from the observed state,

$\bar{a}=\\{a_{1},a_{2},\dots,a_{m}\\}$

where the number of observations and actions in aggregate, $m$, depends on the
accumulated capital $m=kG$.

Units of capital can afford observations and actions to the agents of capital
(e.g. by modifying their observations or available actions), which could take
the form of individuals and groups, like citizens, companies, and nations. The
actions available to the units from their respective observations _may
generate more units_. We develop on the agents of capital in the following
section.

######  Proposition 4.

Actions are time dependent.

It follows from Proposition 5 and Proposition 9 that there may exist actions
from observations at a point in time which did not exist at another point in
time.

$\displaystyle\exists(t,\tau,a)\text{ where }t<\tau\>:$
$\displaystyle\>\forall(o,a^{\prime})\in\mathcal{H}_{t},\>a^{\prime}\neq
a\>\>\cap\>\>\exists(o,a^{\prime})\in\mathcal{H}_{\tau},\>a^{\prime}=a$

######  Proposition 5.

Actions are observation dependent.

Not all actions are possible from different observations. There is an inherent
local sensitivity to the units of capital. Where the units of capital are in
the environment determines the set of actions available to them.

$\exists
o,o^{\prime}\>:\>o\xrightarrow[]{a}\>\cap\>o^{\prime}\not{\xrightarrow[]{a}}$

######  Proposition 6.

Actions can generate new units of capital. A generated unit of capital
observes the next observation.

As a result of taking an action new units of capital may be spawned. Not all
actions lead to new units of capital being generated. This corresponds to the
inherent property of Capital that is seeking to produce more of itself
(surplus value). Given a unit of capital with observation $o$, taking action
$a$, generating units of capital $r$ and observing a next observation
$o^{\prime}$, resulting in the transition tuple $(o,a,r,o^{\prime})$, the
generated units of capital observe the new observation $o^{\prime}$.

######  Proposition 7.

Capital can be partitioned.

Capital may act both independently and collectively. As formulated, each unit
of capital affords observations and actions. Capital can be partitioned into
sets of capital units where capital units in a partition $U$ may be
conditionally dependent. A partition $U$ contains multiple capital units,
$U={u,\dots,x}$ where $u,x\in\mathrm{N}$. The partitioning of capital units
influences the agency that agents of capital have in Capital. Intuitively, a
fine partitioning can reduce the extent of agency, while a coarser
partitioning can increase the extent of agency. The agency of a partition is
the total amount of units of capital in the partition. We refer to the
complete partitioning of Capital as $P$, which is set of all the partitions of
capital.

A specific partitioning of Capital $U$ affords the joint probability
distribution of the units of capital in $U$. The total capital of a partition
is $G(U)=\sum_{u\in U}u$. The partition affords a joint observation space
$\bar{o}_{U}=\\{o\\}_{o\in U}\in\bar{o}$. The history of a partition $U$ is
then defined as an enlarged history $h_{U}$, where each element $h_{t}$ in the
complete history $h_{U}$ is a tuple of the observations and actions afforded
by all the units of capital in $U$: ${(o_{t,x},a_{t,x})}_{x\in U}$.

######  Proposition 8.

The observation space is non-ergodic.

There might be states in the observation space that become unreachable due to
the internal dynamics of the environment or the actions of the units of
capital. This implies the potential non-reversibility of some actions.

$\displaystyle\exists o\in\mathcal{O},\exists
a\in\mathcal{A}\>\text{where}\>o\xrightarrow{a}o^{\prime},h\xrightarrow[]{a,o^{\prime}}h^{\prime}\>:$
$\displaystyle\>\forall
a^{\prime}\in\mathcal{A},(a^{\prime},o)\notin\mathcal{H}^{\lambda,e}_{h^{\prime}}$

######  Proposition 9.

The observation space changes in time.

There may exist different observations at different points in time, which did
not exist at other points in time. This can be due to the internal dynamics of
the environment and the actions of the units of capital. For example, an
exogenous natural catastrophe beyond the control of Capital will alter the
observable environment independently of the actions of the units of capital.
Conversely, the actions of the units of capital can create new things which
themselves become parts of the observations (e.g. a computer allowing higher
dimensional observations of the environment by through computation).

$\displaystyle\exists(t,\tau,o)\text{ where }t<\tau\>:$
$\displaystyle\>\forall(o^{\prime},a)\in\mathcal{H}_{t},\>o^{\prime}\neq
o\>\>\cap\>\>\exists(o^{\prime},a)\in\mathcal{H}_{\tau},\>o^{\prime}=o$

Proposition 8 and Proposition 9 essentially represent the properties of an
open dynamic system, such as the universe, in which Capital is embedded.

## The Agents of Capital

Figure 1: a) The relationship between generating processes and probability
distributions. The generating process determines the probability distribution
which is sampled from to determine the action. b) Generating processes can be
either qualitative or quantitative.

As we have proposed, Capital is composed of units of capital and agents of
capital that determine the probability distributions over actions for units of
capital given their histories. More generally, a capital agent can determine
the probability distribution for a partition of capital $U$. Such a capital
agent can correlate the actions of the capital units such that their
probability distributions are no longer independent.

######  Proposition 10.

The goal of a capital agent is to increase itself by pursuing capital
accumulation as expressed by the amount of units of capital in its partition.

One way to formally define this maximization is through the cumulative sum of
discounted future rewards, for a given discount factor $0\leq\gamma\leq 1$
(following in the reinforcement learning tradition), starting from the current
time $\tau$:

$\pi_{U}=\sum_{t=\tau}^{\infty}\sum_{x\in U}\gamma^{t}r_{x}^{t},$

where $r_{x}^{t}$ is the capital reward obtained by capital unit $x$ at time
$t$. At every time $t$ the goal of a capital agent is to maximize future
capital accumulation. The discount factor $\gamma$ establishes the extent to
which the capital agent over partition $U$ denoted $\lambda_{U}$ is myopic or
far-sighted; the extent to which the agent values short term or long term
rewards.

######  Remark 1.

Maximizing capital is ill-defined for an open system.

The goal of capital maximization can be pursued with optimization techniques,
but there is no guarantee that optimization will reach the maximum in the
system as presented. This is due to the fact that the observation and action
spaces can change in time and that the observation space is non-ergodic.
Nonetheless, a capital agent can still pursue capital accumulation.

######  Remark 2.

The evolution of Capital is modelled as a probability distribution
$\mathbf{Y}$ which is the joint distribution of all the units of capital.
Given a partitioning of capital $P$, each agent $\lambda_{U},U\in P$
establishes the joint distribution $\mathbf{Y}_{U}$ over the capital units
$x\in U$.

Our model propositions define a template which is agnostic to the generating
processes (see Figure 1) of the probability distributions of capital units,
and can then be used to model Capital. A grounding of our model of Capital
adheres to our propositions and additionally inherits the generating processes
of the probability distributions, as in Figure 1(a). Importantly, the
generating processes determine the probability distributions for the agents
which control partitions of capital.

As humans we inhabit a specific grounding of Capital and our direct
experiences reveal broad categorizations of generating processes. Humans are
arguably responsible for modern day Capital, fuelled by the viral desire to
quantify and accumulate it; in so doing humans are agents of capital which
serve and reinforce Capital. They are not, however, the sole agents.
Communities, companies, nations and parallel economic systems are themselves
agents that can control Capital, and they too are agents of capital in so far
as they pursue its accumulation. Specific instances of these groups can be
modelled as partitions of capital. For agents of capital we identify two
general kinds of generating processes: qualitative processes and quantitative
processes (Figure 1(b)). In this split, we implicitly assume there exist
qualities which cannot be quantified, and we also assume that such qualities
are within the realm of experience of human individuals as they possess an own
contingency 101010Such a distinction can be linked to the Lebenswelt or
”Lifeworld” concept popular of the German philosophers of the 20th Century
(Husserl,, 1970): the realm of experience, the state of affairs as
experienced, that which precedes epistemological analysis, and in our
understanding that which precedes quantification.. We posit these methods lie
beyond quantification and are not reducible to it111111An example of a
qualitative generative process could be an ethical value system held by a
given human and used to influence their actions within the system of Capital.
We stand by the argumentation by which quantitative analysis cannot be used to
take normative decisions (Korecki et al.,, 2023)..

On the other hand, quantitative processes are well suited to mathematical
tools. In fact, any pursuit of capital accumulation with quantitative
processes lies within the realm of optimization. We put forward the following
remark:

######  Remark 3.

Any use of Quantitative Processes to generate probability distributions are
subsumed by the methods of optimization.

Our remark stands to indicate that within the realm of processes that operate
on quantities, any process can be modelled using mathematical tools such as
optimization. Later we extend this discussion to understand the manners in
which Capital can be artificially intelligent.

## The Entropy of Capital

Using our model we can reason about the entropy of Capital. The information
entropy of a discrete random variable $X$, with support $\mathcal{X}$, and
distributed according to $p:\mathcal{X}\rightarrow[0,1]$ is,

$H(X):=-\sum_{x\in\mathcal{X}}p(x)\log p(x).$

We take this definition of information entropy and extend it to describe the
entropy of Capital. First, we must clarify that we will define its information
entropy for given timesteps $t$. Then, we identify the random variables of
capital as the discrete distributions over actions given observations induced
by the agents of capital. For each unit of capital $\mbox{\textcent}_{i}$ we
can compute its corresponding entropy, given a history of that unit $h_{i}$,
as the information entropy of a random variable $Y_{i}$ with support
$\mathcal{O}(i)$, and distributed according to
$q:\mathcal{O}(i)\rightarrow[0,1]$,

$q(o_{i})=\sum_{a\in\mathcal{A}(i)}\lambda(a|h_{i})\times e(o_{i}|h_{i},a),$

where $\mathcal{A}(i)$ is the set of actions available to $i$. Thus expressed,
$q$ provides the distribution of next observations with which we can calculate
the entropy of each unit of capital:

$H(Y_{i}):=-\sum_{o\in\mathcal{Y}_{i}}q(o)\log q(o).$

Next, we can express the entropy of Capital as the entropy of the joint
probability distribution:

$\displaystyle H(\mathbf{Y})$ $\displaystyle=H(Y_{1},\dots,Y_{m})$
$\displaystyle=-\sum_{o_{1}\in\mathcal{Y}_{1}}\dots\sum_{o_{m}\in\mathcal{Y}_{m}}q(o_{1},\dots,o_{m})\log(q(o_{1},\dots,o_{m})).$

Establishing conditional dependence between capital units is the main way in
which agents of capital can reduce the entropy of Capital.

The language used by Marx and many others is populated by metaphors that
present Capital as an organism. Nick Land takes it further by seemingly
equating Capital with life. As such Capital is a system that could be of great
interest to the Artificial Life (alife) perspective. Life is often
characterized also by its ability to minimize entropy in a universe whose
entropy is always increasing. Entropy has also been used to analyze capitalism
(Biel,, 2011), though in a reservedly qualitative and discursive manner. By
extending a definition of entropy, whose minimization has been used as an
indicator of what we define as life, to Capital we make it possible to study
the metaphors and links established by the aforementioned authors in a more
formal way.

## The Artificial Intelligence in Capital

In this section we wish to tease out the profound relations that exist between
Capital and Artificial Intelligence (AI). The relation goes both ways. The
first direction is understanding AI as an agent of capital. The second
direction is understanding Capital as AI.

To be clear, by AI we mean deep machine learning models trained via
mathematical optimization. This categorization fits any of the impactful AIs
that we interact with today, from Large Language Models like Chat GPT to
content recommender systems used by Instagram, Youtube and Google. All of
these systems are massive parameter architectures that are fitted through
mathematical optimization tools.

### Artificial Intelligence is an agent of Capital

Through our model we have embedded Capital in a formal language which comes
from the field of reinforcement learning and mathematical optimization found
in control theory and economics. AI as we have it in the year $2024$ is also
built in this language. Thus, our model of Capital takes on a form
‘digestible’ by AI. In concrete terms AI can readily be deployed to interact
with Capital as an agent, given a well defined agent-environment interface and
adherence to our propositions.

Consider an example of a recommender algorithm, which is nothing else than a
massive deep learning model. It is fed the data on the users’ behavior and
preferences (quantified) and outputs content recommendations directly to the
users (Figure 2). The recommendations are optimized such that they maximise
users’ engagement and time spent on a given platform. The data fed into the
model is clearly valuable (because it can be used to produce value), the
output is also clearly valuable (because it drives engagement, which drives
revenue). Here we note that the AI is fed quantified value and outputs more
quantified value, thus, acting, according to our terminology, as an agent of
capital. Additionally, AI processes quantities that are much greater than what
we could process ourselves.

Figure 2: A content platform takes users’ data to produce recommendations, and
in the background uses an AI to process the quantified value, and sell the
attention to an advertiser. The recommender algorithm may be interpreted as an
agent of capital with the data constituting the observation and the
recommednations being the actions.

AI as an active agent of capital may take the form of recommender algorithms
but also algorithmic trading or LLMs and other similarly automated solutions.
This is possible as the generative processes of the probability distributions
of Capital can be purely quantitative in how they specify the pursuit of
accumulation. Thus, AI, being essentially an optimization process (Carissimo
and Korecki,, 2023) fits perfectly within our model of Capital121212Consider a
radical extension of the idea that AI agents can be agents of Capital, where
the Capital is no longer dependent on humanity for its continued functioning.
A fully automated AI driven Capital system is not an impossibility. The
difference between humans and AI as agents of Capital is that humans are
capable of employing both the qualitative and quantitative generative
processes, while AI agents are limited only to the quantitative..

We have argued that the inputs of the AI are quantified value, and in that
sense can be understood as units of Capital. Then, by their very nature, AI
can be designed to produce the ‘best possible’ processing of the quantified
value through quantitative processes. In so doing they pursue Capital
accumulation. AI’s are therefore the best possible agents of capital that
humans have been able to create. In creating AI, humans have also been very
good agents of capital.

######  Remark 4.

We can see AI as a specific subset of Capital, which fits into the
quantitative generative processes of the probability distributions of the
units of capital, and as such is an agent of capital. We can see AI as one of
the ways in which Capital has developed to process large quantities of
information to create more value.

### Capital as Artificial Intelligence

Next we will argue that we can learn about Capital by understanding AI, and
that there are ways in which Capital is an AI. As the first parallel we
indicate that, much like AI can process far more quantified value than humans
are capable of, the economy and markets in the economy are also able to
process much more quantified value than any individual economic agent.

Capital can be understood as an optimization, in the very least as an ongoing
attempt of optimization, when the generating processes are quantitative. We
have also established that Capital processes quantified value in amounts much
larger than individual human beings can. Together, these two points afford a
striking resemblance to AI. The main difference remains that the generating
processes of the distributions of capital may be qualitative, which remains
outside of the scope of AI. In light of this difference, we make the claim
that Capital can be artificially intelligent in proportion to the generating
processes of the probability distributions of capital which are quantitative.

######  Remark 5.

In so far as the generating processes of the probability distributions of
capital are optimization, Capital can be understood as artificially
intelligent in its ability to: process much larger quantities of value than
its constituents, and pursue goal directed capital accumulation which produces
new information in the form of quantified value.

## A Pragmatic Perspective

Thus, we have claimed that Capital can be artificially intelligent, and in so
far as it is artificially intelligent this has implications for the manners in
which we should interpret the value produced by Capital. To do so we must
explore "meaning", where by meaning we denote the interpretation of words or
information as done by the receiver of the information.

When a Large Language Model (LLM), exemplifying ML and trained in the process
of optimization, produces a response to an input prompt we might interpret it
as meaningful. We may know that ultimately the LLM is not trying to make
sense, it is just outputting a probability distribution it has encoded in its
weights, and yet we interpret the things that it says. We can find meaning in
the text which an LLM produces knowing very well that there is no intention
behind it. When optimizations produce outputs that we could not have produced
ourselves, and that we find meaningful, we tend to attribute qualities of
intelligence to them, colloquially referring to them as AI. However, AIs only
have a contingency that can produce meaning by reflection of the contingencies
of the human agents which produce its training data Esposito, (2017), because
optimizations do not possess a normative dimension required for intent
(Korecki et al.,, 2024). If the output of an AI has meaning it is a reflection
of its meaningful inputs.

Then, by composition with our argument that Capital can be artificially
intelligent, there is an extent to which Capital driven by quantified
optimization processes is as devoid of intent as the output of a Large
Language Model. As for AI, Capital driven by quantitative optimization
processes can at best reflect the meaning of the inputs as generated by beings
with their own contingencies like humans.

######  Remark 6.

In so far as the generating processes of the probability distributions of
capital are optimization, we cannot attribute intent to the meaning that we
interpret in the quantified value that Capital produces.

On the other hand, in interpreting the information of Capital, what is its
meaning? Let us look at prices, one of the main quantities produced by
Capital. What meaning can we sensibly ascribe to prices? To an economist,
prices reflect the preferences of market participants. Even if the
relationship between prices is bi-directional (Figure 3), if a market
perfectly reflects preferences, in that the preferences of all capital agents
are considered, this produces a unique equilibrium. If this is true then the
meaning of prices could be the reflection of the preferences of capital
agents, an act of ‘artificial communication’ Esposito, (2017) achieved through
large scale mechanistic apparatus of Capital.

However, consider financial crises, or bubbles, where the sudden change in the
value of an object of capital can alter the preferences of people. Such events
stretch the fabric of Capital revealing a reality of feedback loops in which
it is no possible to disentangle the preferences of capital agents from the
price information produced by Capital, and a reality which equilibrium
concepts do not describe. The preferences of capital agents as revealed
through prices in a market may not be meaningful reflections of static
preferences, but rather dynamic preferences which are themselves influenced by
the pursuit of Capital accumulation. We cannot discover the ‘true’ preferences
of other capital agents through prices, because the preferences are themselves
influenced by prices (Figure 3). We believe this to be a feature which is
inherent to the quantification of value, its subsequent optimization, and its
enmeshment in a social sphere.

We are certainly not the first to propose such systemic perspectives on
complex social systems. In the late 20th Century Niklas Luhmann can be
credited with a most thorough elaboration of a perspective on society as a
system which evolves through functional relationships (Luhmann,, 2012). This
very point was greatly debated by his contemporary Jürgen Habermas who viewed
this functionalisation 131313This term is central to the perspectives of these
philosophers and is used to indicate relationships between objects which can
be formalised, inherently linked to quantification and to the broader debate
of computationalism still strong in modern society. of society as an issue
which could obfuscate those things that could not be cast in functional terms
(Moynahan,, 2018): in our perspective those things which cannot be quantified
and optimized. Thus, rather than a full functionalisation of society, our
model of Capital carries forward this thematic debate by delineating the
functional relationships that exist within the social spheres that are
dominated by value, its quantification and its optimization: the sphere of
Capital.

Viewing Capital with our model has implications for the ways in which we, as
humans, can interact and engage with Capital. We emphasize that if we are
interested in meaning and purpose – qualities characteristic of entities with
their own contingencies – that we cannot rely exclusively on the information
produced by Capital, and must instead look beyond Capital. For example, social
relations, interactions with other living systems like plants and animals,
even interactions with ourselves as composed by trillions of living cells, are
not immediately quantified and allow us to take part in a firstly qualitative
experience of reality where we can seek meaning. It is important to note how
these spheres of social interaction most dear to us are rapidly being
quantified and invaded by Capital accumulation in the early 21st Century, for
example, through the extractive practices which sell user behavioural data on
social networks (dating apps, influencer profiles). This trend has been on the
minds of philosophers for centuries.

Figure 3: Top: prices are determined by preferences, so the meaning of prices
is aggregate preferences. Bottom: prices are determined by preferences, which
are influenced by prices. As prices influence preferences, the aggregate
preferences revealed by prices are tied to the prices themselves, so the
meaning of prices is void in the absence of prices.

## Conclusion

In this paper, we have defined Capital and proposed a model of it. Our model
uses tools from computer science to formalize the agential and historic
process that Capital embodies. In our model, Capital is described through
capital units and agents of capital that pursue capital accumulation. The
capital units afford agency for the capital agents that control them. We
acknowledge this is a self-referential definition. We claim that it serves the
purpose of studying Capital in a holistic manner. Even though we identify
units and agents of capital, we do not reduce Capital to either of them: it is
modelled as their interaction. Moreover, we extend the definition of
information entropy and derive a measure for the entropy of Capital, which can
be useful for further studying the system we model and the connections with
artificial life.

After establishing our model we identify the parallel between Capital and AI.
AI is an agent of capital which processes quantified value to create more
value. Capital can also be understood as AI when the generative processes of
the probability distributions of capital follow optimization. Finally, we
establish a parallel between the meaning of outputs of AI and the meaning of
prices produced by Capital: as the outputs of AI are not intentionally
meaningful, the outputs of Capital (e.g. prices) can be interpreted as
meaningful without reflecting inherent meaning 141414You can find the link. to
our Capital LLM, which trained on economic texts, stands to serve as a means
of interacting with Capital and understanding the parallels between Capital
and AI.

As a further direction, we envision the work on validating the proposed model
through the analysis of historical precedents such as economic crises, large-
scale wealth transfers and the global and local distributions of Capital.

## References

  * Abel et al., (2024) Abel, D., Barreto, A., Van Roy, B., Precup, D., van Hasselt, H. P., and Singh, S. (2024).  A definition of continual reinforcement learning.  Advances in Neural Information Processing Systems, 36.
  * Adam, (2016) Adam, S. (2016).  The wealth of nations.  Aegitas.
  * Arthur, (2018) Arthur, W. B. (2018).  Self-reinforcing mechanisms in economics.  In The economy as an evolving complex system, pages 9–31. CRC Press.
  * Biel, (2011) Biel, R. (2011).  The entropy of capitalism.  In The Entropy of Capitalism. Brill.
  * Braudel, (1992) Braudel, F. (1992).  Civilization and capitalism, 15th-18th century, vol. II: The wheels of commerce, volume 2.  Univ of California Press.
  * Carissimo and Korecki, (2023) Carissimo, C. and Korecki, M. (2023).  Limits of optimization.  Minds and Machines, pages 1–21.
  * Ellul, (2021) Ellul, J. (2021).  The technological society.  Vintage.
  * Esposito, (2017) Esposito, E. (2017).  Artificial communication? the production of contingency by algorithms.  Zeitschrift für Soziologie, 46(4):249–265.
  * Hayek, (2011) Hayek, F. A. (2011).  Law, legislation and liberty, volume 1: Rules and order, volume 1.  University of Chicago Press.
  * Hayek and White, (2019) Hayek, F. A. and White, L. H. (2019).  The pure theory of capital.  Routledge.
  * Heylighen et al., (2006) Heylighen, F., Cilliers, P., and Gershenson, C. (2006).  Complexity and philosophy.  arXiv preprint cs/0604072.
  * Husserl, (1970) Husserl, E. (1970).  The crisis of European sciences and transcendental phenomenology: An introduction to phenomenological philosophy.  Northwestern University Press.
  * Korecki et al., (2023) Korecki, M., Carissimo, C., and Lund, T. (2023).  artificial death: learning from stories of failure.  In ALIFE 2023: Ghost in the Machine: Proceedings of the 2023 Artificial Life Conference. MIT Press.
  * Korecki et al., (2024) Korecki, M., Köstner, G., Martinelli, E., and Carissimo, C. (2024).  The man behind the curtain: Appropriating fairness in ai.  Minds and Machines, 34(1):7.
  * Land, (1988) Land, N. (1988).  Kant, capital, and the prohibition of incest: A polemical introduction to the configuration of philosophy and modernity.
  * Land, (2011) Land, N. (2011).  Critique of transcendental miserablism.  Fanged Noumena: Collected Writings 1987–2007, pages 623–627.
  * Land, (2018) Land, N. (2018).  Crypto-current: An introduction to bitcoin and philosophy.  Retrieved August, 22:2020.
  * Luhmann, (2012) Luhmann, N. (2012).  Theory of society, volume 1.  Stanford University Press.
  * Marx, (2004) Marx, K. (2004).  Capital: volume I, volume 1.  Penguin UK.
  * Maturana and Varela, (1991) Maturana, H. R. and Varela, F. J. (1991).  Autopoiesis and cognition: The realization of the living, volume 42.  Springer Science & Business Media.
  * Mirowski, (2002) Mirowski, P. (2002).  Machine dreams: Economics becomes a cyborg science.  Cambridge University Press.
  * Mirowski, (2014) Mirowski, P. (2014).  Never let a serious crisis go to waste: How neoliberalism survived the financial meltdown.  Verso Books.
  * Moynahan, (2018) Moynahan, G. (2018).  The habermas/luhmann controversy and the “cybernetics moment.  Graduate Faculty Philosophy Journal, 39(1):131–166.
  * Samuelson and Nordhaus, (2012) Samuelson, P. A. and Nordhaus, W. D. (2012).  Economics.  AMGH Editora.


# Image-Based Virtual Try-on System With Clothing-Size Adjustment

###### Abstract

The conventional image-based virtual try-on method cannot generate fitting
images that correspond to the clothing size because the system cannot
accurately reflect the body information of a person. In this study, an image-
based virtual try-on system that could adjust the clothing size was proposed.
The size information of the person and clothing were used as the input for the
proposed method to visualize the fitting of various clothing sizes in a
virtual space. First, the distance between the shoulder width and height of
the clothing in the person image is calculated based on the coordinate
information of the key points detected by OpenPose. Then, the system changes
the size of only the clothing area of the segmentation map, whose layout is
estimated using the size of the person measured in the person image based on
the ratio of the person and clothing sizes. If the size of the clothing area
increases during the drawing, the details in the collar and overlapping areas
are corrected to improve visual appearance.

Index Terms—  virtual try-on, OpenPose, segmentation, clothing size

##  1 Introduction

With the increase in online shopping, the online apparel shopping market is
growing rapidly. Online shopping is a convenient method for purchasing items
over the Internet without physically visiting a store. However, when
purchasing clothes online, the customer cannot try out clothes to check if
they fit.

Han et al. [1] first proposed the virtual try-on (VITON) method, which can
generate visualization images for users trying out clothes online. Allowing
users to virtually try on clothes to check for fitting not only improves their
shopping experience and changes the manner of shopping for clothes but also
reduces costs for retailers. Variants [2, 3, 4, 5, 6] of VITON have been
developed for improving virtual visual appearance. To improve the resolution
of the synthetic image on the virtual try-on, Han et al. [7] predicted the
optical flow maps of clothes and desired clothing regions.

Choi et al. proposed a novel high-resolution virtual try-on method called
VITON-HD [8], in which a novel clothing-agnostic person representation
leveraging the pose information and the segmentation map is plotted.
Furthermore, in the alignment-aware segment normalization introduced in VITON-
HD, information irrelevant to the clothing texture in the misaligned regions
is removed and semantic information is propagated throughout the network.
Although VITON-HD can generate photorealistic fitting images, the clothing
size is not considered. Even if clothes of various sizes are used, the same
output is always generated because the system deforms clothes to fit the shape
and design of a person’s body, and the shape of the clothes to be changed is
based on that of the clothes worn in the person’s image.

In this study, an image-based virtual fitting system that can adjust the
clothing size was proposed. Because VITON-HD generates a segmentation map
based on the clothes a person is wearing, a fitting image that differs from
the clothing size is generated. In the proposed method, the cloth area of the
segmentation map is changed according to the ratio of the actual person and
cloth sizes by using the size information of the person and clothes as the
input. The clothing size is changed according to the size of the person by
calculating the distance between the shoulder width and height of the clothing
in the person image. If only the cloth region is enlarged, then the boundaries
of the cloth region, such as the collar and the overlap with an arm, may
become inconsistent. Therefore, we changed the coordinates of the segmentation
map and corrected the boundary of the cloth region to maintain quality.

##  2 Virtual Try-On System

Given an image of a desired cloth and that of a person, an image-based virtual
fitting system generates an image of the person wearing this cloth. In this
section, we briefly review image-based virtual fitting methods and explain
drawbacks.

###  2.1 VITON-HD

VITON-HD [1, 8] consists of four independent modules, namely pre-processing,
segmentation generation, clothes deformation, and try-on synthesis.

In this model, pose estimation and parts estimation of a person wearing the
target clothes are used for virtual try-on image synthesis. First, the
segmentation map and pose map are used to pre-process an input image as a
clothing-agnostic person image and its corresponding segmentation.
Furthermore, semantic segmentation is used in VITON-HD to separate appearance
and shape generation, which generates spatial and natural results. Thus, a
clothing-independent representation of the person can be achieved, information
about the cloth worn before fitting can be removed, and body information for
reconstruction can be obtained. In skeletal estimation, parts of the body that
are difficult to reconstruct (e.g., arms and hands) are removed.

The semantic segmentation generation module (SGM) generates masks for exposed
body parts (i.e., composite body part masks) and warped clothing regions using
semantic segmentation of body parts and clothing. The SGM has a UNet [9, 10]
structure consisting of a convolutional layer, a downsampling layer, and an
upsampling layer. Furthermore, two multiscale discriminators [11] are used for
conditional adversarial loss. The SGM generates semantic masks in a two-step
process—the body part is first generated and then the clothing mask is
synthesized, which renders the original clothing shape in the person image
completely network agnostic.

In clothes deformation, input clothes are deformed by a geometric
transformation to fit a given person. A thin plate spline (TPS) transformation
[12] is then used to deform the clothing image, which is an extension of the
spline curve to a two-dimensional plane. These parameters were used to warp
the clothing image to fit the estimated layout.

The UNet-based network is used to generate virtual-fitting images using
intermediate products (target layout, area of compositing process, and warped
clothing images) obtained up to this key point as the input. The try-on
synthesis module explicitly determines areas that cannot be complemented by
the cloth warping and inputs them to the generation module to reduce image
quality degradation resulting from misalignment.

###  2.2 Problem

Conventional image-based virtual fitting systems depend on the clothing size a
person is originally wearing. In VITON-HD, a segmentation map is generated
from an image of a person by using face, arms, legs, and clothes worn by the
person as feature points. The segmentation map is then used as an input for
geometric transformation processing, which generates a virtual try-on image
according to the size of the clothes worn by the person before try-on.

If the clothing region indicated by a segmentation map is extended, then
nonclothing areas are affected and the clothing area is enlarged. However, in
this module, appearance cannot be changed to a specified size.

##  3 Proposed Method

In the proposed method, the clothing size is changed to match the actual size
by adding real-space size information and using a correction process to VITON-
HD segmentation.

###  3.1 Overview

To change the clothes size according to the person, the actual sizes of the
person and clothes were added as inputs to the correction process. Size
information is defined as the shoulder width in the horizontal direction and
the height of clothes in the vertical direction. Virtual space information is
calculated based on the coordinates of the key points detected using OpenPose
[13, 14] and the distance between the shoulder width and height of the person
image. Figure 2 displays these details.

Only the clothing region of the segmentation map is resized, the layout of
which is estimated using the size of the person measured in the person image
based on the ratio of the actual person size to the clothing size. By changing
the clothing size ratio to match the person in the person image based on the
size ratio between the actual person and the clothing, the clothing size in
the person image is changed to the clothing size corresponding to the person.

Changing the clothing size results in the coordinates of the segmentation map
to differ from those of VITON-HD, which causes inconsistencies at the
boundaries. Therefore, a correction technique was applied to the boundary of
the clothing region to maintain the same quality as that of VITON-HD.

###  3.2 Ratio to clothes Size

To reflect the clothing size in the person image, the actual clothing and
person size information were first added as inputs. Size information is
obtained from the shoulder width and height of the clothes in the horizontal
and vertical directions, respectively. In OpenPose, the clothing size can be
applied to the person image according to the input size information.

Fig. 1: Some key points detected by OpenPose.

Fig. 2: Irregularities at the collar region and overlap with the clothing
area.

####  3.2.1 Adjustment of the clothing size

To reflect the clothing size in the person image, the clothing and person size
information are first added as inputs. Size information is calculated from the
shoulder width and height of the clothes in the horizontal and vertical
directions, respectively.

The vertical size of the clothing in a person image is obtained by detecting
25 key points by using OpenPose and calculating the ratio of the coordinate
distance between key points 1 and 8, the size of the actual clothes, and the
height of the person. Suppose that the height of a person’s clothing region is
$H_{p}$, the height of the clothing is $H_{c}$, and the coordinate of key
point $t$ is $(x_{t},y_{t})$. The clothing’s height $\tilde{H}$ in a virtual
space can be obtained using the following equation:

$\tilde{H}=\frac{H_{c}}{H_{p}}\times\delta_{1,8},$ (1)

where $\delta_{i,j}$ is the Euclidean distance between key points $i$ and $j$
is defined as follows:

$\delta_{i,j}=\sqrt{(x_{i}-x_{j})^{2}+(y_{i}-y_{j})^{2}}.$ (2)

Similarly, the horizontal size $\tilde{W}$ of the clothing in a virtual space
can be calculated using the following equation:

$\tilde{W}=\frac{W_{c}}{W_{p}}\times\delta_{2,5},$ (3)

where $W_{c}$ and $W_{p}$ are the shoulder widths of the clothing and person,
respectively.

The lateral size of the clothing was calculated from the distance between key
points 2 and 5 in the figure, the distance between key points 3 and 2, and the
distance between key points 5 and 6. In this case, (sleeve length + shoulder
width) $\alpha$ in the person’s image can be calculated using the following
equation:

$\alpha=\delta_{2,5}+\delta_{2,3}+\delta_{5,6}$ (4)

From the segmentation map obtained by layout estimation, only the clothes
regions were obtained by color extraction. Using $\tilde{H}$, $\tilde{W}$, and
$\alpha$, the clothing area was enlarged according to the vertical and
horizontal distances to reflect the size of the clothes to be changed.

###  3.3 Adjustment of Details

As displayed in Fig. 2, when clothes are enlarged, some irregularities may
occur. To suppress such side effects, the following two post-operations were
performed using the proposed method.

####  3.3.1 Collar Region

When the segmentation map was merged by enlarging the clothes region, a gap
appeared around the collar because the size of the segmentation map created by
VITON-HD differed from that of the original shoulder coordinates when the
clothes region was enlarged, and the other segmentation maps were combined.

In the proposed method, an erosion operator [15] was applied in morphological
image processing to the collar area of the clothes region to generate a
natural fitting image. Because clothes shrink if the contraction is applied to
the entire clothes region, limiting the contraction to the collar region is
necessary. Therefore, we consider a rectangle whose center is key point 1 in
Fig. 4, whose horizontal direction $s_{x}$ is proportional to the distance
between key points 2 and 5, and whose vertical direction $s_{y}$ is
proportional to the cloth’s size. In this study, we use $s_{x}=3\tilde{W}/4$
and $s_{y}=3\tilde{H}/4$.

####  3.3.2 Overlapping Arms in Clothes Area

If an arm overlaps the clothing region, as displayed in the red frame in Fig.
2, the positions of the clothes and arm regions become misaligned because the
clothes region is divided and the coordinates of only the clothes region
change when enlarged, whereas the coordinates of the arm region do not.
Therefore, the coordinates of the shortest distance between the clothes and
arm regions were obtained before and after enlargement, and the coordinates of
the clothes region were corrected while maintaining the shortest distance
between the two regions based on the coordinates of the clothes region.

Before enlargement, the distance between the two clothes areas is calculated
for some combinations of the path in the two separated regions, and the
minimum distance $D$ is determined, as displayed in Fig. 4. When the two
clothes regions are enlarged from the center coordinates, the coordinates with
the shortest distance are calculated. The coordinates with the shortest
distance are then obtained by finding all the distances between the two
regions by the combination of the path and obtaining the coordinate with the
minimum value.

Fig. 3: Correction of the collar region.

Fig. 4: Shortest distance between two separated regions.

The difference between the coordinates of the shortest distance before
enlargement and the coordinates of the shortest distance after enlargement is
determined, and the center coordinates of each object is moved by the
difference in the shortest distance. Let $(x_{C},y_{C})$ denote the center
coordinates of regions, and let $(x,y)$ and $(x^{\prime},y^{\prime})$ denote
the coordinates before and after enlargement, respectively. By using the
differences in the horizontal $\Delta_{x}=x^{\prime}_{C}-x_{C}$ and vertical
$\Delta_{y}=y^{\prime}_{C}-y_{C}$ directions before and after enlargement, the
smaller region is shifted to a larger region. By moving the center coordinates
of the two clothing regions, the clothes region can be expanded while
maintaining the distance between the arm regions.

##  4 Evaluation

To evaluate whether the proposed method can perform virtual fitting
corresponding to the sizes of actual clothes, the person’s images, person, and
clothes sizes were measured.

The segmentation map generated in VITON-HD [8] and the proposed method are
displayed in Fig. 5. The segmentation map of the proposed method is enlarged
based on information of the shoulder width and height of the person and
clothing. In the collar area, the clothes area can be enlarged while
maintaining the same height as the person’s shoulder. Even when the clothes
area overlaps with the arm area, the distance between the arm area and the
clothes area can be enlarged without shifting the position of the arm area and
the clothes area.

Fig. 5: Comparison of the generated segmentation map.

The generated fitted images with VITON-HD using the segmentation map created
by the proposed method are presented in Fig. 6. The body and clothes length
information was set to 66 and 73 cm, respectively, and the clothes size varied
considerably. In VITON-HD, the fitting image is generated according to the
size of the clothing before changing, and the size of the clothing to be
changed is not changed, whereas the proposed method changes the clothing area
in the fitting image according to the size of the clothes. Even when an arm
overlapped the clothing area, the proposed method generated a fitting image of
the appropriate size without separating it from the arm.

Fig. 6: Comparison of generated images.

To confirm the effectiveness of the proposed method, a subjective evaluation
experiment was conducted to evaluate whether the proposed method exhibits
superior performance. Four testers (A, B, C, and D) tried on clothes and the
shoulder width and height of the clothes, and ten students in our faculty
responded to the questionnaire. The number of students participating in the
proposed method are summarized in Table 1. To maintain fair evaluation, the
order of VITON-HD and the proposed methods was randomly changed, and the
proposed method was not revealed. The results of the questionnaire revealed
that the proposed method is more natural and appropriate for resizing than
VITION-HD is.

Table 1: Results of the questionnaire from ten users, where the number of users who chose the proposed method is displayed. | tester |
---|---|---
|  A |  B |  C |  D | clothes
size | height | 52 | 45 | 46 | 38 | 62
(cm) | width | 47 | 41 | 43 | 32 | 51
Results | 10 | 8 | 7 | 10 | —

##  5 Conclusions

In this study, an image-based virtual fitting system that corresponds with the
clothing size was proposed, which applies OpenPose in the image segmentation
region. The segmentation map generated by VITON-HD can be resized accurately
by estimating the skeletal structure using OpenPose and adding clothing size,
body length, and shoulder width as inputs. In the future, appropriate size
changes not only for T-shirts but also for various other types of clothing and
the generation of natural-looking virtual fitting images according to the
shooting environment will be studied.

## References

  * [1] X. Han, Z. Wu, Z. Wu, R. Yu, and L. S Davis,  “VITON: An image-based virtual try-on network,”  in Proc. CVPR’18, 2018, vol. 9, pp. 7543–7552.
  * [2] B. Wang, H. Zheng, X. Liang, Y. Chen, L. Lin, and M. Yang,  “Toward characteristic-preserving image-based virtual try-on network,”  in Proc. ECCV’18. 2018, pp. 607–623, Springer International Publishing.
  * [3] H. Dong, X. Liang, X. Shen, B. Wu, B.-C. Chen, and J. Yin,  “FW-GAN: Flow-navigated warping GAN for video virtual try-on,”  in Proc. ICCV’19, 2019, pp. 1161–1170.
  * [4] R. Yu, X. Wang, and X. Xie,  “VTNFP: An image-based virtual try-on network with body and clothing feature preservation,”  in Proc. ICCV’19, 2019, vol. 9, pp. 10511–10520.
  * [5] H. Dong, X. Liang, X. Shen, B. Wang, H. Lai, J. Zhu, Z. Hu, and J. Yin,  “Towards multi-pose guided virtual try-on network,”  in Proc. ICCV’19, 2019, pp. 9026–9035.
  * [6] H. Yang, R. Zhang, X. Guo, W. Liu, W. Zuo, and P. Luo,  “Towards photo-realistic virtual try-on by adaptively generating preserving image content,”  in Proc. CVPR’20, 2020, pp. 7847–7856.
  * [7] X. Han, X. Hu, W. Huang, and M. R Scott,  “Clothflow: A flow-based model for clothed person generation,”  in Proc. ICCV’19, 2019, vol. 9, pp. 10471–10480.
  * [8] S. Choi, S. Park, M. Lee, and J. Choo,  “VITON-HD: High-resolution virtual try-on via misalignmentaware normalization,”  in Proc. CVPR’21, 2021, vol. 9, pp. 14131–14140.
  * [9] O. Ronneberger, P. Fischer, and T. Brox,  “U-net: Convolutional networks for biomedical image segmentation.,”  in Proc. MICCAI’15, 2015, vol. 7, pp. 234–241.
  * [10] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida,  “Spectral normalization for generative adversarial networks.,”  in Proc. ICLR’18, 2018, p. 1–26.
  * [11] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro,  “High-resolution image synthesis and semantic manipulation with conditional GANS,”  in Proc. CVPR’18, 2018, vol. 9, p. 8798–8807.
  * [12] F. L. Bookstein,  “Principal warps: Thin-plate splines and the decomposition of deformations,”  IEEE Trans. Pattern Analalysis Machine Intelligence, vol. 16, no. 6, pp. 567–585, 1989.
  * [13] Z. Cao, T. Simon, S. Wei, and Y. Sheikh,  “Realtime multi-person 2D pose estimation using part affinity field,”  in Proc. CVPR’17, 2017, pp. 7291–7299.
  * [14] Z. Cao, G. Hidalgo, T. Simon, S. Wei, and Y. Sheikh,  “OpenPose: Realtime multi-person 2D pose estimation using part affinity fields,”  IEEE Trans. Pattern Analysis Machine Intelligence, vol. 43, no. 1, pp. 172–186, 2021.
  * [15] P. Soille,  Morphological Image Analysis: Principles and Applications,  Springer Berlin Heidelberg, 2013.