# Come Closer: The Effects of Robot Personality on Human Proxemics Behaviours

Meriam Moujahid1, David A. Robb, Christian Dondrup and Helen Hastie For the
purpose of open access, the author has applied a Creative Commons Attribution
(CC BY) license to any Author Accepted Manuscript version arising.1School of
Mathematical and Computer Sciences, Heriot-Watt University, EH14 4AS, UK.
{M.Moujahid, d.a.robb, c.dondrup<EMAIL_ADDRESS>
###### Abstract

Social Robots in human environments need to be able to reason about their
physical surroundings while interacting with people. Furthermore, human
proxemics behaviours around robots can indicate how people perceive the robots
and can inform robot personality and interaction design. Here, we introduce
Charlie, a situated robot receptionist that can interact with people using
verbal and non-verbal communication in a dynamic environment, where users
might enter or leave the scene at any time. The robot receptionist is
stationary and cannot navigate. Therefore, people have full control over their
personal space as they are the ones approaching the robot. We investigated the
influence of different apparent robot personalities on the proxemics
behaviours of the humans. The results indicate that different types of robot
personalities, specifically introversion and extroversion, can influence human
proxemics behaviours. Participants maintained shorter distances with the
introvert robot receptionist, compared to the extrovert robot. Interestingly,
we observed that human-robot proxemics were not the same as typical human-
human interpersonal distances, as defined in the literature. We therefore
propose new proxemics zones for human-robot interaction.

##  I INTRODUCTION

Social robots are entering our public and social spaces, where they need to be
human-aware and follow social etiquette. Social robots can be valuable in the
service industry, such as receptionists [1], baristas [2], and bartenders [3].
If we observe how people interact with each other in physically situated face-
to-face settings, the interaction goes well beyond the spoken words. Hence,
the physical space in which the interaction takes place can have implications
for the design of effective human-robot interaction. For such an effective
interaction with humans, robots need to perceive and reason deeply about their
physical surroundings, understand the physics of human interaction and engage
in fluid interaction with humans in their physical environment [4]. One of the
commonly used measures to understand human social behaviours in a physical
space is distance and positioning between interaction partners, often referred
to as _proxemics_.

Initial work on the concept of proxemics [5] provided a systemic basis for
research into social and personal spaces between humans. It was demonstrated
that social spaces substantially reflect and influence social relationships
and the attitudes of people towards each other. It is not unreasonable to
assume that such proxemics behaviours might also hold true in human-robot
interaction (HRI). Furthermore, it is essential to understand how design
decisions for social robots could influence human proxemics behaviours around
these robots. To this end, this work looks at evaluating the influence of
different robot personalities, i.e. introvert vs extrovert, on proxemics using
a fixed position robot. Furthermore, we compare these proxemics to human-human
interpersonal distances.

According to E. Howarth [6], it is preferred to have extrovert traits when
working in service industry jobs, this is valid for the receptionist role.
Previous work [7] demonstrated that manipulating linguistic and prosodic cues
of a social robot can be used to portray extroversion and introversion
personalities traits for the robot. Furthermore, an extrovert robot has been
shown to be preferred and trusted over an introvert one [8]. However, these
studies did not delve into the proxemics aspect.

Our contributions in this paper include a unique study “in the wild” showing
that human-robot proxemics are influenced by a robot’s displayed personality.
We use a Furhat111https://furhatrobotics.com robot receptionist, connected to
a visitor management system and a TDS
kiosk222https://www.timedatasecurity.com/product/tds-visitor, which is a self-
service visitor check-in kiosk, as shown in Figure1. The robot is able to
reason about its physical surroundings and interact using a combination of
verbal and non-verbal communication [9]. We address two broad research
questions (RQ):

RQ1: Can the robot’s personality influence human proxemics behaviours?

RQ2: Are Human-robot interpersonal distances different to those found for
human-human interpersonal distances?

Figure 1: Furhat Robot Receptionist connected to a visitor management system
and a TDS kiosk for visitors’ check-in and check-out.

##  II BACKGROUND

Edward T. Hall [10] introduced the theory of proxemics, which refers to the
personal space that people maintain around themselves. This space differs
depending on the social setting and their cultural backgrounds. Michael Argyle
[11] suggested the intimacy equilibrium model, which links mutual gaze and
proxemics behaviours [12]. This model illustrates how people react to the
violation of their personal space by reducing mutual gaze and/or moving
backward.

A person’s space is not only a physical buffer zone, but also a psychological
one [13]. The invasion of our personal space can be uncomfortable and
disquieting. Such a breach occurs when our intimate zone (0.00m to 0.45m) is
violated by someone who is not an intimate connection. The ideal zone reserved
for personal interactions among good friends or family is between 0.45 to 1.2,
while the social zone reserved for interactions among acquaintances is between
1.2m and 2.1m [10]. These guidelines used in social psychology are based on
human-human interaction, and can only map to human-robot interaction if we
concede that a robot is perceived as a social actor and comparable to a human
[14] [15]. However, people do not always interact with robots in the same way
that they interact with each other [16] [17]. Previous research suggests that
humans approach robots with distances reserved for a close acquaintance or a
family member [18], which is considered within the intimate zone.

In human-robot interactions, there are many factors that could influence human
proxemics behaviours. Some factors are related to the robots such as its voice
[18], appearance [19], speed [20], and height [21]. Other aspects are related
to humans such as their age [22], personalities [23] [24], prior experience
with a robot [21], gender [25] or social norms [11]. Thus, it is critical to
comprehend which robot design decisions impact human proxemics behaviours
around robots, in order to enhance interactions between robots and humans.

It is yet unclear if robot personality can have any effects on human proxemics
behaviours, given that it has been shown that we can model certain personality
traits [8]. In this study, we test if proxemics vary depending on the robot
personality type. Previous research on robot personality aims attention at the
facet of extroversion, for being one of the easier personality traits to
exhibit, even in shorter interactions [26]. On that account, we deemed it
feasible to focus on extroversion and introversion as types of robot
personalities.

Commonly, introverts are monotone and use fewer pauses and hesitations [27],
while extroverts use extensive vocabulary and speak with a faster speech rate
and louder voice, in addition to higher fundamental frequency and broader
frequency range [28], [29]. Earlier work [30] [31] demonstrates how
calibrating prosody can portray extroversion and introversion as personality
traits in artificial agents, and can influence how we perceive their
personality.

Other previous research [32] explored the influence of voice pitch on how
people perceived a social robot receptionist. The findings point out that a
high-pitched robot with a female voice was rated more attractive, emotional,
and extrovert.

Similarly, the voice of the robot can influence approaching distances [18].
Trovato et al. [33] demonstrate that any uncomfortable noise produced by a
robot can lead to greater human-robot distance when people approach it.
However, this proxemics effect can be masked by adding ambient background
music.

According to Bhagya et al. [34], proxemics distances preferred by humans
increase with the increased volume of the internal noises (i.e.: machine
noise) of a robot. However, this was not tested with a robot that uses
advanced speech capabilities and natural language. Macmillan [35] and
Chepesiuk [36] demonstrate that the ideal sound of a normal conversation is
about 60dB, while any sound greater than 70db will be considered annoying
noise for human ears, which might result in changing proxemics behaviours.
During this experiment, we therefore kept the volume of the robot speaker
between 58dB and 64dB, measured using a sound meter.

##  III METHODOLOGY

As mentioned above, Lim et al. [8] showed that one can successfully represent
extroversion and introversion using linguistic features and vocal cues, they
then went on to show that an extrovert robot is preferred and trusted more
than an introvert robot in a robot barista setting. We followed a similar
approach to Lim et al. [8], using prosodic parameters: volume, pitch, and
tempo. We created two robot personalities with different traits of
extroversion and introversion. However, as the context and domain of a robot
receptionist was different to the robot barista, in their study, we relied on
other previous work [29] to add pragmatic [7] and lexical/syntactic [37]
differences between the two personalities.

We explore how different types of robot personalities, specifically
introversion and extroversion, can lead to different human proxemics
behaviours. Furthermore, in the wild studies can give us new insights into
spontaneous interactions with robots, giving people the freedom to approach
the robot (or not) at a comfortable distance. This work addresses these
questions and fills this gap by comparing how an introvert vs. extrovert robot
receptionist will affect proxemics behaviours.

##  IV EXPERIMENTAL DESIGN AND SETUP

Charlie, the Robot Receptionist set-up, consists of a Furhat robot [38] from
Furhat Robotics333https://furhatrobotics.com that is connected to a visitor
management system through an API, in order to get information about calendar,
meetings, employee profile and to guide visitors with the check-in and check-
out process. The robot is placed in the reception area of the UK National
Robotarium444https://thenationalrobotarium.com, as shown on Figure 2. The
robot can interact with users using verbal and non-verbal communication and
has advanced natural language understanding and dialogue management
capabilities. The receptionist can help visitors with the registration process
in order to print a badge, and give them information about directions, news
and events. The Robot Receptionist is aware of its physical surroundings.
Furthermore, it can reason and react to events in the physical space around
itself.

Figure 2: The reception area in the National Robotarium where the robot
receptionist is placed

###  IV-A Conditions

We created two robot personalities with different traits of extroversion and
introversion, by manipulating the lexicon (See Table I), volume, pitch and
tempo (following [8]) as follows:

  1. 1.

The Introvert: The robot speech was given a lower pitch (-20%), volume (-6dB)
and speaking rate (-20%).

  2. 2.

The Extrovert: The robot speech was given a higher pitch (+20%), volume (+6dB)
and speaking rate (+20%).

TABLE I:  Extrovert and Introvert statements and linguistic features applied
to the robot interaction

Introvert Robot Receptionist |  Extrovert Robot Receptionist |  Linguistic feature
---|---|---
Hello! Welcome to the National Robotarium! I am Charlie the receptionist! I will be happy to assist you while entering your details. |  Hi! Welcome to the National Robotarium! Lovely to see you ! By the way, I am Charlie the receptionist. I will be happy to assist you while entering your details! |  Subject implicitness, positive emotion, self-reference.
Just remember to speak when my green light is on! That’s when I can hear you! |  Actually, It will be great if you can speak when my green light is on! That’s when I can hear you! |  Positive emotion, acknowledgement.
Did you receive an invitation email with a QR code prior to your visit? |  Firstly, I would like to know if you received an invitation email with a QR code prior to your visit? |  Self-reference
Can you please select the option, check in, on the kiosk, then scan the QR code? |  That’s Great! Basically, I just need you to select the option, check in, on the kiosk, then scan the QR code? |  Positive emotion, acknowledgement, self-reference
You will need To register! Please press the first button that says Register! |  No worries at all ! Actually, I just need you to register on the kiosk ! Please press the first button that says register. |  Positive emotion, acknowledgement, self-reference.
Sorry! I didn’t hear you. Would you speak a little louder when my green light is on please? |  Sorry I didn’t catch that ! Is it possible for you to raise your voice, a little bit please, When my green light is on? Just so  I can hear you clearly ! |  informal, self-reference, acknowledgement
Bye for now. |  Have a nice day! Bye! |  Positive emotion

We observed 3 types of interaction with the robot receptionist, therefore we
classified participants into 3 groups:

  1. 1.

Verbal interactions: People used verbal communication to interact with the
robot.

  2. 2.

Non-verbal interactions: People were standing in the robot field of view, and
looking towards the robot for more than 15 seconds but without talking.

  3. 3.

No interaction: People just pass by, without showing any interest. These were
identified by a very short time period (less than 15 seconds) between being
detected in the scene and leaving the scene.

The experiment was between-participants by design. The independent variables
are thus:

\- The Robot Personality as a manipulated independent variable with 2 levels:
Introversion vs Extroversion policy.

\- The Type of Interaction as a non manipulated independent variable with 3
levels: Verbal, Non-verbal or No interaction

The main dependent variable for the experiment is the Shortest Distance
between the robot and the user during the entire interaction.

During the experiment, a total of 120 participants took part (60 interactions
for each personality) over 6 weeks. All interaction data were collected with
the exact user position coordinates and orientation at 5 different time stamps
with $t_{1}$ when the user is first detected and $t_{5}$ when the user leaves
the robot’s scene. We also collected the robot-transcribed utterances, and the
duration of the interaction. The participants’ visual or personal data were
not collected. No audio or video data was collected, also no demographic
information was collected as the experiment was in the wild. The procedure was
ethically approved by our institution’s ethics board.

###  IV-B Hypotheses

In order to address our research questions, we formulated the following
hypotheses:

  * •

H1: There is a significant difference for the shortest distance between the
two robot personality types (Introvert/Extrovert)

  * •

H2: There is significant difference for the shortest distance between the
types of interaction (Verbal, Non-verbal and No Interaction)

  * •

H3: There is a significant interaction effect between robot personality types
and the types of interaction, with respect to shortest distance

##  V RESULTS AND ANALYSIS

Firstly, we grouped the participants into 3 groups based on the Type of
Interaction they had with the robot. Table II represents the number and
percentage of users engaging in each Type of Interaction.

TABLE II: The number of users and types of their interactions with the Robot
Receptionist using different policies

Type of Interactions | With the introvert Robot | With the extrovert Robot
---|---|---
|  Number |  Percentage |  Number |  Percentage
Participants using Verbal Interaction |  22 |  36.66% |  18 |  30.00%
Participants using Non-verbal Interaction |  19 |  31.66% |  20 |  33.33%
No Interaction |  19 |  31.66% |  22 |  36.66%

Figure 3: Extrovert Robot: Scatter plot of the users X and Y coordinates
showing the users’ position on the reception scene in front of the Extrovert
Robot receptionist. (The black dot represents the robot’s position at
(0.0,0.0))

Figure 4: Introvert Robot: Scatter plot of the users X and Y coordinates
showing the users’ position on the reception scene in front of the Introvert
Robot receptionist. (The black dot represents the robot’s position at
(0.0,0.0))

Using the users’ location coordinates, we computed the Euclidean distance for
each logged interaction. We used the Shortest Distance between the participant
and the robot. The Shortest Distance is preferred over average distance as it
is more accurate [39]. Furthermore, the users used the kiosk next to the robot
(Figure1), which will result in them standing at a specific distance from the
robot for an unequal amount of time. This would lead to inconsistent results
when calculating the average distance. Figures 4 and 4 show plots of users
interacting with each personality. Table III shows the descriptive statistics
of the Shortest Distance in each of the two personality conditions across all
three Types of Interaction and over all the interactions as a whole.

TABLE III:  Shortest Distance maintained by participants while interacting
with the Robot Receptionist using different personalities

Type of Interactions | Introvert Robot | Extrovert Robot
---|---|---
|  Min |  Max |  Mean |  Std Dev |  Min |  Max |  Mean |  Std Dev
Participants using Verbal Interaction |  0.41m |  0.91m |  0.61m |  0.12 |  0.55m |  1.47m |  1.19m |  0.46
Participants using Non-verbal Interaction |  0.23m |  1.35m |  0.64m |  0.32 |  0.47m |  2.08m |  1.01m |  0.51
No Interaction |  0.78m |  2.34m |  1.78m |  0.48 |  0.54m |  2.72m |  1.79m |  0.54
All Interactions |  0.23m |  2.34m |  0.99m |  0.63 |  0.47m |  2.72m |  1.36m |  0.60

A factorial ANOVA (also known as two-way ANOVA) was carried out in IBM SPSS
version 28.

Robot Personality: There was a significant main effect of Robot Personality,
with participants’ Shortest Distance being closer to the Introvert Robot
(Mean, 0.99) than the Extrovert Robot (Mean, 1.36), F(1,114) = 16.76, p<.001.
We can therefore accept H1. Furthermore, the ANOVA showed that the Shortest
distance for the Extrovert group was significantly higher than the Introvert
group as shown on Figure 5.

Figure 5: Box plots showing that the users approached the introvert robot
within a significantly shorter distance compared to the extrovert robot.
(boxes show the means and the two middle quartiles, while the whiskers show
maximums and minimums other than any notional outliers)

Type of Interaction: There was a significant main effect of Type of
Interaction on the participants’ Shortest Distance from the Robot, F(2,114) =
61.69, p<.001. We can therefore accept H2. Post hoc tests using Bonferroni’s
correction for multiple comparisons, revealed that the participants’ Shortest
Distance was significantly shorter for those in the Verbal (Mean, 0.88) and
Non-verbal (Mean, 0.84) Interaction groups (p<0.001 in both cases) compared to
those in the No Interaction group (Mean, 1.79). There was no significant
difference in Shortest Distance between participants in the Verbal and Non-
Verbal Interaction groups (p =.902).

Interaction Effect: There was a significant interaction effect between the
Robot Personality and the Type of Interaction measured by Shortest Distance
F(2,114) = 4.57, p =.012. This effect indicates that participants’ Shortest
Distance from the Introvert Robot and the Extrovert Robot was affected
differently by their Type of Interaction. We can therefore accept H3. Figure 6
shows the Means of the Shortest Distance for each group along with their 95%
confidence intervals. We can see this indicates that the Shortest Distance for
those participants in both Verbal and Non-Verbal groups was influenced by
Robot personality (with them being closer to the Introvert Robot). On the
other hand, the Shortest Distance of those who did not interact with the robot
was not affected by robot personality.

Figure 6: Mean Shortest Distances (in metres) for the different groups, along
with the 95% confidence intervals.

##  VI Discussion

RQ1: Can the robot’s personality influence human proxemics behaviours?

In support of H1, the results show that different types of robot
personalities, in this case introversion and extroversion, can lead to
different human proxemics behaviours. Furthermore, the distance was shorter
with the introvert robot, which indicates that participants were more
comfortable standing closer to the introvert robot. Shorter proxemics could be
linked to positive feelings about the robots.

There are wider implications of these results for human-robot interaction
theory and the design of a social robot and specifically for designing its
personality.

RQ2: Are Human-robot interpersonal distances different to those found for
human-human interpersonal distances?

By accepting H2, we demonstrate that the Shortest Distance between human and
robot was different to what would be expected between two humans, based on the
Type of Interactions regardless of the robot personality. We notice that none
of the distance ranges fall within the human-human social zone, which
indicates that people did not follow the same human-human proxemics rules [10]
to interact with the robot, as shown on Figure 7.

Figure 7: Human-human interaction zones according to proxemics theory compared
to interaction zones with the Robot Receptionist

Participants using verbal communication kept a distance between 0.41m and
1.47m. Nonetheless, this distance does not fit with the social zone for human-
human interactions (1.2m to 3.6m). It is however very close to the personal
zone (0.45m to 1.2m) as considered by Hall’s proxemics theory [10].

Similarly, participants using non-verbal communication kept a distance between
0.23m and 2.08m. This distance overlap between the social zone for human-human
interactions (1.2m to 3.6m) and the personal zone (0.45m to 1.2m).

Lastly, participants who avoided interacting with the robot used a zone in the
reception area, which is $\geq$0.78m away from the robot. This is again
considered in human-human interaction as being in a personal and social zone.

These results contradict the media equation theory [14] [40] and Hall’s
proxemics theory [10]. While these theories rely on principles from social
psychology and social science, our results suggest that these theories do not
always map to human robot-interaction. Therefore, in Figure 7c, we propose a
new set of proxemics zones for this form of interaction with a stationary
robot.

Our findings suggest the comfortable distance people like to keep when
approaching the robot might vary between 0.23m and 2.08m (III, second row,
Non-verbal Interaction, Introvert Robot, Min, 0.23m and Extrovert Robot, Max,
2.08m).

The grey coloured zones in Figure 7 b and c, represent the narrow zone very
close to the robot, for which we have no data. The vision system within the
Robot itself is unable to sense and gather accurate data within this zone very
close to it. Thus, it represents what might be termed a blind spot. Thus we
have no data on whether or not people were approaching closer than this. For
this reason, we represent this space in grey as unknown. This close zone is a
space for future research.

As this experiment was done “in the wild”, participants were interacting with
the robot in a spontaneous way. Some participants were using only non-verbal
cues. While the robot was verbally interactive, they chose not to verbally
respond to the robot. This behaviour could be explained by people meeting the
robot for the first time, or being curious about the robot and observing its
behaviour.

This type of behaviour will be unusual in the context of human-human
interaction, again contradicting the media equation theory. However, this
suggests that the non-verbal social communication of the robot could be
improved to also respond using more non-verbal cues in these types of
situations.

##  VII CONCLUSION AND FUTURE WORK

We show that the distance between robot and human can be different depending
on the robot personality. To demonstrate this, we focused on introversion and
extroversion as personality traits. We provide empirical evidence that people
who had a verbal interaction with the robot maintained a shorter distance with
an Introvert robot compared to the Extrovert version.

Overall, the distance people maintained with the robot, regardless of the
robot personality, was between 0.41m and 1.47m for verbal interaction, and
between 0.23m to 2.08m for non-verbal interaction. This range is overlapping
between human-human social zone and personal zone. These results imply that
proxemics theory does not map directly to human-robot interactions as people
do not interact with robots in the same way they interact with each other.

There are broader implications of these results in the design of social robots
and interactions. For instance, factors related to the robot personality
traits can influence human proxemics behaviours around these robots.

In the future, further data will be collected to analyse other parameters that
can influence proxemics behaviours, such as the robot’s physical aspects and
gestures. Moreover, we want to analyse further variables such as the duration
of the interaction and look into the human trajectories in more detail, and
how these proxemics behaviours change over time.

## Acknowledgements

This work was funded and supported by the UKRI Node on Trust (EP/V026682/1)
https://trust.tas.ac.uk. We would like to thank the National Robotarium
Engineers https://thenationalrobotarium.com and the TDS support team
https://www.timedatasecurity.com/product/tds-visitor.

## References

  * [1] M. Moujahid, B. Wilson, H. Hastie, and O. Lemon, “Demonstration of a robot receptionist with multi-party situated interaction,” in _Proceedings of the 2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_. IEEE, 2022, pp. 1202–1203.
  * [2] M. Y. Lim, J. D. A. Lopes, D. A. Robb, B. W. Wilson, M. Moujahid, and H. Hastie, “Demonstration of a robo-barista for in the wild interactions,” in _Proceedings of the 2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_. IEEE, 2022, pp. 1200–1201.
  * [3] S. Keizer, M. Ellen Foster, Z. Wang, and O. Lemon, “Machine learning for social multiparty human–robot interaction,” _Proceedings of the ACM transactions on interactive intelligent systems (TIIS)_ , vol. 4, no. 3, pp. 1–32, 2014.
  * [4] S. Andrist and D. Bohus, “Accelerating the development of multimodal, integrative-ai systems with platform for situated intelligence,” _arXiv preprint arXiv:2010.06084_ , 2020.
  * [5] E. T. Hall and T. Hall, _The silent language_. Anchor books, 1959, vol. 948.
  * [6] E. Howarth, “Expectations concerning occupations in relation to extraversion-introversion,” _Psychological Reports_ , vol. 24, no. 2, pp. 415–418, 1969, pMID: 5809034. [Online]. Available: https://doi.org/10.2466/pr0.1969.24.2.415
  * [7] M. Neff, Y. Wang, R. Abbott, and M. Walker, “Evaluating the effect of gesture and language on personality perception in conversational agents,” in _Proceedings of the International Conference on Intelligent Virtual Agents_. Springer, 2010, pp. 222–235.
  * [8] M. Y. Lim, J. D. A. Lopes, D. A. Robb, B. W. Wilson, M. Moujahid, E. De Pellegrin, and H. Hastie, “We are all individuals: The role of robot personality and human traits in trustworthy interaction,” in _Proceedings of the 2022 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_. IEEE, 2022, pp. 538–545.
  * [9] M. Moujahid, H. Hastie, and O. Lemon, “Multi-party interaction with a robot receptionist,” in _Proceedings of the 2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_. IEEE, 2022, pp. 927–931.
  * [10] E. T. Hall and E. T. Hall, _The hidden dimension_. Anchor, 1966, vol. 609.
  * [11] M. Argyle, _Bodily communication_. Routledge, 2013.
  * [12] M. Argyle and J. Dean, “Eye-contact, distance and affiliation,” _Sociometry_ , pp. 289–304, 1965.
  * [13] M. A. Dosey and M. Meisels, “Personal space and self-protection.” _Journal of personality and social psychology_ , vol. 11, no. 2, p. 93, 1969\.
  * [14] C. Nass, J. Steuer, and E. R. Tauber, “Computers are social actors,” in _Proceedings of the SIGCHI conference on Human factors in computing systems_ , 1994, pp. 72–78.
  * [15] B. Reeves and C. Nass, “How people treat computers, television, and new media like real people and places,” 1996.
  * [16] J. Mumm and B. Mutlu, “Human-robot proxemics: physical and psychological distancing in human-robot interaction,” in _Proceedings of the 6th international conference on Human-robot interaction_ , 2011, pp. 331–338.
  * [17] A. Sardar, M. Joosse, A. Weiss, and V. Evers, “Don’t stand so close to me: users’ attitudinal and behavioral responses to personal space invasion by robots,” in _Proceedings of the seventh annual ACM/IEEE international conference on Human-Robot Interaction_ , 2012, pp. 229–230.
  * [18] M. L. Walters, D. S. Syrdal, K. L. Koay, K. Dautenhahn, and R. Te Boekhorst, “Human approach distances to a mechanical-looking robot with different robot voice styles,” in _Proceedings of the RO-MAN 2008-The 17th IEEE international symposium on robot and human interactive communication_. IEEE, 2008, pp. 707–712.
  * [19] D. S. Syrdal, K. Dautenhahn, M. L. Walters, and K. L. Koay, “Sharing spaces with robots in a home scenario-anthropomorphic attributions and their effect on proxemic expectations and evaluations in a live hri trial.” in _Proceedings of the AAAI fall symposium: AI in Eldercare: new solutions to old problems_ , 2008, pp. 116–123.
  * [20] J. T. Butler and A. Agah, “Psychological effects of behavior patterns of a mobile personal robot,” _Autonomous Robots_ , vol. 10, no. 2, pp. 185–202, 2001.
  * [21] M. L. Walters, “The design space for robot appearance and behaviour for social robot companions,” Ph.D. dissertation, University of Hertfordshire, 2008.
  * [22] D. Tokmurzina, N. Sagitzhan, A. Nurgaliyev, and A. Sandygulova, “Exploring child-robot proxemics,” in _Proceedings of the Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction_ , 2018, pp. 257–258.
  * [23] D. S. Syrdal, K. Dautenhahn, S. Woods, M. L. Walters, and K. L. Koay, “’doing the right thing wrong’-personality and tolerance to uncomfortable robot approaches,” in _Proceedings of the ROMAN 2006-The 15th IEEE International Symposium on Robot and Human Interactive Communication_. IEEE, 2006, pp. 183–188.
  * [24] M. L. Walters, K. Dautenhahn, R. Te Boekhorst, K. L. Koay, C. Kaouri, S. Woods, C. Nehaniv, D. Lee, and I. Werry, “The influence of subjects’ personality traits on personal spatial zones in a human-robot interaction experiment,” in _Proceedings of the ROMAN 2005. IEEE International Workshop on Robot and Human Interactive Communication, 2005._ IEEE, 2005, pp. 347–352.
  * [25] D. S. Syrdal, K. L. Koay, M. L. Walters, and K. Dautenhahn, “A personalized robot companion?-the role of individual differences on spatial preferences in hri scenarios,” in _Proceedings of the RO-MAN 2007-The 16th IEEE International Symposium on Robot and Human Interactive Communication_. IEEE, 2007, pp. 1143–1148.
  * [26] L. Robert, “Personality in the human robot interaction literature: A review and brief critique,” in _Robert, LP (2018). Personality in the Human Robot Interaction Literature: A Review and Brief Critique, Proceedings of the 24th Americas Conference on Information Systems, Aug_ , 2018, pp. 16–18.
  * [27] A. Furnham, “Handbook of language and social psychology, chapter language and personality,” 1990.
  * [28] J. Pittam, _Voice in social interaction_. Sage, 1994, vol. 5.
  * [29] K. J. Tusing and J. P. Dillard, “The sounds of dominance. vocal precursors of perceived dominance during interpersonal influence,” _Human Communication Research_ , vol. 26, no. 1, pp. 148–171, 2000.
  * [30] M. Schmitz, A. Krüger, and S. Schmidt, “Modelling personality in voices of talking products through prosodic parameters,” in _Proceedings of the 12th international conference on Intelligent user interfaces_ , 2007, pp. 313–316.
  * [31] A. Tapus, C. Ţăpuş, and M. Matarić, “User-robot personality matching and robot behavior adaptation for post-stroke rehabilitation therapy,” _Intelligent Service Robotics_ , vol. 1, pp. 169–183, 04 2008.
  * [32] A. Niculescu, B. Van Dijk, A. Nijholt, and S. L. See, “The influence of voice pitch on the evaluation of a social robot receptionist,” in _Proceedings of the 2011 International Conference on User Science and Engineering (i-USEr)_. IEEE, 2011, pp. 18–23.
  * [33] G. Trovato, R. Paredes, J. Balvin, F. Cuellar, N. B. Thomsen, S. Bech, and Z.-H. Tan, “The sound or silence: investigating the influence of robot noise on proxemics,” in _Proceedings of the 2018 27th IEEE international symposium on robot and human interactive communication (RO-MAN)_. IEEE, 2018, pp. 713–718.
  * [34] S. Bhagya, P. Samarakoon, M. Viraj, J. Muthugala, A. Buddhika, P. Jayasekara, and M. R. Elara, “An exploratory study on proxemics preferences of humans in accordance with attributes of service robots,” in _Proceedings of the 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_. IEEE, 2019, pp. 1–7.
  * [35] R. Macmillan, “74.20 decibel arithmetic,” _The Mathematical Gazette_ , vol. 74, no. 468, pp. 150–153, 1990.
  * [36] R. Chepesiuk, “Decibel hell: the effects of living in a noisy world,” 2005.
  * [37] F. Mairesse and M. Walker, “Personage: Personality generation for dialogue,” in _Proceedings of the 45th annual meeting of the association of computational linguistics_ , 2007, pp. 496–503.
  * [38] S. Al Moubayed, J. Beskow, G. Skantze, and B. Granström, “Furhat: A back-projected human-like robot head for multiparty human-machine interaction,” in _Cognitive Behavioural Systems_ , A. Esposito, A. M. Esposito, A. Vinciarelli, R. Hoffmann, and V. C. Müller, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, pp. 114–130.
  * [39] J. N. Bailenson, J. Blascovich, A. C. Beall, and J. M. Loomis, “Equilibrium theory revisited: Mutual gaze and personal space in virtual environments,” _Presence: Teleoperators & Virtual Environments_, vol. 10, no. 6, pp. 583–598, 2001.
  * [40] B. Reeves and C. Nass, “The media equation: How people treat computers, television, and new media like real people,” _Cambridge, UK_ , vol. 10, p. 236605, 1996.


# “Way back then”: A Data-driven View of 25+ years of Web Evolution

Vibhor Agarwal  Department of Computer Science
University of SurreyGuildfordSurreyUnited Kingdom<EMAIL_ADDRESS>and
Nishanth Sastry  Department of Computer Science
University of SurreyGuildfordSurreyUnited Kingdom<EMAIL_ADDRESS>
(2022)

###### Abstract.

Since the inception of the first web page three decades back, the Web has
evolved considerably, from static HTML pages in the beginning to the dynamic
web pages of today, from mainly the text-based pages of the 1990s to today’s
multimedia rich pages, _etc._. Although much of this is known anecdotally, to
our knowledge, there is no quantitative documentation of the extent and timing
of these changes. This paper attempts to address this gap in the literature by
looking at the top 100 Alexa websites for over 25 years from the Internet
Archive or the “Wayback Machine”, archive.org. We study the changes in
popularity, from Geocities and Yahoo! in the mid-to-late 1990s to the likes of
Google, Facebook, and Tiktok of today. We also look at different categories of
websites and their popularity over the years and find evidence for the decline
in popularity of news and education-related websites, which have been replaced
by streaming media and social networking sites. We explore the emergence and
relative prevalence of different MIME-types (text _vs._ image _vs._ video
_vs._ javascript and json) and study whether the use of text on the Internet
is declining.

web history, internet archive, wayback machine, archive.org

††copyright: acmcopyright††journalyear: 2022††conference: Proceedings of the
ACM Web Conference 2022; April 25–29, 2022; Virtual Event, Lyon,
France††booktitle: Proceedings of the ACM Web Conference 2022 (WWW ’22), April
25–29, 2022, Virtual Event, Lyon, France††price: 15.00††doi:
10.1145/3485447.3512283††isbn: 978-1-4503-9096-5/22/04††ccs: Information
systems World Wide Web

##  1\. Introduction

Since the first webpage was put up by Sir Tim Berners-Lee at CERN 30 years
ago111http://info.cern.ch/hypertext/WWW/TheProject.html, last accessed 9 Dec
2021., the World Wide Web has expanded dramatically year on year, and now
contains at least 1,200 petabytes of data (Starry.com, 2019). It is estimated
that the size of the Web follows Moore’s Law (Zyga, 2009) and doubles in every
few months. Despite this wealth of knowledge and information now available at
our fingertips, the Web can be an ephemeral experience as thousands of URLs
become unreachable and thousands of new URLs are created every minute.
Therefore, capturing the history of the Web is an important and challenging
exercise.

Anecdotally speaking, the Web and its technologies as well as the dominant and
popular platforms or websites have changed considerably since its inception.
Although some of the fundamental underlying technologies such as HTML and HTTP
have remained relatively stable despite newer versions, the visual appearance
of the top websites as well as which websites are popular can and have changed
significantly over the years. For example, GeoCities, an extremely popular web
hosting site in the 1990s, is no longer present, but new sites such as TikTok
are gaining traction rapidly.

In this paper, we are interested in enhancing our quantitative understanding
of the above kinds of change. To examine this, we focus on the Top 100
websites in the Web, as determined by alexa.com, a popular source for ranking
and categorising websites (all ranks as determined in Nov 2021). We also
examine Google Trends data, which allows us to compare the relative popularity
of two search terms or websites over time. Finally, we utilise data from the
Wayback machine or archive.org, which takes periodic snapshots of much of the
Web to get a historic view of these websites from up to 25 years back.

Developing a broad and over-arching quantitative historical understanding of
the evolution of the Web requires us to move beyond individual websites and to
study large collections of websites and web resources, which are identified by
Uniform Resource Locators (URLs)222Datasets and code are available at
https://github.com/socsys/wayback-web-history.. Therefore, although our
datasets have rich data about individual websites and even individual URLs, we
look at them collectively, asking questions about how users spend their time
on different categories of websites as well as how the popularity of
individual websites within a category (such as social media), or the relative
popularity of different categories of websites (_e.g._ news _vs._ social
media), have changed over time. We also group together URLs by their MIME
types333 MIME stands for Multimedia Internet Mail Extension and MIME types
were developed in the context of email. IANA has changed the terminology and
MIME types are now called media types (Freed et al., 2021) as their use has
extended well beyond emails. However, it appears that the usage “MIME type” is
more common, _e.g._ one of our primary data sources, archive.org, uses MIME
rather than media types. Therefore this paper also uses MIME type as it may be
more familiar to the readers., which have historically been used to identify
different kinds of multimedia content, and thus provide a convenient lens to
study whether and to what extent the Web has moved away from its initial
mainly (hyper)text-based origins.

Our analysis reveals a number of interesting trends. For example, we show
evidence for decreased popularity of news and education websites, which have
been replaced by streaming media and social networking. Using Google Trends,
we are able to chart and compare the rise or decline of competing websites,
such as different social media platforms, which may all be competing for a
similar user base. We can also show, using the archive.org data, how the
number of URLs in each website has been increasing steadily over time, and how
the Web is increasingly turning to images, videos and other sophisticated
forms of content. Despite these changes, we still find that text dominates in
terms of the number of web resources (_i.e._ number of Uniform Resource
Locators or URLs) available on most website categories. The time correlations
that can be observed suggest that some of the changes, such as increased usage
of JavaScript Object Notation (JSON) objects or JavaScript URLs may be related
to the rise in popularity of web programming frameworks such as AJAX and
Representational State Transfer (REST).

The rest of this paper is organised as follows. Section 2 discusses related
work including other attempts to take a historical look at the evolution of
the Web. Section 3 describes the datasets we use and how we collect it.
Section 4 then explores how the popularity of different websites and different
website categories has changed in the last 25+ years. Section 5 studies the
MIME types of different URLs and the evolution of multimedia and other forms
of content on the Web. Section 6 concludes with a discussion of our findings,
limitations of our approach and directions for future work.

##  2\. Background and Related work

One of the earliest accounts we could find is a history of hypertext in the
Web, from one of the founding figures of the Web conference, Robert
Cailliau(Cailliau and Ashman, 1999). Maddux (Maddux and Johnson, 1997) focuses
on educational website design. (Foot and Schneider, 2010) introduces the idea
of a historiography of the Web. (Aya et al., 2006) discusses how to build a
research library for the history of the Web.

Within the last decade, considerable scholarly interest has emerged in the
Web’s history. A few studies by Brugger and others (Brügger and Schroeder,
2017; Brügger, 2011; Ben-David and Amram, 2018; Balbi and Magaudda, 2018;
Brugger, 2018) have pioneered more recent approaches to creating a history of
the Web but have mostly focused on specific perspectives rather than broad
quantitative understanding: (Brügger, 2011) focuses on the consequences of the
archiving process for the Internet scholar. (Brügger and Schroeder, 2017)
studies history of media and government, size of web domains, and cultural and
political histories. (Ben-David and Amram, 2018) analyses the socio-technical
epistemic processes behind the construction of historical facts by the
Internet Archive, Wayback Machine. It uncovers specific epistemic processes
through a case-study on the archiving of the North Korean Web. (Jackson, 2012)
explores the UK Web history. It looks at image, HTML, and PDF resources, and
shows how the usage of different formats, versions, and software
implementations has changed over time.

Yet, to our knowledge there is no comprehensive review and the history of the
Web remains relatively understudied. Initial attempts that approach include
Brügger et al. (Brügger and Milligan, 2018), who provides a multifaceted
understanding of the web history. Goggin et al. (Goggin and McLelland, 2017)
brings together the diverse Internet histories that have evolved in different
regions, languages, and social contexts across the globe. (Arora, 2014) uses
spatial metaphors to talk about web history.

In popular perception, the history of the Web is divided into multiple
versions (Shivalingaiah and Naik, 2008): Web 1.0 refers to the first stage of
the World Wide Web evolution with static web pages based on basic HTML
templates. Web 2.0 refers to worldwide websites which highlight user-generated
content, usability, and interoperability for end users. Web technologies used
in Web 2.0 development include AJAX and JavaScript frameworks for dynamic
content. Web 3.0 refers to the evolution of web utilization and interaction
which includes altering the Web into a database and adding semantics. (Allen,
2013) studies what we learn from major “versions” of the Web. The paper
explores Web 2.0 as the marker of a discourse about the nature and purpose of
the Internet in the recent past. It focuses on how Web 2.0 introduced to our
thinking about the Internet a discourse of versions. Such a discourse enables
the telling of a ‘history’ of the Internet which involves a complex
interweaving of past, present and future, as represented by the additional
versions which the introduction of Web 2.0 enabled.

Our quantitative approach relies heavily on web archives. Web Archiving is any
form of deliberate and purposive preserving of web material (Brügger, 2011).
Web Archiving began in 1996 (Brügger and Milligan, 2018) with a stated purpose
of preserving the digital culture of the world. The Internet Archive was
created in 1996 as a non-profit organization to preserve historical
collections that exist in digital format. The Web collection is based on web
crawls performed by Alexa Internet, a for-profit organization and crawls are
done based on where the links point and where the users go. The main archiving
strategy is the snapshot approach, where all the web material that the web
crawler encounters is archived periodically (Brügger, 2011). The frequency of
archival depends upon the rate at which a website is updated, _e.g._ ,
Google’s snapshot is taken from hundreds to thousands of times per
day444https://web.archive.org/web/*/www.google.com, last accessed 9 Dec 2021..
Till date, archive.org has collected more than two decades of Web history,
through regular snapshots of websites, preserved, and made them accessible.
(Rogers, 2017) suggests to use Internet archive for Web history as it provides
websites homepages as they were in the past.

We also rely on Google Trends, a freely accessible tool that allows users to
interact with Internet search data and may provide deep insights into
population behavior and other social phenomena (Nuti et al., 2014). It gives
data based on the search queries on Google Search. (Choi and Varian, 2012)
uses Google trends data to predict important tasks such as unemployment
claims, consumer confidence, automobile sales, etc. (Nuti et al., 2014)
studies the importance of Google trends data in healthcare research. Stephens-
Davidowitz (Stephens-Davidowitz, 2013) pioneered the use of Google trends
data. In the book “Everybody lies” (Stephens-Davidowitz, 2017), he talks about
Google search as a “truth serum” as it is not intended to present or project a
certain persona to everyone (so searches may reveal information such as
personal illnesses which may have a stigma attached to them). In this study,
we also use Google trends data as an indicator of the popularity of websites
over time. The interest in a particular search keyword over time gives the
popularity of that keyword/topic.

There are other tools and datasets which could potentially be used, but are
not considered in this paper. For example, the use of historical crawls of the
web such as the Common Crawl555https://commoncrawl.org, last accessed 26 Jan
2022.. (Mirtaheri et al., 2014) is a brief history of web crawlers. (Ben-
David, 2016) considers reconstructing history of deleted websites through a
case study of the .yu Top Level Domain (which was removed after yugoslavia
ceased to be a separate country).

Future historians using Web Archive data should be aware of attacks such as
possible rewriting of web history (Lerner et al., 2017). We assume in this
study that the archival data we look at have not been tampered with. Archive
data can also be biased. For example, (Thelwall and Vaughan, 2004) explores
the (un)fairness of archive.org based on country balance. (Stock et al., 2017)
presents other angles, such as a historical overview of client side
insecurities in the web browser.

The two main gaps in the literature which we hope to fill are  (1) a
quantitative understanding of web history  (2) a study of how the Web has
evolved over a long period of 25+ years.  This is our long-term research
agenda, and this paper represents a first attempt towards this goal.

##  3\. Methodology

In this section, we describe our methodology for curating a selection of
websites and crawling the historical data of these websites.

###  3.1. Websites List Curation

Alexa is an Internet company which provides web analysis including website
traffic statistics, website comparisons, and website audience. Alexa’s
measurement panel includes a diverse set of 25,000 browser extensions and
plug-ins used by millions of Internet users (Orilind, 2017). We collect the
top 100 websites globally from Alexa.com666https://www.alexa.com/topsites,
last accessed 8 Dec 2021. during November 2021, according to the Alexa traffic
rank. The Alexa rank data is derived from the traffic data provided by the
users in Alexa’s Global Data Panel over a rolling 3 months period and is based
on the browsing pattern of people (Internet, 2021). We also get information –
“Daily Time on Site” which is an estimated daily time spent per visitor on the
website, and “Daily Pageviews per Visitor” which is an estimated daily unique
pageviews on the website, from Alexa.com. For each website, we also obtain the
category of the website, using fortiguard.com777https://fortiguard.com, last
accessed 3 Dec 2021.. In total, we get 24 distinct website categories within
these top 100 Alexa websites.

###  3.2. Crawling Historical Websites Data

To get historical website data, we use the “Wayback Machine”, also known as
Internet Archive — archive.org888https://web.archive.org, last accessed on 3
Dec 2021.. We crawl historical information about each of the 100 websites of
§3.1. We are able to go as far back as 1998 and collect data up to November
2021 from archive.org using selenium999https://www.selenium.dev, last accessed
3 Dec 2021.. Selenium automates web browsing and the Selenium WebDriver drives
a browser natively, as a user would. For example, to get google.com related
data, we crawl the summary view from
archive.org101010https://web.archive.org/details/google.com, last accessed 3
Dec 2021.. We programmatically change the range of years in the web form on
archive.org to 1 year (starting from 1998 all the way to 2021) and get
information related to different MIME-types (text, application, javascript,
images, videos, etc.) such as the number of captured URLs, and new URLs for
each MIME-type per year. In total, we crawl 2400 (=24*100) Internet archive
pages for these top 100 Alexa websites across 24 years (from 1998 to 2021).

We also use Google Trends to get data related to the popularity of search
topics or keywords worldwide (Trends, 2021) for each year. Google Trends data
is available from 2004 onwards, so we obtain information from 2004–2021. For
the popular search engine and social networking websites, we crawl Google
Trends data for their search interest over time as it can be taken as an
indicator of the popularity of a website. The value returned by Google Trends
is termed as “Interest Over Time” and represents the volume of searches
relative to the highest point on the chart. Hence, an “Interest Over Time”
value of 100 stands for the highest number of searches within a set of values
returned. As an example, if we compare two search terms “foo” (which receives
1k, 5K and 2k searches respectively over a three month duration under study)
and “bar” (which was searched 5K, 10K and 8K searches in the same three month
duration), then 10K is taken as “100”, and foo will have “Interest Over Time”
values of 10, 50 and 20, in comparison with bar, which will have values of 50,
80 and 100. Because popularity is computed on a relative basis for a given set
of search terms and normalised to a value of 100, the popularity of different
websites can be compared within a single Google Trends comparison (in Figure
2), but comparison is not possible across the sub-figures. We call this
measure of popularity as the “Google Trends popularity” to distinguish it from
other possible measures of popularity (_e.g._ number of monthly active users
or number of app installs of a website).

We also use data of the number of visits during each year from 1996–2019 to
the top 10 websites across the world that year, compiled from media and
benchmark reports by YouTube user “Data is Beautiful” (Beautiful, 2019). We
cross-check that this data is accurate by comparing with Wikipedia, which
maintains similar data from 2012 onwards, based on Alexa historical data
(Wikipedia.org, 2021). Although this page lists the current top sites
according to Alexa, Wikipedia’s edit history function reveals previous Alexa
rankings. It so happens that this page has been maintained by Wikipedia
contributors since 2012, allowing us to verify the data collected by “Data is
Beautiful” from 2012–19. We also use Wikipedia to extend the data beyond 2019,
to include the number of visits to the top 10 websites in 2020 and 2021.

##  4\. Dynamics of Web Popularity

In this section, we analyse how the Web has evolved over the years, by first
looking at the relative popularity of different website categories currently
and then examining how this has changed over the years.

###  4.1. Popularity of different website categories

We first look at the website categories represented in the top 100 Alexa
websites and the collective popularity of the set of websites from each
category. Figure 1(a) shows the categories represented in the Alexa top 100.
The highest number of websites (21) belongs to the search engine category,
followed by shopping (14), information technology (14), and social networking
(10). Out of 21 search engine domains, 7 domains are Google variants adapted
for different countries, such as google.com (USA and default for worldwide),
google.co.in (Google India), or google.co.jp (Google Japan). Other search
engines include Yandex (Russia) or Baidu (China) which are popular in specific
countries. Interestingly, four adult and pornographic websites appear in the
top 100, making them as numerous as the finance and banking category (also
four websites). From a long time, pornographic and adult content has been
driving media choices — _e.g._ Betamax _vs._ VHS, but also driving the sales
of early pamphlets, news papers, _etc._ (Coopersmith, 1998).

(a) Categories of top 100 Alexa websites

(b) Average daily time (in minutes) spent on the website per visitor.

(c) Average daily pageviews on the website per visitor.

Figure 1. Different bar plots with website categories on x-axis, using the
websites traffic data collected from Alexa.com.

The number of websites in a category can be different from the total time
spent by users in a particular category of websites. Figure 1(b) shows the
average daily time spent on the website per visitor for each category over a
3-month period from Sep to Nov 2021. Surprisingly, visitors spend the most
time on business-related websites daily, followed by shopping websites. In our
list, business websites such as Alibaba.com are at the top, likely because of
the additional services being offered by Alibaba and other websites. Social
networking websites are at the third position, with visitors spending an
average 8 minutes daily. The social network Facebook is at the top with 18
minutes as the daily time spent per visitor, followed by Twitter (12.5
minutes), and LinkedIn (11.25 minutes). These values may be indicative of how
Alexa.com computes the time on each website: As it is based on toolbars
installed in the browsers of its panel of users, we conjecture that the
sojourn time on any website is based on new websites being loaded, and
therefore may be missing pages being updated dynamically using AJAX and other
techniques. Thus, the above timings might represent a lower bound on the
actual times spent on each website.

Figure 1(c) shows a different measure of how frequently each category of
website is used, by measuring average daily page views per visitor over a
3-month period from Sep to Nov 2021 for each of the website categories. Again,
business websites have the highest (9) unique daily page views per visitor.
However, other categories such as social media, which typically rely on AJAX
to dynamically load more content to the same web page on scrolling, have fewer
distinct number of pages recorded and therefore slip in the rankings.

###  4.2. Changes in Popularity

(a) Yahoo! vs. Google.

(b) Social Networking websites.

(c) Facebook vs. Google.

Figure 2. Google Trends Popularity of different websites based on the search
interest over time.

The previous results show the relative popularity of different categories of
websites today. However, the popularity of a website can change dramatically
over time (_e.g._ once dominant search engines such as Altavista, or turnkey
web page creation and hosting services such as GeoCities are no longer
around). To illustrate this dynamic and capture changes in popularity, we turn
to Google Trends, which can provide (normalised) data about the relative
popularity of two or more search keywords or topics over time (_cf._ §3.2).

As a first case study, Figure 2(a) shows the relative number of searches for
Google and Yahoo!, two dominant search engines. Google Trends captures every
search, including search terms entered by people in the URL bar of their
browsers, which are commonly designed as a combined URL bar and a search bar.
Google Chrome, when first released in 2008, introduced this combined URL and
search bar feature called Omnibox (Nahian, 2014). Other major browsers such as
Mozilla Firefox introduced this feature in 2014 (Firefox, 2014). Thus, if a
user enters the name of a website rather than its full URL, this results in a
search in the default search engine (which may likely be Google), and
therefore ends up being captured in Google Trends as one data point. Note that
Google Trends only captures the searches on google.com. It is therefore
interesting to note that despite this obvious bias, there were more searches
for Yahoo! until nearly 2011 (_i.e._ even users whose default search engine
was Google, as configured either manually or as setup automatically upon
installation, were preferring to search for Yahoo!, likely because of the
additional services being offered on the Yahoo! Portal). We also observe that
the relative number of searches for Google on Google.com (likely a result of
users typing in “google” into their browsers’ URL/Search bars) has been in
decline since 2015. Some believe that this may be a result of increasing shift
towards app-based access111111https://www.businessinsider.com/the-number-of-
people-using-search-engines-is-in-decline-2015-9, last accessed 21 Jan 2022..

As a second case study, we look at the category of social networking websites.
Social networks tend to be a “winner take all” market (Prakash et al., 2012)
as users need to be on the same platform as their friends. Therefore, a
handful of websites end up capturing large proportions of users. Figure 2(b)
explores how the Google Trends popularity of major social networks changed
over time. In 2004, the earliest date for Google Trends data, Orkut and
MySpace were the most widely used social networking websites, as evident from
their relative popularity in Google Trends when compared with other prominent
social networking websites. Only around 2008 were they beaten by Facebook, and
later on, by Twitter, based on the number of Google Searches made. Since 2010,
the micro-blogging website Twitter became very popular and attained its peak
in 2013. Around 2012, WhatsApp and Instagram came into existence and started
to capture some of the Google Searches. By 2017, Instagram beat Twitter in
Google Trends popularity. In 2019, WhatsApp beat Instagram in Google Trends
popularity and became the most popular instant messaging application in the
world, according to Google Trends data. In 2021, WhatsApp and Instagram are
the two most popular social networking websites, according to Google Trends
data. TikTok, a short video streaming application introduced in 2017, is
rapidly gaining traction but has still not yet reached the same volume of
searches on Google as the other sites. In contrast, Facebook is not shown as
its search volume is much higher, making it difficult to visualise the changes
in popularity of the other sites. Figure 2(c) compares the popularity of the
two of the most popular websites, Facebook and Google. It can be seen that in
2008 Facebook surpassed Google in Google Trends popularity and reached its
peak in 2013-14. After 2018 Cambridge Analytica Scandal, Facebook’s reputation
has been affected (Peruzzi et al., 2018) and its Google Trends popularity has
deteriorated rapidly.

Figure 3. Percentage of visits to different website categories.

Given this constant flux in the popularity of different websites, we wished to
investigate whether users have been always interested to the same extent in
the same categories of websites and different websites end up being dominant
in each category at different time points (_e.g._ replacing MySpace and Orkut
with Facebook in the social networking category without changing the
proportion of visits to social networking sites), or whether users’ interests
have fundamentally changed over time.

To measure the relative popularities of different categories of websites, we
use data of the number of visits during each year from 1996–2019 to the top 10
websites across the world that year, compiled from media and benchmark reports
by YouTube user “Data is Beautiful” (Beautiful, 2019) (_cf._ description in
Section 3.2).

Figure 3 shows the percentage of visits captured by the top websites of
different categories in each year. Note that the category of search engines
capture a large proportion of visits, and so this category is shown separately
as a line. In the initial years (1995 to 1998), search engine websites
(Yahoo!, Altavista _etc._) secured more than $80\%$ of the visits. Their
visits percentage slowly dropped as new categories of websites became popular.

The stacked bar chart of Figure 3 shows the fraction of visits to the
categories of websites other than the dominating category of search
(normalised to 100% after removing search). This reveals a number of
interesting trends: Web hosting sites such as GeoCities, a highly popular
category in 1995–98 time frame, practically disappears from the top 10 by
2000. Similarly, education disappears after the initial years of popularity of
the Web. News shows a decreasing trend and disappears from the top 10 after
2010. These categories are replaced by social network sites, which emerge
around 2005, and streaming media, which becomes popular around 2010. We also
note the new rise in popularity of online shopping sites in 2021, presumably a
side-effect of the COVID-19 pandemic.

##  5\. Increase in Website Complexity

(a) Number of distinct types for each MIME category per year.

(b) Number of new URLs for different MIME-types for all the top 100 websites
across the years, on log10 scale.

Figure 4. Number of distinct MIME-types and URLs with these MIME-types.

(a) The total number of new Image URLs for each website category, normalized
by the number of websites per category and sorted in descending order.

(b) The total number of new Application URLs for each website category,
normalized by the number of websites per category and sorted in descending
order.

Figure 5. Number of URLs on top websites and their MIME-types.

(a) “Social Networking” category.

(b) “Search Engine” category.

Figure 6. Number of new URLs for different MIME-types for website categories,
on log10 scale.

We next turn our attention to the size and complexity of individual websites
and look at the number of web resources (URLs) on individual websites, as well
as the diversity of the kinds of resources (_e.g._ text _vs._ image _vs._
video) found at these URLs. The number of URLs required to fully render a page
is indicative of the complexity of the page or site. A rich diversity of MIME
types requires more complex code in the browser and ability to display
different types of content. The number of MIME types therefore can be seen as
another measure of website complexity.

Figure 7. Number of new URLs for Image MIME-type for “Social Networking”
category.

Figure 4(a) captures the overall trend, by looking at the number of distinct
MIME types represented in URLs of the Alexa Top 100 websites, collected in Nov
2021. Since 1998, when about 19 distinct MIME types were present on these top
websites, there has been over a five fold increase to 113 different MIME types
in 2021. We also see a rapid increase in the number of different application-
specific MIME types. Note that the number of distinct MIME types is different
from the actual prevalance of web resources (URLs) with these MIME types.
Figure 4(b) shows the total raw numbers of new URLs (_i.e._ URLs added since
the previous year) across the Alexa top 100 sites, from 1998 to 2021. This
shows a general trend of websites getting increasingly complex with more and
more URLs over the years (note that the Y axis is in log-scale). We can also
see an increase in usage of web resources with application-specific MIME
types, and in addition an increased usage of images over the years. Despite
this, it is interesting to see that even in 2021, text-based MIME types
(mainly HTML) still constitute a significant proportion of the Web, by number
of URLs.

To further study the usage of images and application-related MIME types, we
explore how their usage varies across websites of different categories in
Figure 5(a) and Figure 5(b) respectively. In both cases, we find that the
social networking category has the maximum number of image as well as
application (mainly, json and javascript) URLs. Fig 6(a) digs deeper, showing
the numbers of URLs of different MIME types across the years in websites
belonging to the social networking category. This clearly shows the rise in
URLs of MIME type application/* starting in 2005, with the rise of AJAX and
the RESTful programming paradigm that characterised Web 2.0 (o’Reilly, 2009).
Similarly, the usage and the number of image and other non-text MIME types
appears to increase as Internet bandwidth availability increases over the
years. We find similar trends for other categories of websites as well. As one
example, Figure 6(b) shows that the search engine category follows a similar
trend.

Each category of MIME types discussed above (_e.g._ image or text) aggregates
various different formats of a certain type (_e.g._ images can be image/jpeg,
image/gif _etc._). Each such aggregate category itself has an increase in
complexity, with new formats being added under each category over the years.
Figure 7 illustrates this for image-related MIME types, by looking at the
number of different image-type URLs over the years and the different kinds of
image MIME types being used on Alexa top 100 sites, as recorded by
archive.org. In 1998 and 2000, GIF and JPEG images dominated. Since then, PNG
has started to emerge as a common choice of image format. More recently,
vector graphics (SVG) have started to be used as well for images. Again, we
observe similar trends for other MIME-type classes such as application/* (not
shown).

##  6\. Discussion and Conclusions

Through the Web, humanity has created a vast treasure trove of data the likes
of which has never been seen before. Recently, there have been concerns that
such data is being exploited for commercial and other more malicious ends. For
example, detailed online historical profiles of the users are created by
various third-parties (Hu et al., 2020; Hu and Sastry, 2019, 2020), mainly to
target ads and generate revenue (Agarwal et al., 2021, 2020; Vekaria et al.,
2021). It also poses various privacy issues for the users online (Beigi and
Liu, 2019). However, this data revolution has also presented the historians
with unparalleled amounts of primary data, which can yield interesting
insights and new ways to develop historical understanding of our digital
lives.

To take advantage of this data, we need to develop new ways to glean the
historical patterns present in the data. This work represents a first attempt
to develop a quantitative understanding of the evolution of the Web. We looked
at several sources of data, ranging from the periodic snapshots of websites on
archive.org to volumes of searches for different keywords, as captured on
Google Trends. To keep this study tractable, we also focused on the most
popular websites, using rankings such as Alexa Top websites.

Our initial study provided quantitative evidence for several interesting
patterns, such as the decline in popularity of traditional news media and the
increasing usage of social and streaming media instead. We also showed how the
Web has become increasingly complex with different kinds of media types being
used over the years. Websites can also change in popularity over time and we
showed how Google Trends can be a simple yet effective tool to chart this, for
example on social networking websites.

We believe that the techniques we developed in this paper, simple though they
are, have potential to be developed further for a more thorough and in-depth
understanding of the Web and its effects on humanity and society. These
avenues for future work also highlight important limitations of our current
work: For example, as a first attempt at a quantitative historical
understanding, we only focus on the Alexa Top 100 websites. However,
archive.org has data about many other websites which we plan to study in
future work. Similarly, the common crawl121212https://commoncrawl.org/, last
accessed 26 Jan 2022. represents a huge dataset of Web URLs and their content,
that can provide additional information. Alexa ranking is also not the only
and the definitive ranking of websites; thus results may be slightly different
if other rankings of Websites (_e.g._ Similarweb) are used. Also, we do not
know the demographics of the users or the kinds of devices they use for
browsing with Alexa plug-ins that generate Alexa rankings and Websites traffic
information. Secondly, our focus on the “worldwide” top websites does not, in
general, include sites that may be more relevant in non-western cultures,
although some populous countries such as China are already represented in the
Global top 100 through websites such as Baidu, Weibo and Alibaba. Future work
may want to focus on country- or region-specific issues. Our list is also
heavily biased towards the English Web, so a study of history of the Web in
other languages is an important understudied problem. Other data sets we use,
such as Google Trends, also introduce biases into our study: _e.g._ Google
Trends does not capture the interests of users of search engines other than
Google; thus it becomes important to check whether other search engines such
as Bing or DuckDuckGo see similar trends as seen on Google. Finally, we also
note that a quantitative study can miss several of the nuances that come from
close study in the social sciences, thus there is a need to bridge Big Data
methods such as ours with more traditional social science approaches
(Karamshuk et al., 2017). Qualitative studies and approaches such as the
regularly conducted Pew Internet
Surveys131313https://www.pewresearch.org/politics/methodology/collecting-
survey-data/internet-surveys/, last accessed 26 Jan 2022. add important
dimensions that cannot be easily captured by purely observational data-based
studies such as ours.

## References

  * (1)
  * Agarwal et al. (2020) Pushkal Agarwal, Sagar Joglekar, Panagiotis Papadopoulos, Nishanth Sastry, and Nicolas Kourtellis. 2020.  Stop tracking me bro! differential tracking of user demographics on hyper-partisan websites. In _Proceedings of The Web Conference 2020_. 1479–1490.
  * Agarwal et al. (2021) Vibhor Agarwal, Yash Vekaria, Pushkal Agarwal, Sangeeta Mahapatra, Shounak Set, Sakthi Balan Muthiah, Nishanth Sastry, and Nicolas Kourtellis. 2021.  Under the Spotlight: Web Tracking in Indian Partisan News Websites. In _Proceedings of the International AAAI Conference on Web and Social Media_ , Vol. 15. 26–37.
  * Allen (2013) Matthew Allen. 2013\.  What was Web 2.0? Versions as the dominant mode of internet history.  _New Media & Society_ 15, 2 (2013), 260–275.
  * Arora (2014) Payal Arora. 2014\.  _The leisure commons: A spatial history of Web 2.0_.  Routledge.
  * Aya et al. (2006) Selcuk Aya, Blazej J Kot, Lucia Walle, Ruth Mitchell, Pavel Dmitriev, and William Y Arms. 2006\.  Building a research library for the history of the web. In _Proceedings of the 6th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL’06)_. IEEE, 95–102.
  * Balbi and Magaudda (2018) Gabriele Balbi and Paolo Magaudda. 2018.  _A history of digital media: An intermedia and global perspective_.  Routledge.
  * Beautiful (2019) Data Is Beautiful. 2019\.  Most Popular Websites 1996 - 2019.  (2019).  https://www.youtube.com/watch?v=2Uj1A9AguFs
  * Beigi and Liu (2019) Ghazaleh Beigi and Huan Liu. 2019.  ” Identifying novel privacy issues of online users on social media platforms” by Ghazaleh Beigi and Huan Liu with Martin Vesely as coordinator.  _ACM SIGWEB Newsletter_ Winter (2019), 1–7.
  * Ben-David (2016) Anat Ben-David. 2016\.  What does the Web remember of its deleted past? An archival reconstruction of the former Yugoslav top-level domain.  _New Media & Society_ 18, 7 (2016), 1103–1119.
  * Ben-David and Amram (2018) Anat Ben-David and Adam Amram. 2018.  The Internet Archive and the socio-technical construction of historical facts.  _Internet Histories_ 2, 1-2 (2018), 179–201.
  * Brügger (2011) Niels Brügger. 2011\.  Web archiving—Between past, present, and future.  _The handbook of Internet studies_ (2011), 24–42.
  * Brugger (2018) Niels Brugger. 2018\.  _The archived web: Doing history in the digital age_.  MIT Press.
  * Brügger and Milligan (2018) Niels Brügger and Ian Milligan. 2018.  _The SAGE handbook of web history_.  Sage.
  * Brügger and Schroeder (2017) Niels Brügger and Ralph Schroeder. 2017.  _The web as history: Using web archives to understand the past and the present_.  UCL Press.
  * Cailliau and Ashman (1999) Robert Cailliau and Helen Ashman. 1999.  Hypertext in the Web—a history.  _ACM Computing Surveys (CSUR)_ 31, 4es (1999), 35–es.
  * Choi and Varian (2012) Hyunyoung Choi and Hal Varian. 2012.  Predicting the present with Google Trends.  _Economic record_ 88 (2012), 2–9.
  * Coopersmith (1998) Jonathan Coopersmith. 1998\.  Pornography, technology and progress.  _Icon_ (1998), 94–125.
  * Firefox (2014) Mozilla Firefox. 2014\.  Firefox Release 33.0 Notes.  (2014).  https://www.mozilla.org/en-US/firefox/33.0/releasenotes/
  * Foot and Schneider (2010) Kirsten Foot and Steven Schneider. 2010.  _Object-oriented web historiography_.  na.
  * Freed et al. (2021) Ned Freed, Alexey Melnikov, and Murray Kucherawy. 2021\.  Media Types.  (2021).  https://www.iana.org/assignments/media-types/media-types.xhtml
  * Goggin and McLelland (2017) Gerard Goggin and Mark J McLelland. 2017.  _The Routledge companion to global Internet histories_.  Routledge New York.
  * Hu et al. (2020) Xuehui Hu, Guillermo Suarez de Tangil, and Nishanth Sastry. 2020.  Multi-country study of third party trackers from real browser histories. In _2020 IEEE European Symposium on Security and Privacy (EuroS &P)_. IEEE, 70–86.
  * Hu and Sastry (2019) Xuehui Hu and Nishanth Sastry. 2019.  Characterising Third Party Cookie Usage in the EU after GDPR. In _Proceedings of the 11th ACM Conference on Web Science_. ACM.
  * Hu and Sastry (2020) Xuehui Hu and Nishanth Sastry. 2020.  What a Tangled Web We Weave: Understanding the Interconnectedness of the Third Party Cookie Ecosystem. In _Proceedings of the 12th ACM Conference on Web Science_. ACM.
  * Internet (2021) Alexa Internet. 2021\.  How are Alexa’s traffic rankings determined?  (2021).  https://support.alexa.com/hc/en-us/articles/200449744-How-are-Alexa-s-traffic-%20rankings-determined
  * Jackson (2012) Andrew N Jackson. 2012\.  Formats over time: Exploring uk web history.  _arXiv preprint arXiv:1210.1714_ (2012).
  * Karamshuk et al. (2017) Dmytro Karamshuk, Frances Shaw, Julie Brownlie, and Nishanth Sastry. 2017.  Bridging big data and qualitative methods in the social sciences: A case study of Twitter responses to high profile deaths by suicide.  _Online Social Networks and Media_ 1 (2017), 33–43.
  * Lerner et al. (2017) Ada Lerner, Tadayoshi Kohno, and Franziska Roesner. 2017\.  Rewriting history: Changing the archived web from the present. In _Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security_. 1741–1755.
  * Maddux and Johnson (1997) Cleborne D Maddux and D LaMont Johnson. 1997.  The World Wide Web: History, cultural context, and a manual for developers of educational information-based Web sites.  _Educational Technology_ 37, 5 (1997), 5–12.
  * Mirtaheri et al. (2014) Seyed M Mirtaheri, Mustafa Emre Dinçktürk, Salman Hooshmand, Gregor V Bochmann, Guy-Vincent Jourdan, and Iosif Viorel Onut. 2014\.  A brief history of web crawlers.  _arXiv preprint arXiv:1405.0749_ (2014).
  * Nahian (2014) Jaber Al Nahian. 2014\.  Combine Firefox Address Bar and Search Bar into One Like Chrome.  (2014).  https://www.techgainer.com/combine-firefox-address-bar-search-bar-like-chrome/
  * Nuti et al. (2014) Sudhakar V Nuti, Brian Wayda, Isuru Ranasinghe, Sisi Wang, Rachel P Dreyer, Serene I Chen, and Karthik Murugiah. 2014.  The use of google trends in health care research: a systematic review.  _PloS one_ 9, 10 (2014), e109583.
  * o’Reilly (2009) Tim o’Reilly. 2009\.  _What is web 2.0_.  ” O’Reilly Media, Inc.”.
  * Orilind (2017) Greg Orilind. 2017\.  Top 6 Myths about the Alexa Traffic Rank.  (2017).  https://blog.alexa.com/top-6-myths-about-the-alexa-traffic-rank/
  * Peruzzi et al. (2018) Antonio Peruzzi, Fabiana Zollo, Walter Quattrociocchi, and Antonio Scala. 2018.  How news may affect markets’ complex structure: The case of cambridge analytica.  _Entropy_ 20, 10 (2018), 765.
  * Prakash et al. (2012) B Aditya Prakash, Alex Beutel, Roni Rosenfeld, and Christos Faloutsos. 2012.  Winner takes all: competing viruses or ideas on fair-play networks. In _Proceedings of the 21st international conference on World Wide Web_. 1037–1046.
  * Rogers (2017) Richard Rogers. 2017\.  Doing Web history with the Internet Archive: screencast documentaries.  _Internet Histories_ 1, 1-2 (2017), 160–172.
  * Shivalingaiah and Naik (2008) D Shivalingaiah and Umesha Naik. 2008.  Comparative study of web 1.0, web 2.0 and web 3.0.  (2008).
  * Starry.com (2019) Starry.com. 2019\.  How big is the internet?  (2019).  https://starry.com/blog/inside-the-internet/how-big-is-the-internet
  * Stephens-Davidowitz (2013) Seth Stephens-Davidowitz. 2013\.  _Essays using Google data_.  Harvard University.
  * Stephens-Davidowitz (2017) Seth Stephens-Davidowitz. 2017\.  _Everybody Lies: The New York Times Bestseller_.  Bloomsbury Publishing.
  * Stock et al. (2017) Ben Stock, Martin Johns, Marius Steffens, and Michael Backes. 2017\.  How the web tangled itself: Uncovering the history of client-side web (in) security. In _26th $\\{$USENIX$\\}$ Security Symposium ($\\{$USENIX$\\}$ Security 17)_. 971–987.
  * Thelwall and Vaughan (2004) Mike Thelwall and Liwen Vaughan. 2004.  A fair history of the Web? Examining country balance in the Internet Archive.  _Library & information science research_ 26, 2 (2004), 162–176.
  * Trends (2021) Google Trends. 2021\.  Search Interest over Time.  (2021).  https://trends.google.com/trends/explore
  * Vekaria et al. (2021) Yash Vekaria, Vibhor Agarwal, Pushkal Agarwal, Sangeeta Mahapatra, Sakthi Balan Muthiah, Nishanth Sastry, and Nicolas Kourtellis. 2021\.  Differential Tracking Across Topical Webpages of Indian News Media. In _13th ACM Web Science Conference 2021_. 299–308.
  * Wikipedia.org (2021) Wikipedia.org. 2021\.  List of most visited websites.  (2021).  https://en.wikipedia.org/wiki/List_of_most_visited_websites
  * Zyga (2009) Lisa Zyga. 2009\.  Internet Growth Follows Moore’s law Too.  (2009).  https://phys.org/news/2009-01-internet-growth-law.html