# “Novae, supernovae, or something else?” – (Super-)nova highlights
from Hoffmann & Vogt are quite certainly comets (AD 668 and 891)

Ralph Neuhäuser,1 Dagmar L. Neuhäuser,2 Jesse Chapman3
1Astrophysikalisches Institut und Universitäts-Sternwarte, Friedrich-Schiller-
Universität Jena, Schillergäßchen 2-3, 07745 Jena, Germany
2Independent scholar, 39012 Meran/o, Autonome Provinz Bozen,
Südtirol/Provincia Autonoma di Bolzano, Alto Adige, Italy
3Department of East Asian Studies, New York University, New York NY, 10003,
United States E-mail<EMAIL_ADDRESS>
(Accepted 2020 October 26. Received 2020 October 26; in original form 2020
September 21)

###### Abstract

Galactic novae and supernovae can be studied by utilizing historical
observations, yielding explosion time, location on sky etc. Recent
publications by Hoffmann & Vogt present CVs, supernova remnants, planetary
nebulae etc. as potential counterparts based on their list of historically
reported transients from the Classical Chinese text corpus. Since their
candidate selection neglects the state-of-the-art (e.g. Stephenson & Green),
and since it includes ‘broom stars’ and ‘fuzzy stars’, i.e. probable comets,
we investigate their catalogue in more detail. We discuss here their two
highlights, the suggestion of two ‘broom star’ records dated AD 667 and 668 as
one historical supernova and of the ‘guest star’ of AD 891 as recurrent nova U
Sco. The proposed positional search areas are not justified due to translation
and dating problems, source omission, as well as misunderstandings of
historical Chinese astronomy and unfounded textual interpretations. All
sources together provide strong evidence for comet sightings in both AD 668
and 891 – e.g., there are no arguments for stationarity. The AD 667 record is
a misdated doublet of 668. Our critique pertains more generally to their whole
catalogue of ‘24 most promising events’: their speculations on counterparts
lack a solid foundation and should not be used in follow-ups.

###### keywords:

novae, cataclysmic variables – supernovae – comets – comets: individual: 668,
X/891 J1

††pubyear: 2020††pagerange: “Novae, supernovae, or something else?” –
(Super-)nova highlights from Hoffmann & Vogt are quite certainly comets (AD
668 and 891)–“Novae, supernovae, or something else?” – (Super-)nova highlights
from Hoffmann & Vogt are quite certainly comets (AD 668 and 891)

##  1 Introduction

There are many historical records about transients, which can include
supernovae (SN), novae, comets, variable stars, but also aurorae, meteors,
bolides, etc. Since it is difficult to distinguish between such phenomena, one
has to set criteria (e.g., Stephenson & Green 2005; R. Neuhäuser & D.L.
Neuhäuser 2015): while novae and SNe are stationary and star-like, comets move
relative to the stars and often show tails – and this has to become clear from
the transmitted sources (e.g., D.L. Neuhäuser et al. 2020). For application to
modern astrophysical problems, one has to follow a reliable methodology:
literal translations based on critical text editions, possibly dating
corrections, context, parallel sources, further transmissions, close reading
etc. (R. Neuhäuser, D.L. Neuhäuser & Posch 2020).

Most recently, by searching for historical nova eruptions, Hoffmann, Vogt &
Protte (2020, henceforth HVP20) present a list of ‘24 most promising events’
based on parts of the Classical Chinese text corpus. They claim to have used a
new method, e.g. re-interpretating the texts and excluding comets and meteors.
Astrophysical objects in their positional error circles, like CVs, are then
discussed in Hoffmann & Vogt (2020a, henceforth HV20a) and Hoffmann & Vogt
(2020b, henceforth HV20b). As their main highlights, they feature:
(i) the ‘broom star’ records for AD 667 and 668 together would be a SN seen
for one year (and they suggest a SN remnant), and
(ii) the ‘guest star’ of AD 891 would be recurrent nova U Sco.
We present strong evidence that the records are parts of comet occurrences –
the state-of-the-art is not considered in HVP20, HV20a,b, where published
evidence that contradicts their claims is missing.

Historical records from East Asian court astronomers of the last two millennia
use their nomenclature, calendar, and celestial system to specify details
regarding each records, for instance observing time and location on sky;
background is provided, e.g., in Sun & Kistemaker (1997, henceforth SK97);
philological and cultural competence is needed (Chapman, Csikszentmihalyi &
Neuhäuser 2014; Chapman et al. 2015).

A recent review on historical SNe is found in Stephenson & Green (2002); a few
additional Arabic language records of SNe 1006, 1572, and 1604 are collected
in Rada & R. Neuhäuser (2015), R. Neuhäuser et al. (2016, 2017a), and R.
Neuhäuser, Ehrig-Eggert & Kunitzsch (2017b). Recent work on historical ‘guest
stars’ including novae and SNe can be found in Stephenson & Green (2005,
2009), a cometography in Kronk (1999), a critical collection of Chinese
records of ‘broom’ and ‘fuzzy stars’ in Pankenier, Xu & Jiang (2008,
henceforth P+08) – all ignored in HVP20 and HV20a,b. The ‘24 most promising
events’ in HVP20 include transients called ‘broom star’ or ‘fuzzy star’,
and/or were rejected previously for other reasons; we investigate as examples
their two main highlights: the Japanese ‘guest star’ record of AD 891 is not
about recurrent nova U Sco, but most probably about the last position of a
well-known comet also observed in China, Europe, and Arabia (Sect. 2), and the
‘broom star’ records of AD 667 and 668 together are not one new historical SN,
but a comet with tail and motion relative to the stars in AD 668 (AD 667
misdated). We finish with a brief summary.

##  2 The Japanese ‘guest star’ record in AD 891: not nova U Sco, but most
likely the last sighting of Alfred’s comet

The interpretation of the AD 891 guest star in HVP20 and HV20a,b is based
purely on the record from Japan as translated in Xu, Jiang & Pankenier (2000,
pp. 136, 332), additions in square brackets:
891 May 11: ‘3rd year of the Kampyō reign period, 3rd month, 29th day [of
lunar month], [day] jimao [16]. At the hour of hai [21-23h], there was a guest
star [ke xing] east of the star Dongxian at a distance of about one cun’ (Mei
getsu ki ch. 23, compiled by Fujiwara Sadaie AD 1162-1241, identical in Nihon
kiryaku zen 21).

Classical Chinese does not distinguish between singular or plural, e.g.,
‘star’ or ‘stars’; the four stars of Dongxian built the eastern wall of an
enclosed market, most likely $\phi,\chi,\psi,\omega$ Oph (SK97, pp. 72, 150),
which one was meant is unspecified. The linear scales cun (also chi and zhang)
were applied as angles to tail length and angular separations between
celestial objects (e.g., the last position of 1P/Halley in AD 760, Xu et al.
2000; D.L. Neuhäuser et al. 2020). The ganzhi date jimao is from the 60-day
(sexagenary) cycle, an alternative to the day within the lunar month, here
‘29th day’.

HVP20 ‘selected those [old events] … not only visible within a certain hour
(to exclude meteors)’ (abstract) – while the small asterism Dongxian is seen
all night on May 11, the phenomenon is reported for 21-23h in the two Japanese
records in Xu et al. (incorrect in HVP20 table 3); this record should be
excluded, but it is listed as one of their ‘24 most promising events’ for
novae etc. (HVP20; HV20a,b).111Also the event for 1497 Sep 20 from China
contradicts their own criterion, reported just ‘at dusk’ (Xu et al. 2000, p.
143), although the position would be visible all night. The given ‘$\beta$
UMi’ as alleged identification of ‘Tianji star’ (HVP20, table 3) demonstrates
that what was announced as their own ‘new method’, namely ‘a careful re-
reading of the respective historical records and an independent interpretation
of the positions given therein (with regard to the existing works on the
identification of old ancient names)’ (HVP20), was not applied: ‘Tianji’ is in
the translation of Xu et al. (2000), a typo for ‘Tianjiu’, see Chinese text on
their pp. 336/337 (and Stephenson & Green 2009). ‘Tianjiu’ is an asterism of
ten stars with 22 And (SK97, p. 191) and/or $\theta$ And (P+08, p. 462),
ancient maps confirm the number (Stephenson & Green 2009: ‘Tianjiu consists of
the three stars $\theta,\rho,\sigma$ And’, which is incomplete). Three
asterisms with different Chinese characters are all transcribed ‘Tianji’, none
of them with $\beta$ UMi, which is one of the five stars of Beiji – a
prominent asterism near the North Pole (SK97, pp. 217ff). Since HVP20 use only
this source (without duration, stationarity is questionable), it seems to be a
brief, unspecified transient event. However, HV20b suggest recurrent nova U
Sco as its counterpart. Furthermore, in their table 3 regarding AD 891, HVP20
claim to ‘consider only stars in the eastern half’ – U Sco is west of the
Dongxian skeleton (HV20a figure A-16); neither their assumption that the
‘principle star’ of Dongxian was meant (it is disputed, which star was the
principle star, see SK97, p. 48) nor the size of their positional error circle
of $3^{\circ}$ (HVP20 table 3) is justified. While the source says 1 cun
($\sim\,0.1^{\circ}$) east of ‘an unspecified member’ (Stephenson & Green
2009, p. 44), HV20b ‘postulate a corrupted text preserving 1 cun instead of 1
chin ($\sim 1^{\circ}$)’ (section 5.3) – they mean ‘chi’. In sum, only the
textual (against their own selection criterion) and positional interpretation
seems to be corrupt. As we will see next, further historical records and
scholarly research with evidence against their claim is not considered; they
do not discuss alternatives, e.g., planet Uranus ($\sim 5.5$ mag) inside their
error circle on the given date – however, another option is more reliable.

P+08 (pp. 102, 535-6) list more records from China (a and b); text b is
already in Ho (1962) no. 313 together with the ‘guest star’ report from Japan
(May 11), with a similar translation as above – Ho is used in HVP20 and
HV20a,b, but not here; AD 891 May 12:
(a) ‘2nd year of the Dashun reign period of Emperor Zhaozong of the Tang
Dynasty, 4th month, day gengchen [17]; a broom star [hui xing] entered Taiwei’
(Xin Tang shu: Zhaozong ji, AD 1061).
(b) ‘[same date] there was a broom star [hui xing] over 10 zhang long in
Santai that travelled eastward and entered Taiwei, sweeping Dajiao and
Tianshi. In the 5th month, day jiaxu [11 = July 5], it no longer appeared
[lit.: was not seen]’ (Xin Tang shu: tianwen zhi).

A hui xing is a star (xing) with a tail like a broom (hui), the common wording
for comets: probably, the longest tail length here was ‘over 10 zhang’ ($\sim
100^{\circ}$). These reports are transmitted in the Xin Tang shu, the ‘New
history of the Tang dynasty’. Text (a) is shortened, the comet was discovered
in the night May 12/13 presumably when entering the enclosure Taiwei, but (b)
gives more information on the comet’s path: it appeared at Santai
($\iota,\kappa,\lambda,\mu,\nu,\xi$ UMa) moving east to the Taiwei enclosure
(stars in CrB, Leo, Vir), then passing Dajiao ($\alpha$ Boo) and the Tianshi
enclosure (stars in Her, Ser, Oph, Aql); applying source critique shows that
text (a) combined the first date with a later position – the transmitted
records are compilations, which shortened significantly most of the original
protocols.

Stephenson & Green (2009) thought that ‘it seems likely’ that the reports from
Japan and China pertain to two different objects, because ‘on May 12, the
broom star was at Santai, some 90 deg from Dongxian’ (p. 44); however, let us
consider the comet path: Dongxian in Oph is just off the south gate of the
Tianshi enclosure; hence, the position of the ‘guest star’ (Japan) fits well
with the last reported area for the ‘broom star’ (China), Tianshi, close to
the ecliptic, a location typical for departing comets (see Ho 1962, no. 313;
Kronk 1999). The timing also fits: the comet record from China (b) gives ‘not
seen’ on July 5, that could be due to clouds, faintness, or disappearance of
the comet; either way, during July Dongxian is seen from Kyoto best ‘at the
hour of hai’ (21-23h), in the previous double hour the Sun is setting,
afterwards the asterism sets. Thus, most likely, the Japanese compilation
transmits the position, where the comet vanished – however, recorded under the
date of first sighting, similar to text (a) from China.

Weather may have influenced the observational records from China; a misdated
doublet of the Chinese transmission to AD 893 (P+08 (b), pp. 103, 536, also Ho
1962 no. 315, the title of the reign period is confused) gives: ‘then [the
broom star] was not seen due to clouds and overcast’ [condition]. A bad
weather period is also reported before the comet appeared (additionally to
P+08 for AD 893 (b), our translation): ‘Year 2, 2nd month: the sky was
overcast for a long time (jiu yin). Upon the yiyou [22] night of the 4th month
[converted to AD 891: May 17/18], the clouds opened some. There was a comet in
Shangtai [upper step of Santai] …’ (then like above). In this misdated
transmission, which could come from a different location, the comet seems to
be detected about May 17/18. Hasegawa (2002) calculated approximate orbital
elements for AD 893 – but the ganzhi dates are only consistent for AD 891:
there would be no ‘day jiaxu [11] in the 5th month’ in AD 893 (see source b
from AD 891).

The appearance of the impressive comet is also attested in Europe and Arabia
(e.g., Kronk 1999), not noticed by HVP20 and HV20a,b, a reliable source is the
Anglo Saxon Chronicle (England):
‘AD 891 … And the same year after Easter [Apr 4], at the rogation days [May
10-12] or before, there appeared the star which is called in Latin cometa.
Some men say that it is in English the long-haired star, for there shines a
long ray from it, sometimes on one side, sometimes on every side’ (Whitelock
1979, p. 201).

From Arabia, Ibn al-Jawzī (al-Muntazam fī al-tārīkh, AD 1201) reported: ‘AH
278 (AD 891): A very brilliant star rose on 28 Muḥarram [May $13\pm 2$] and
then its brilliance became locks of hair’ (Cook 1999); similar by Ibn al-Athīr
(al-Kāmil fī al-tārīkh).

The discovery might have been in England; with full historical right, Schove
(1984) called this long-haired star ‘Alfred’s comet’ after the influential
King of Wessex and then of all Anglo-Saxons (AD 848/9-899), who initiated the
Chronicle compilation; designation in Kronk (1999): comet X/891 J1. Sometimes
the comet is dated AD 892, which is not justified – a contemporary source
clarifies the year: ‘AD 891. Stella cometis. Eclypsis solis’, first the comet,
then the solar eclipse of 891 Aug 8 (Ann. Alamannic. cont. Sangallensis
tertia, St. Gallen, AD 926, MGH SS 1, p. 52, ed. Pertz).

In sum, caution is needed when considering historical observations of
transients as evidence for rare high-energy events like novae or SNe: similar
to AD 891, records of ‘guest stars’ in AD 85 and 1166 pertain most likely to
‘broom star’ sightings, for a ‘guest star’ in AD 64, among others, a cometary
interpretation is more reliable (Stephenson & Green 2009) – the latter is also
one of the ‘most promising [non-cometary] events’ in HVP20. In P+08 many such
conversions are attested, but already in Ho (1962) the problem was well-known
(e.g., no. 228, regarded as nova by others): ‘we shall come across several
cases where guest stars turned into (hui) comets and vice versa’ (p. 137).
Comets can be observed as small, large, or so-called fuzzy (guest) stars, in
particular during the first and last sighting (see also the records from
England and Arabia) – these phenotypical descriptions do not contradict their
physical nature.

##  3 The ‘broom star’ records for AD 668 and 667: neither a new supernova in
AD 667-8 nor
Nova V392 Per, but most certainly a comet

HV20a present the event from AD 668 as nova. HV20b interpret the ‘broom star’
records for AD 667 and 668 together as historical SN seen for one year. Since
the reported phenomena are explicitly called hui xing (lit. ‘broom star’, star
with tail), they are not listed in Xu, Jiang & Pankenier (2000), a compilation
in particular of likely stationary ke xing (lit. ‘guest star’). The AD 667 and
668 reports are instead found in the new collection of Chinese records of
‘broom’ and ‘fuzzy stars’ as well as likely non-stationary guest stars
(Pankenier, Xu & Jiang 2008). HVP20 use the old, partial translation from Ho
(1962), a collection of 579 ‘comets and novae’ – completely revised by Xu et
al. (2000) and P+08 with classical Chinese texts. We will see that the AD 667
text is a misdated copy of the AD 668 sources and that the transmissions can
only be interpreted as a single comet.

We present step by step the transmitted texts from China and Korea as
translated in P+08 (pp. 73-74, 518-519), variants from Ho (1962, no. 251, 252,
but in Pinyin transcription) in round brackets.
First, 668 May 17 - June 14, i.e. the full 4th lunar month, P+08:
(c) ‘1st year of the Zongzhang reign period of Emperor Gaozong of the Tang
Dynasty, 4th month; a broom [hui] star appeared [or: was seen] at (Ho no. 252:
above) Wuche … on the 22nd day, the star [xing] was extinguished’ (Jiu Tang
shu 36.1320: tianwen zhi, compiled AD 945, for text omission, see below).

This short notice (as given by Ho 1962, no. 252) about a broom star in the
‘Old history of the Tang Dynasty’ is the basis for the interpretation as
recurrent nova in HV20a and (together with the misdated doublet in 667, Ho no.
251) as SN in HV20b. The asterism Wuche ($\alpha$ Aur + 4 stars) is mainly the
same as the western constellation figure Auriga; Ho (1962, no. 252) translated
erroneously ‘above Wuche’.222The preposition shang, i.e. ‘above’, does not
belong to ‘Wuche’: after the Chinese text ‘hui jian Wuche’ (‘broom [star]
appeared [at] Wuche’), text (c) has ‘shang bi zheng dian’ for ‘the Emperor
avoided the main hall’ – ‘shang’ belongs to the following clause and refers to
the emperor; the separation between shang and the preceding clause is further
clarified in text (a) below, where the ganzhi date yichou is in between
‘Wuche’ and ‘shang’; another parallel text (b) has yu for ‘at’ (Wuche). Thus,
the positional interpretation by HVP20 (table 2) is not justified from the
Chinese transmission: the alleged ‘above’ is interpreted for sightings during
‘evening’ or ‘morning’, leading to two error circles (with questionable
different radii, HVP20 table 3 and HV20a figure A-12). However, the text also
has ‘in the northeast’, omitted by Ho (1962), see below, so that the
observation was obviously performed at the end of the night.

Although HV20a (section 4) ‘believe’ that an ‘earlier eruption was detected by
ancient Far Eastern observers’, their ‘excellent candidate for a recurrent
nova’, namely V392 Per (also given as S1065, but in fact referring to S10653),
is not found in their figure A-12: in table 2, it is stated that V392 Per was
‘not visible at twilight’, while in section 4, they wrote that ‘the historical
nova … was only visible at twilight’, yet ‘a coincidence with a possible nova
in 668 CE is not completely excluded’. At the end of the reported range of
dates (‘22nd day’ is June 6/7), the position of their unjustified search
circle could be seen before astronomical twilight (already $\sim 10^{\circ}$
above horizon). HVP20 have a question mark for the duration in table 3, but it
is clearly given, see source (a) next.

Further references are mentioned already in Ho 1962 (no. 252), but parallel
transmissions were not accounted for by HVP20, HV20a,b – P+08 give additional
sources, both 668 May 18:
(b) ‘1st year of the Zongzhang reign period of Emperor Gaozong of the Tang
Dynasty, 4th month, day bingchen [53]; [there was] a broom star [hui xing]
[that] emerged at [or: in (yu)] Wuche …’ (Xin Tang shu 3.36-67: Gaozong ji, AD
1061, omissions below).
(a) ‘1st year of the Zongzhang reign period of Emperor Gaozong of the Tang
Dynasty, [summer,] 4th month, day bingchen [53]; [there was] a broom star [hui
xing] [that] appeared [or: was seen] between [zhi jian, lit.: in the space of]
Bi and Mao. … on day yihai [12 = June 6], the broom star was extinguished [or:
vanished]’ (Jiu Tang shu 5.91-92: Gaozong, AD 945, omissions below).

All texts are consistent regarding the duration: while (c) has only the lunar
month, (a) and (b) give a date, 4th month, ganzhi day bingchen [53], i.e.
night May 18/19, for first detection (not given in HVP20, HV20a,b);
disappearance is stated in (c) for the 22nd day of the 4th lunar month, which
is ganzhi day yihai [12] in the 60-day-cycle, recorded in (a), the night June
6/7 (19 nights).

The ‘space of Bi and Mao’ cover right ascension ranges, so-called lunar
mansions (LM): Mao is LM 18 from 7 or 17 Tau to $\epsilon$ Tau, Bi is LM 19
from $\epsilon$ Tau to $\phi$ or $\lambda$ Ori (e.g. SK97, p. 52). Mao and Bi
are also asterisms, Mao is identical with the Pleiades (7 stars), Bi with
$\alpha$ Tau, the Hyades, etc. (in total 8 stars). However, the wording jian
in combination with such entities (here: Bi and Mao) indicates lunar mansion
space, which often feature in comet reports (e.g. 1P/Halley AD 760 and 837,
e.g. Xu et al. 2000). We have now various information about the ‘broom star’:
first seen at the asterism Wuche, which lies (almost entirely) in LM Bi, but
not in LM Mao; and, also, since LM Bi (19) is reported before LM Mao (18),
probably not just by chance, we may conclude that the comet was flying from
East to West (LM 19 to 18). (Even if jian did point to ‘between the asterisms
Bi and Mao’, this position would be disjunct from ‘at Wuche’, and hence
showing motion.) Thus, the observations were at the end of the night – the
comet had just passed the Sun.

This scenario is supported by a further source from Korea (see P+08 (a), pp.
74, 519), interestingly also given under Ho no. 252, but not taken into
account by HVP20 nor HV20a,b.
668 May 17 - June 14, 4th lunar month, P+08 (a): ‘8th year of King Munmu of
Silla, 4th month; a broom star guarded [shou] Tianchuan’ (Samguk sagi, Jeungho
munheon bigo); text (b) in footnote 5.

Although this Korean chronicle is sometimes a few years off, date and position
fit well to the path given in China: since the eastern end of the asterism
Tianchuan (from west to east along $\eta,\gamma,\alpha,\delta,\mu$ Per, in
total 9 stars) is situated in LM Mao, and the verb shou, i.e. guard or linger,
means to ‘remain stationary within 1 to 2 du [$1-2^{\circ}$] of another body’
(P+08, p. 467), the report seems to give information where the ‘broom star’
slowed down and was seen last.

Next, we will discuss the text omitted in the above quotations (a) to (c) from
China. Although the positions for the ‘broom star’ differ, (a) has ‘in the
space of Bi and Mao’, (b) and (c) ‘at Wuche’, the omitted context in (a) and
(c) is almost identical, (b) is very shortened – this shows that these
transmissions refer to the same ‘broom star’. Parts of the omissions are given
in P+08 (c), but only (a) has a further date, so we choose it for our
translation (from JTS):
‘[Day] yichou [2 = May 27], the emperor avoided the main hall and reduced [his
consumption of] rich foods. [He] summoned … groups of officials [asking] each
to submit a memorial, … their words: “When the Star became fuzzy [xing
bo]333P+08: ‘the star subsequently became fuzzy’, but ‘subsequently’ is
implied, there is no character for it in the Chinese text., its bright rays
[guang mang, lit. also: light thorn/s] small/er, this is not an ill-omen for
the state and is insufficient [cause] for his majesty to labour [himself with]
sagely worries. Please sit [in] the main hall, restore the normal meals.” The
Emperor said: “I received [the duty to] offer [sacrifices] … [When a sign of]
blame is seen in the heavens, it admonished [me for] my lack of virtue …” The
many ministers again advanced saying (text c names Xu Jingzong, AD 592-672):
“The star became fuzzy [xing bo] in the northeast (text c adds: the royal
troups denounce the guilty) – this is a sign that Goryeo is about to be
destroyed.” The Emperor said: “The people of Goryeo are my people. When acting
as the lord of the myriad nations, how can I push [my] errors over a little
hedge.” In the end, he did not follow the requests made of him.’

On day yichou [2], May 27, the experts brief the emperor that the broom star
(detected on May 18/19), which they might have observed the nights before ‘in
the northeast’, now ‘became fuzzy’ and ‘its bright rays small/er’: evidently,
the tail of the comet (‘broom’) had lost already much of its length and/or
brightness, the head of the comet (‘star’) had only short, weak rays.444P+08,
p. 6: ‘In the past, some scholars have been perplexed by the compound xingbo,
which appears to defy Chinese grammatical conventions by having xing
‘star/celestial body’ modify bo ‘be fuzzy/bristle’. However, bo has a verbal
sense here, meaning ‘to become fuzzy or bushy’. This is entirely consistent
with cometary records where it is generally used to describe the appearance of
tail-less comets or the changed aspect … Rendering xing bo as “bushy [or
fuzzy] star” … is misleading in that it obscures the possibility that bo may
imply a change of appearance’ (P+08), see also Ho (1962, pp. 136-7) and
Stephenson & Green (2009, p. 31). The ‘24 most promising [non-cometary]
events’ in HVP20 contain at least five ‘broom stars’ plus five more which are
given as ‘fuzzy stars’; the terms ‘broom’ and ‘fuzzy’ are neglected in HVP20
(section 2.1) for discriminating between phenomena. This change of getting
smaller and fainter is interpreted to signal China’s loss of the kingdom
Goryeo in AD 668 thereby uniting the Korean peninsula under the Silla
dynasty.555P+08 text (b) from Korea (very much shortened from the Chinese
sources a and c) in the chronicle of the kingdom Goryeo: ‘27th year of King
Bojang (r. AD 642-668, d. 682) of Goguryeo [=Goryeo], summer, 4th month (668
May 17 - June 14); a broom star was seen between Bi and Mao’ from Jeungho
munheon bigo, which continues (our translation): ‘In Tang (i.e. China), Xu
Jingzong said: “a broom (sic) appears [in the] northeast, it is a sign that
Goguryeo is about to be destroyed” ’.

It is beyond the scope of this paper to survey further transmissions from
Europe etc.; it is known that Theophanes reported a ‘sign’ at this time
(Schove 1984) – unfortunately, it was not very specific, but comets are often
reported this way.

In sum, the transmitted details from East Asia fulfill all five comet criteria
which help to identify a likely true comet (D.L. Neuhäuser et al. 2020) – with
full right, the ‘broom star’ is listed in various catalogues of comets (e.g.,
Kronk 1999):
(i) Timing: observed at night-time or twilight – is given.
(ii) Position of first and/or last sighting: often close to Sun and/or near
ecliptic – is given.
(iii) Colour and form: the phenotypical description as hui xing indicates a
star with a tail (‘broom’), whose change (‘became fuzzy’ etc.) is also
mentioned.
(iv) Motion relative to stars: detection in/at Wuche (Aur) and probably last
sighting at Tianchuan (Per), together with probably first in LM Bi (19), then
in LM Mao (18), the comet path is given.
(v) duration: 19 nights – consistent with comets (if ‘extinguished’ (mie,
texts a and c) means that it was already too faint June 6/7).

As mentioned, HVP20 and HV20a,b use only the brief text from source (c), as in
Ho (1962) no. 252, quoted at the beginning of this section, with an erroneous
translation (‘above Wuche’) as well as presumably questionable duration
(HVP20, table 3) or unclear dating (‘only several days (maximum three weeks)’
in HV20b section 5.3, but see also their footnote 7). HV20b argue that the
phenomenon was ‘only visible in twilight’ (not correct) and, hence, ‘probably
rather bright’, which would ‘make[s] the observation of a SN more likely’ than
a nova666Nota bene: A very bright nova or SN appearing extended by strong
scintillation (therefore called ‘fuzzy’) can be an acceptable alternative only
in exceptions like day-time sighting and explicit mention of stationarity –
however, even SNe 1006 and 1054 were not described as ‘fuzzy’. – their
‘excellent candidate for a recurrent nova’, namely V392 Per (HV20a), is not
mentioned any more in the summarizing results (HV20b, table 9). Yet, HV20b
state that the short duration ‘would be atypical’ for a SN – a comet (sic) or
nova would be more likely. However, then HV20b suggests ‘a new hypothesis’,
because ‘something else appears really strange’: ‘In two subsequent years,
there are two comets (or novae) at almost the same position in the sky (close
to the Wuche-asterism)’; HV20b ‘consider it more likely that the two records
together report one supernova than that there were two comets at almost the
same position’.

Let us take a look at the text from China from the presumably preceeding year,
which is meant here – as translated in P+08 (pp. 73, 518), variants from Ho
(1962, no. 251) in round brackets:
AD 667 May 24: ‘2nd year of Qianfeng reign-period of Emperor Gaozong Tang
Dynasty, 4th month, day bingchen [53]. There was a broom star [hui xing] in
the northeast, between (Ho: among) Wuche, Bi [LM 19], and Mao [LM 18] [lit.:
it was located in/at [zai] Wuche, in the space [jian] Bi, Mao]; by day yihai
[12], it did not appear [or: was not seen]’ (Xin Tang shu 32.837, Tianwen
zhi).

This shortened transmission in the ‘New history of the Tang’ is likely
concatenated from the older Jiu Tang shu (texts a and c), or common sources.
This procedure is not unusual. Compilations work that way, copying errors
happen. The misdating was already noticed by P+08 (p. 73): there was no day
yihai, i.e. day 12 in the 60-day-cycle, in the 4th month for the given reign-
period, but one year later it did exist. 777Probably the occurrence was first
listed in Qianfeng, year 3 rather than Qianfeng, year 2: the reign-period
changed from Qianfeng to Zongzhang sometime in the third month of AD 668
(April 17 - May 16). If the record was originally dated to Qianfeng, year 3, a
copyist might not have realized that Qianfeng had ended by the fourth month,
and, rather than correctly changing the reign period, incorrectly changed the
year.

Since all other information given – name of emperor, day numbers, lunar month,
‘broom star’, position, path, duration (again 19 days, in HVP20 table 3 only
‘18’) – are identical, one can conclude that the text of AD 667 belongs
clearly to the transmitted sources for AD 668. In HVP20, this misdated doublet
is listed as one of the ‘24 most promising events’. In HV20b, table 9, the
best results are summarized: for AD 667 ‘appearance of a SN’; for AD 668
‘disappearance of SN 667 (consider SNR G160.9+02.6 (?))’. In the ‘discussion
of individual events’ (HV20b, section 5.3), their ‘new hypothesis’ postulates
the need to ‘re-interpret the position’ for AD 667: ‘it was originally not
meant to have occurred between three asterisms (which would be, indeed, a very
unusual description) but next to the Wuche asterism (as event 668)’, HV20b
mean their wrongly deduced position (‘above Wuche’). Thus, the ‘two separate
sightings’ are fabricated to pertain to ‘one supernova observation from May
667 to June 668 CE’ (HV20b, abstract) – but in fact almost none of the six
criteria for reliable historical SNe (given in Stephenson & Green 2005, pp.
218-219) are fulfilled.888‘Long duration’ (i), ‘fixed location’ (ii), ‘low
Galactic latitude’ (iii), ‘no evidence of significant angular extent’ (iv),
and ‘unusual brilliance’ (v) are construed for AD 667-8 in the
overinterpretation by HVP20 and HV20a,b, even (iii) is not that certain
regarding the true scenario, and ‘independent records’ (vi) are absent.

The unfounded speculations (one more column in HV20b section 5.3 is filled
with SNR considerations)999Earlier, Xi & Po (1966) suggested as SNR
counterpart for the AD 668 ‘broom star’ the radio source CTB-13, based on
partly incomplete sources and questionable translation, but the record for AD
667 is already omitted; Chu (1968) on AD 667 and 668: ‘Since these
constellations cover a wide area … it hardly seems to be the same localized
event’, observed ‘when the Sun is near to these constellations’, i.e.
‘probably a comet’ in May 668. These facts are not mentioned by HVP20 etc.,
but HV20a do cite both otherwise. The AD 667 record is not listed in
Stephenson (1976) on novae and SNe. For the records AD 667 and 668, Kronk
(1999) has one comet in AD 668. in HVP20 and HV20a,b can be seen as an example
for how transmissions about transients should not be studied: their approach
runs the risk of treating historical records as quarry, thereby introducing
unwarranted possibilities into the general astrophysical discourse on
historical novae (see D.L. Neuhäuser, R. Neuhäuser & Chapman 2018, R.
Neuhäuser, D.L. Neuhäuser, Posch 2020).

##  4 Summary

As one of two main highlights, HV20a,b feature a presumable new historical SN
from AD 667 to 668 from two ‘broom star’ records in these two years: the
former record, however, is a misdated doublet of the latter, and there are
additional Chinese and Korean accounts on the transient object in AD 668
(partly already in Ho 1962, fully in P+08), which provide timing and duration,
positions and path, and that the ‘broom star’ became fuzzy – all typical for
comets. Their 2nd highlight is the interpretation of a single brief Japanese
account on a ‘guest star’ in AD 891 as recurrent nova (U Sco), but also here
they neglect additional published records from China, Europe, and Arabia,
which together show that the Japanese account most likely just gives the last
position of Alfred’s comet.

In addition, their ‘24 most promising events’ as nova candidates (HVP20)
include several more ‘broom’ and ‘fuzzy stars’. E.g., for the five ‘broom
star’ events in HVP20 (tables 2 & 3), only the texts considered by them (from
Ho 1962) already fulfil most to all comet criteria (as listed in Sect. 3). If
certain information is missing, e.g. the comet path, one cannot conclude on
stationarity. While HVP20 claim to have eliminated comets (abstract: ‘we
selected those without movement and without tail (to exclude comets)’), there
are obviously many comets in their ‘representative shortlist’ (of 24 events).
HVP20 do not take into account the state-of-the-art (e.g. Stephenson & Green
2009 on ‘guest stars’) and important relevant literature (e.g., Kronk 1999 and
P+08 on comets). Their candidates, positions, and counterparts (HV20a,b)
should not be used for astronomical follow-up observations: even for the few
potential nova candidates among their list, positional search fields by others
(e.g., Stephenson & Green 2009) are more reliable.

Data Availability. All data are incorporated into the article.

## Acknowledgements

We would like to thank the anonymous referee for in-depth analysis, good
suggestions, and encouraging words.

## References

  * [1] Chapman J., Csikszentmihalyi M., Neuhäuser R., 2014, AN, 335, 964
  * [2] Chapman J., Neuhäuser D.L., Neuhäuser R., Csikszentmihalyi M., 2015, AN, 336, 530
  * [3] Chu S., 1968, JKAS, 1, 29
  * [4] Cook D., 1999, JHA, 30, 131
  * [5] Hasegawa I., 2002, PASJ, 54, 1091
  * [6] Ho P.Y., 1962, Vistas, 5, 127
  * [7] Hoffmann S.M., Vogt N., 2020a, MNRAS, 494, 5775 (HV20a)
  * [8] Hoffmann S.M., Vogt N., 2020b, MNRAS, 496, 4488 (HV20b)
  * [9] Hoffmann S.M., Vogt N., Protte P., 2020, AN, 341, 79 (HVP20)
  * [10] Kronk G.W., 1999, Cometography, vol. 1, Cambridge University Press, Cambridge
  * [11] Neuhäuser D.L., Neuhäuser R., Chapman J., 2018, AN, 339, 10
  * [12] Neuhäuser D.L., Neuhäuser R., Mugrauer M., Harrak A., Chapman J., 2020, Icarus, submitted
  * [13] Neuhäuser R., Neuhäuser D.L., 2015, AN, 336, 225
  * [14] Neuhäuser R., Rada W., Kunitzsch P., Neuhäuser D.L., 2016, JHA, 47, 359
  * [15] Neuhäuser R., Neuhäuser D.L., Rada W., Chapman J., Luge D., Kunitzsch P., 2017a, AN, 338, 8
  * [16] Neuhäuser R., Ehrig-Eggert C., Kunitzsch P., 2017b, AN, 338, 19
  * [17] Neuhäuser R., Neuhäuser D.L., Posch T., 2020, Terra-Astronomy – Understanding historical observations to study transient phenomena, in: Lago T. (Ed.) Astronomy in Focus, Cambridge University Press, pp. 145-147
  * [18] Pankenier D.W., Xu Z., Jiang Y., 2008, Archeoastronomy in East Asia, New York, Cambria (P+08)
  * [19] Rada W., Neuhäuser R., 2015, AN, 336, 249
  * [20] Schove D.J., 1984, Chronology of eclipses and comets AD 1-1000, St. Edmundsburry Suffulk
  * [21] Stephenson F.R., 1976, QJRAS, 17, 121
  * [22] Stephenson F.R., Green D.A., 2002, Historical supernovae and their remnants, Claredon, Oxford
  * [23] Stephenson F.R., Green D.A., 2005, JHA 36, 217
  * [24] Stephenson F.R., Green D.A., 2009, JHA, 40, 31
  * [25] Sun X.S., Kistemaker J., 1997, The Chinese sky during the Han: constellating the stars and society, Leiden, Brill (SK97)
  * [26] Whitelock D. (Ed.), 1979, English historical documents c. 500-1042, 2nd ed., Oxford University Press, New York
  * [27] Xi Z.Z., Po S.J., 1966, Science, 154, 597
  * [28] Xu Z., Jiang Y., Pankenier D.W., 2000, East Asian Archaeoastronomy: historical records of astronomical observations of China, Japan, and Korea, Gordon and Breach, Amsterdam


Over-parameterized deep neural networks are able to achieve excellent training accuracy while maintaining a small generalization error. It has also been found that they are able to fit arbitrary labels, and this behaviour is referred to as the phenomenon of memorization.
In this work, we study the phenomenon of memorization with turn-over dropout, an efficient method to estimate influence and memorization, for data with true labels (real data) and data with random labels (random data).
Our main findings are: (i) For both real data and random data, the optimization of easy examples (e.g., real data) and difficult examples (e.g., random data) are conducted by the network simultaneously, with easy ones at a higher speed; (ii) For real data, a correct difficult example in the training dataset is more informative than an easy one.
By showing the existence of memorization on random data and real data, we highlight the consistency between them regarding optimization and we emphasize the implication of memorization during optimization.

§ INTRODUCTION

Over-parameterized deep neural networks (DNNs), which have more number of trainable parameters than that of data instances, usually achieve excellent performance on both the training dataset and the test dataset.
However, it has also been found that a DNN has the capability to perfectly fit pure random labels, and this behaviour of a DNN when trained on random labels or random inputs has been implicitly referred to as the phenomenon of memorization [Zhang et al., 2017, Arpit et al., 2017, Brown et al., 2021, Maennel et al., 2020].
Understanding the memorization of deep neural networks is a key step towards characterizing its optimization dynamics and interpreting the model's generalization ability.

Zhang et al., 2017 formulated the memorization phenomenon by showing that a neural network with sufficient capacity can memorize completely random labels or random noise data (inputs) without substantially longer training time, and the explicit regularization is neither necessary nor sufficient to control the generalization error.
Arpit et al., 2017 further varied the amount of random labels, and concluded that DNNs first learn general patterns of the real data before fitting the random data.

The phenomenon of memorization can also be observed on a real dataset when there are no random labels.
Feldman and Zhang, 2020 studied memorization through the lens of self-influence estimation, and provided compelling evidences for the long tail theory [Feldman, 2020]. They believe that memorization is necessary for achieving close-to-optimal generalization error when the data distribution is long-tailed.
Modern datasets usually follows a long tail distribution, where the subpopulations at the head of the distribution include a large amount of frequent and easy examples while the subpopulations at the tail include rare and difficult examples.

However, the insights of these prior work normally are limited through some computationally intensive estimation.
For example, Arpit et al., 2017, Gu and Tresp, 2019 only study the memorization phenomenon on random data, and it is not clear if such observations can be generalized to real data.
Similarly, the study on influence estimation [Feldman and Zhang, 2020, Koh and Liang, 2017] focuses on the cross-influence of a training instance over a test instance, instead of self-influence nor its implication on optimization.
To this end, we introduce the memorization score (i.e. self-influence)—as a unified and computationally efficient metric—on both real data and random data to estimate the difficulty of training instances; such efficient estimation allows us to study the interpolation between memorization, optimization, and its implication on generalization.

Our main contributions are summarized as follows:

* We leverage turn-over dropout to quantify the degree of memorization by estimating its memorization score.
We show it is feasible and efficient on different network architectures.

* For real data and random data, easy and difficult examples are learnt simultaneously.
However, the optimization on easy examples converges faster than that of difficult ones, due to the higher gradient similarity between easy examples.

* For real data, difficult examples contain information about easy ones but easy examples have very little information about difficult ones. Easy examples can benefit from both easy and difficult examples, and hence the optimization speed of easy examples is faster.

§ INFLUENCE ESTIMATION WITH TURN-OVER DROPOUT

§.§ Preliminary

We denote $\cD = \{\xx_i, \yy_i\}_{i=1}^N$ as the training dataset, $\cX$ as the input space (e.g., images) and $\cY$ as output space (e.g., labels).
Let $\zz_i:=(\xx_i, \yy_i)$, where $\zz$ is a data example and represent a pair of input $\xx_i \in \cX$ and its output $\yy_i \in \cY$.
A model trained on dataset $\cD$ is denoted as $f_\cD: \cX \rightarrow \cY $.
The loss function $\cL(f, \zz_i)$ represents the loss of the model's prediction on $\xx_i$ over its label $\yy_i$.

The phenomenon of memorization and generalization can be quantified by the leave-one-out influence function [Feldman and Zhang, 2020, Koh and Liang, 2017], which measures the influence of example $\zz_i$ to the prediction of $\zz_{target}$.
Let $f_{\cD}$ be the model trained on the full training dataset $\cD$ and $f_{\cD \backslash \{\zz_{i}\}}$ be the model trained on $\cD$ excluding example $\zz_i$, then the influence is defined as:

\begin{equation} \label{eq: inter-influence}
\cI(\zz_{target}, \zz_{i} ; \cD) :=
\cL(f_{\cD \backslash \{\zz_{i}\}}, \zz_{target})- \cL(f_{\cD}, \zz_{target}) \,,
\end{equation}

where $\zz_{target}$ can be an instance in the training set or the test set.
When $\zz_{target} = \zz_{i} \in \cD$, memorization can be defined as self-influence [Feldman and Zhang, 2020]:

\begin{equation} \label{eq: self-influence}
\cI( \zz_{i}, \zz_{i} ; \cD) :=
\cL( f_{ \cD \backslash \{ \zz_{i}\}}, \zz_{i}) - \cL(f_{\cD}, \zz_{i}) \,.
\end{equation}

The memorization score computed by Equation (<ref>) indicates the extent of memorization.
A large memorization score means that $\zz_i$ is memorized to be fitted, as the model that is trained without $\zz_i$ cannot correctly classify $\zz_i$.
A small score indicates that the label of $\zz_i$ can be inferred from other training data, and hence $\zz_i$ is part of a generalizable pattern learnt by the model.

For a more fine-grained understanding, we define the cosine similarity of a set of gradient vectors $\{v_i\}, i \in [n]$ as the average pair-wise cosine similarities, as

\begin{align}
\text{cosine similarity} = \EEb{i, j}{
\frac{ \vv_{i} \cdot \vv_{j} }{ \norm{\vv_i}_2 \cdot \norm{\vv_j}_2 }
}, \;\; \forall i, j \in [n] \: and \: i \neq j \,.
\label{eq: cosine sim}
\end{align}

We further define the contribution of an example's gradient to the actual gradient update step as the projection of the example's gradient on the gradient of the mini-batch $\gg$, i.e.
\begin{align}
\text{contribution of } \vv_i =
(\vv_{i} \cdot \gg) / \norm{\gg}_2
\,.
\label{eq: contribution}
\end{align}

§.§ Turn-over Dropout as an Efficient Estimation for Memorization

It is computationally infeasible to calculate the exact leave-one-out influence and memorization scores through Equation (<ref>) and (<ref>), due to the significant overhead introduced by training the extra model from scratch for each training instance.

For the ease of getting a deeper understanding on the optimization and generalization (as our main contribution), we follow the idea of turn-over dropout [Kobayashi et al., 2020], which is stated as follows (and the workflow refers to Figure <ref>).
We also visualize some cases of easy and difficult examples identified by turn-over dropout in Appendix <ref>.

The turn-over dropout mechanism.
Turn-over dropout generates a deterministic mask for each training example $z$ and only updates the un-muted sub-network.
The muted sub-network is not updated with respect to $z$.
The influence estimation is the loss difference of the two sub-networks.
Cited and adapted from [Kobayashi et al., 2020].

The turn-over dropout generates a deterministic mask $\mm(\zz_i)$ for each training instance $\zz_i$, and the mask is fixed throughout training and evaluation. The masks are controlled to ensure no two training instances will share the same mask, and each mask splits the model into two sub-networks: the muted sub-network and the unmuted sub-network.

We use $f^{\tilde{\mm}(\zz_i)}$ and $f^{\mm(\zz_i)}$ to approximate $f_{\cD \backslash \{\zz_{i}\}}$ and $f_{\cD}$ respectively, given the fact that the muted sub-network $f^{\tilde{\mm}(\zz_i)}$ is updated by $\cD \backslash \{\zz_{i}\} $ and the un-muted sub-network $f^{\mm(\zz_i)}$ is updated by $\cD$.
Hence, the influence score in Equation (<ref>) can be estimated by

\begin{equation} \label{eq: turn-over inter-influence}
\cI(\zz_{target}, \zz_{i} ; \cD) \approx
\cL(f^{\tilde{\mm}(\zz_i)}, \zz_{target})- \cL(f^{\mm( \zz_i )}, \zz_{target}) \,,
\end{equation}

and the memorization (self-influence) score can be estimated by

\begin{equation} \label{eq: turn-over self-influence}
\cI(\zz_{i}, \zz_{i} ; \cD) \approx
\cL(f^{\tilde{\mm}(\zz_i)}, \zz_{i})- \cL(f^{\mm( \zz_i)}, \zz_{i}) \,.
\end{equation}

Note that the prior work [Kobayashi et al., 2020] uses turn-over dropout as an addition to DNNs to increase the model's interpretability.
With turn-over dropout only added to deep layers, their implementation does not sacrifice the model's performance too much.
Our work instead applies turn-over dropout to all layers and uses the adapted model as a proxy to estimate self-influence.
Furthermore, we extend the self-influence information to the training procedure and study its implication on optimization.

§ OBSERVATIONS AND INSIGHTS

§.§ Unify random data and real data by memorization scores

A commonly-used method to study memorization is to corrupt the training data by replacing part of true labels with random labels.
With this method, previous work [Zhang et al., 2017, Arpit et al., 2017, Maennel et al., 2020] focuses on comparing the differences between the scenario of label corruption and the scenario of no corruption to understand memorization.
On the contrary, we juxtapose the two scenarios and focus on comparing the easy and difficult examples to find what is in common regarding optimization.

[Uncorrupted MNIST.]

[MNIST corrupted with 20,000 random labels.]

Memorization scores of MNIST computed from a one-hidden layer MLP with turn-over dropout. (a) Data follows a long tail distribution. (b) Data with random labels (corrupted) have higher memorization scores than data with real labels (uncorrupted).

A corrupted dataset is a more difficult version of the uncorrupted one.
Figure <ref> shows the histogram of the memorization scores (computed from turn-over dropout) on uncorrupted MNIST and corrupted MNIST.
When there is no corruption, the scores follow a long tail distribution as described in Feldman, 2020 (c.f. Figure <ref>).
Corrupting an instance significantly increases its difficulty (memorization score): due to the random pattern/relationship between the image and the label, the model has no option but to memorize the instance.
Random examples in the corrupted scenario are the counterpart of the difficult examples in the uncorrupted scenario, as shown in Figure <ref>.
Hence, random data and real data are consistent with respect to memorization scores, and the methodology to study the properties of random data sheds light on difficult examples in the uncorrupted scenario.

§.§ Training on random labels

In this section, our investigation starts from the revisiting on the experiments of Arpit et al., 2017 by corrupting the dataset with different degree of corruption.

Easy examples are learnt faster.
In contrast to the common belief introduced in Arpit et al., 2017[
We also reproduce these experiments in Appendix <ref>.
] that a model first learns the simple and general patterns (easy examples) before fitting the noise (difficult examples), we provide a refined insight below: easy and difficult examples are learnt by the model simultaneously, with easy examples learnt faster.
As indicated in Section <ref>, in the presence of label corruption, data with random labels are of the highest difficulty.

Figure <ref> shows the training and test dynamics of a one-hidden-layer MLP on the MNIST dataset with 20,000 instances replaced by random labels.
We keep track of training dynamics of an easy subset and a difficult subset of the training data.
It can be observed that easy examples are learnt much faster than difficult ones. The build-up of the speed difference reflects on the dynamic of test accuracy: the test accuracy reaches the peak (at around 20th epoch), and that is when the model has successfully fitted the real data but barely fitted the random noise yet.

However, we also observe from Figure <ref> (and Figure <ref> later) that there are small improvement on the difficult subset at the initial epochs. Albeit the improvement is small, it indicates that difficult examples are in fact learnt simultaneously with easy examples, and Figure <ref> in the next section shows stronger proof on this point.

[Training accuracy.]

[Test accuracy.]

The training and test dynamics of a one-hidden-layer MLP on the MNIST dataset when 20,000 of the training dataset is corrupted by random labels. Results are sliding-window smoothed.
The black line (“full”) in (a) and the line in (b) shows the dynamics on the entire training/test dataset respectively.
(a) also keeps track of the training accuracy of a 10,000 (16.67%) real-data subset (blue) and a 10,000 (16.67%) random-label subset (red). The percentage in the legend indicates the size of the subset.
(b) shows that the test accuracy reaches the peak with the build-up of the speed difference and drops with the random data improving its training accuracy.

Why are easy examples learnt faster than difficult examples?
Below we try to answer the question from the perspective of optimization (SGD updates) of real data and random data[
We use Backpack [Dangel et al., 2020] to extract the gradient of each example in the batch during the backward pass.
Figure <ref> shows the evolution of the average cosine similarity (Equation (<ref>)) and the average contribution (Equation (<ref>)) of easy examples and difficult examples.

[Cosine similarity]

Dynamics of cosine similarity and contribution of an easy (real) subset and a difficult (random) subset of the training data (MLP trained on MNIST with 20,000 instances corrupted). (a) The cosine similarity between easy examples is much higher. (b) The difficult examples are the driving force of SGD update throughout training.

The cosine similarity between easy examples is much higher than that of difficult ones, as shown in Figure <ref>.
Easy examples reside at the head of the long tail distribution and belong to similar subpopulations, and this similarity between images engenders similar gradients. These similar gradients strengthen each other to make a SGD update that can benefit them all, which results in the faster optimization speed for easy examples.
This observation is in agreement with [Chatterjee, 2020], who uses a measure of gradient coherence to show that the parameter update of the network can benefit many examples when such similarity exists.

Additionally, we observe from Figure <ref> that the average contribution of difficult examples is constantly higher than easy examples and the entire dataset.
The high contribution of difficult examples throughout training shows that difficult examples are learnt simultaneously with easy examples, in contrast to the prior belief that DNNs prioritize the learning of easy examples.

§.§ Training on real data

[Training accuracy on MNIST.]

[Test accuracy on MNIST.]

[Training accuracy on CIFAR-10.]

[Test accuracy, CIFAR-10]

Training and test dynamics on easy and difficult subsets of MNIST (with MLP) and CIFAR-10 (with VGG-11). The memorization scores are used to partition the dataset and the percentage in parentheses indicates the size of the subset. Difficult examples (red) is learnt simultaneously with easy examples. As a control, the dotted line in (a) and (b) indicates training on the training data with the difficult subset removed. Thus, we confirm that the improvement of difficult examples' training accuracy is not a side effect of the patterns learnt from easy examples.

Training dynamics on real data.
Figure <ref> shows how the easy/difficult/full subsets of the training/test data evolve with respect to epochs on MNIST and CIFAR-10.
We compute the memorization scores of the training data and test data beforehand, and monitor an easy subset and a difficult subset during training and validation.
The observation is consistent with the random-label case: easy examples and difficult examples are learnt simultaneously, with easy examples fitted faster than difficult ones.

Figure <ref> and <ref> shows the test dynamics are similar to the training dynamics: easy examples benefit from learning quicker than difficult ones and reach a higher test accuracy. This similarity reflects the consistency of the subpopulations identified.

Difficult examples are more informative.
As explained in the random label setting (Section <ref>), easy examples are learnt faster than difficult ones due to the higher gradient similarity of easy examples.
Complementing this observation, we further highlight a new insight in Figure <ref> below for real data, which again explains the faster convergence on the easy examples, that is difficult examples contain information about the easy ones while the easy examples have very little information about the difficult ones.

Figure <ref> presents the training dynamics of two manually-constructed sub-datasets: with the memorization scores computed for the training data, we sample an easy subset and a difficult subset that are of the same size from the whole training dataset; and for each time, we train with one subset, and use the other one as the validation set.
We can witness from Figure <ref> that training on the easy subset enables a fast convergence on the corresponding training accuracy, but a marginal improvement on that of the difficult subset.
While in Figure <ref>, despite the relatively slow convergence on the difficult subset, the training benefits both the difficult subset and the easy subset substantially.
It proves that the memorization of difficult examples improves the model's ability to generalize on easy examples, while the generalizable patterns learnt from easy examples do not transfer to difficult examples.

A difficult but correct example contains some (relatively weak) degree of similarity with the easy examples, and it results in a slow but still increasing training accuracy on the easy subset when the model is trained on the difficult subset.
A difficult subset contains more (atypical and exclusive) subpopulations than an easy subset and thus is more diverse, so a model that is trained on an easy subset is very unlikely to acquire this unique information.
Therefore, we conclude that difficult examples are more informative and to memorize difficult examples accelerates and benefits the model's ability to generalize.

[Train on an easy subset.]

[Train on a difficult subset.]

Training dynamics on two disjoint subsets of CIFAR-10 with VGG-11.  (a) shows the training accuracy when trained only with the easy subset. (b) shows the training accuracy when trained only with the difficult subset. Difficult examples contain information about the easy ones while the easy examples has very little information about the difficult ones.

§ CONCLUSION

Our work studies memorization regarding optimization in the context of random data and real data. We use turn-over dropout to efficiently evaluate memorization scores of training examples, and we unify random data with real data by showing that examples with random labels are simply extremely difficult examples.
We find there are qualitative differences between the easy and difficult examples, and it is consistent for random data and real data that easy and difficult examples are learnt simultaneously, with difficult ones learnt slower.
On real data, we also conclude that difficult examples contain more information than easy ones, and to memorize difficult examples may improve the model's ability to generalize, which may inspire core-set selection or more efficient training algorithms as future work.

§ ACKNOWLEDGEMENTS

We acknowledge funding from a Google Focused Research Award, Facebook, and European Horizon 2020 FET Proactive Project DIGIPREDICT.

[Arpit et al., 2017]
Devansh Arpit, Stanisław Jastrzębski, Nicolas Ballas, David Krueger,
Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron
Courville, Yoshua Bengio, et al.
A closer look at memorization in deep networks.
In International Conference on Machine Learning, pages
233–242. PMLR, 2017.

[Brown et al., 2021]
Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar.
When is memorization of irrelevant training data necessary for
high-accuracy learning?
In Proceedings of the 53rd Annual ACM SIGACT Symposium on
Theory of Computing, pages 123–132, 2021.

[Chatterjee, 2020]
Satrajit Chatterjee.
Coherent gradients: An approach to understanding generalization in
gradient descent-based optimization.
In ICLR, 2020.

[Dangel et al., 2020]
Felix Dangel, Frederik Kunstner, and Philipp Hennig.
Backpack: Packing more into backprop.
In ICLR, 2020.

[Deng, 2012]
Li Deng.
The mnist database of handwritten digit images for machine learning
IEEE Signal Processing Magazine, 290 (6):0
141–142, 2012.

[Feldman, 2020]
Vitaly Feldman.
Does learning require memorization? a short tale about a long tail.
In Proceedings of the 52nd Annual ACM SIGACT Symposium on
Theory of Computing, pages 954–959, 2020.

[Feldman and Zhang, 2020]
Vitaly Feldman and Chiyuan Zhang.
What neural networks memorize and why: Discovering the long tail via
influence estimation.
In NeurIPS, 2020.

[Gu and Tresp, 2019]
Jindong Gu and Volker Tresp.
Neural network memorization dissection, 2019.

[Hassani et al., 2021]
Ali Hassani, Steven Walton, Nikhil Shah, Abulikemu Abuduweili, Jiachen Li, and
Humphrey Shi.
Escaping the big data paradigm with compact transformers.
arXiv preprint arXiv:2104.05704, 2021.

[Kobayashi et al., 2020]
Sosuke Kobayashi, Sho Yokoi, Jun Suzuki, and Kentaro Inui.
Efficient estimation of influence of a training instance.

[Koh and Liang, 2017]
Pang Wei Koh and Percy Liang.
Understanding black-box predictions via influence functions.
In International Conference on Machine Learning, pages
1885–1894. PMLR, 2017.

[Krizhevsky et al., 2009]
Alex Krizhevsky, Geoffrey Hinton, et al.
Learning multiple layers of features from tiny images.

[Maennel et al., 2020]
Hartmut Maennel, Ibrahim Alabdulmohsin, Ilya Tolstikhin, Robert JN Baldock,
Olivier Bousquet, Sylvain Gelly, and Daniel Keysers.
What do neural networks learn when trained with random labels?
In NeurIPS, 2020.

[Simonyan and Zisserman, 2015]
Karen Simonyan and Andrew Zisserman.
Very deep convolutional networks for large-scale image recognition,

[Zhang et al., 2017]
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.
Understanding deep learning requires rethinking generalization.
In ICLR, 2017.

§ EXPERIMENT SET-UP

For MNIST ([Deng, 2012]), we use a one-hidden layer MLP with 4096 neurons in the hidden layer trained with a 0.06 learning rate on 200 epochs and no momentum, except for Figure <ref>, in which we switch to a learning rate of 0.01 for clearer result.
For CIFAR-10 ([Krizhevsky et al., 2009]), we use a VGG-11 network ([Simonyan and Zisserman, 2015]) without batch-norm, and it is trained on 200 epochs, with the learning rate starts at 0.01 (with 0.9 momentum) and divided by 10 at epoch 100 and epoch 150 respectively. Additionally, spatial turn-over dropout is applied to VGG-11 on its convolutional layers. The batch size is 256 for training and all the models use the ReLU activation.

§ IMPLEMENTATION OF TURN-OVER DROPOUT

For linear layers, turn-over dropout can be efficiently implemented to mask the neurons for two sub-networks.
For convolutional layers, 2D spatial turn-over dropout is implemented, which deterministically masks filters (feature maps) instead of neurons.
Half of the feature maps will be set to zero, and the other half will be passed to the next layer.

§ EASY AND DIFFICULT EXAMPLES IDENTIFIED BY TURN-OVER DROPOUT

Figure <ref> and Figure <ref> show the easy (low memorization score) and difficult (high memorization score) examples that are identified with turn-over dropout on MNIST and CIFAR-10 respectively. The easy examples do follow a typical pattern and shares a lot of similarities, and both the unmuted sub-network and the muted sub-network agree with the ground truth. The difficult examples seem to have some peculiar traits.

[Low memorization score examples on MNIST dataset.]
[High memorization score examples on MNIST dataset.]
Low and high memorization score examples on MNIST dataset. Each column corresponds to one class.
The number above the image represents the memorization score and the triplet above the image represents (ground truth label, unmuted sub-network's prediction, muted sub-network's prediction).

[Low memorization score examples on CIFAR-10 dataset.]
[High memorization score examples on CIFAR-10 dataset.]
Low and high memorization score examples on CIFAR-10 dataset.
Each column corresponds to one class.
The number above the image represents the memorization score and the triplet above the image represents (ground truth label, unmuted sub-network's prediction, muted sub-network's prediction).

§ EARLY STOP

[Fashion MNIST.]

Training accuracy of the muted and unmuted sub-networks in a model with turn-over dropout. (a) a one-hidden-layer MLP trained on MNIST with turn-over dropout. (b) a compact vision transformer [Hassani et al., 2021] trained on fashion MNIST with turn-over dropout.

Figure <ref> shows the training accuracy of the muted and unmuted sub-networks in a model with turn-over dropout. Both the muted and unmuted sub-networks converge very quickly within the first a few epochs and the stabilize. This motivates us to train the model with turn-over dropout for only a few epochs to calculate the memorization scores.

§ TRAINING AND TEST DYNAMICS OF DIFFERENT DEGREE OF CORRUPTION

Figure <ref> shows the training and test dynamics of a VGG-11 network trained on CIFAR-10, of which the labels of the training dataset are corrupted by different extent and the test dataset remains unchanged.

It can be observed that there are qualitative differences between the learning curves of different degree of corruption, which indicates that DNNs conduct memorization in a data-aware fashion.
There are two main observations:

* Difficult instances (random noises among the labels) make the learning process slower.
The more difficult examples, the longer it takes to fully fit the dataset.

* When there exists noise in the dataset, the test accuracy firstly reaches to a peak before it drops and converges.

[Training accuracy.]

[Test accuracy.]

Training and test dynamics of VGG-11 network on CIFAR-10 where the training dataset is corrupted by different extent and the test dataset remains unchanged.
The legend shows the number of examples with random labels.
It should be mentioned that since we use a multi-step learning rate decay schedule instead of a cosine annealing schedule, the model becomes incompetent to fit the dataset when all labels are replaced by random noise. This also proves that the extent and capability of memorization is influenced by the training procedure as well.