Long Secondary Periods in Pulsating Red Giants: A Century of Investigation

John R. Percy
Department of Astronomy and Astrophysics, and
Dunlap Institute of Astronomy and Astrophysics
University of Toronto
Toronto ON
Canada M5S 3H4

Email<EMAIL_ADDRESS>

Abstract Red giants (RGs) are unstable to radial pulsation. About a third of
them also show a long secondary period (LSP), 5-10 times the pulsation period.
The LSPs were recently ascribed to eclipses of the RG by a low-mass dust-
enshrouded companion. LSPs have been known for over a century. In this paper,
I use primarily AAVSO visual and photoelectric observations to look for
evidence of LSPs in 103 red giant stars listed by Nancy Houk in 1963 as having
LSPs, based mostly on photographic photometry. I have determined LSPs in 37
stars, and upper limits (some of them not very stringent) in 25. In the
former, the ratio of LSP to pulsation period peaks strongly at 10, which
suggests that most of the stars are pulsating in the first overtone. The LSPs
are consistent with Houk’s in 33 of the 37 stars. I have identified 16 stars
as bimodal pulsators; their period ratios are consistent with previous
observational and theoretical results. For 14 stars, the periods in the
General Catalogue of Variable Stars are incorrect or absent.

ADS keywords = stars; stars: late-type; techniques: photometric; methods:
statistical; stars: variable; stars: oscillations; clusters, globular

1\. Introduction

Red giants (RGs) are unstable to pulsation in one or more radial modes. In
general, the larger and brighter the star, the longer the pulsation period (P)
and the larger the amplitude (A). About a third of RGs also have a long
secondary period (LSP), 5 to 10 times longer than the dominant pulsation
period, depending on whether the pulsation period is the fundamental or first
overtone (e.g. Wood 2000). Until recently, the cause of the LSP was uncertain,
but Soszyński et al. (2021) have made a strong case for a binary mechanism:
the companion was initially a planet, which later accreted material from the
RG wind, and grew into a brown dwarf or low-mass star, which produces the
observed LSP velocity variations in the RG. The dust-enshrouded companion
eclipses the RG periodically, producing the LSP.

LSPs have been known for almost a century; a series of three papers helped to
bring them to attention (O’Connell 1933, Payne-Gaposchkin 1954, and Houk
1963). These papers use photographic photometry, some of it extending back to
the late 19th century. O’Connell (1933) pointed out that the ratio of LSP/P
was about 10, but only four of his stars were actually red giants, and his
result depends strongly on one star – V Hya. Payne-Gaposchkin (1954) collected
information on many more red giants, of various types, (but even included RR
Lyrae stars showing the Blazhko Effect, which is a totally different
phenomenon). Houk (1963), in her Table III, lists 103 PRGs with LSPs.

I was interested in how effective the photographic photometry was in
identifying and measuring LSPs, and whether the LSPs would stand up with time.
Therefore, in the present paper, I use visual v and photoelectric V
observations from the American Association of Variable Star Observers (AAVSO)
International Database, and V observations from the All-Sky Automated Survey
for Supernovae (ASAS-SN) to investigate the primary and secondary periods, and
to compare them with the periods listed in Houk (1963). Previous studies of
LSPs in RGs have been carried out on selected stars, using the same databases.
These include Percy and Deibert (2016) and Percy and Leung (2017), who studied
RGs with long and dense AAVSO data, and Percy and Fenaux (2019), who carried
out a pilot study of RGs using the much-shorter ASAS-SN data.

2\. Data and Analysis

For the 103 stars listed in Table III of Houk (1963), I analyzed visual (v)
and photoelectric (V) data from the AAVSO International Database (AID), and V
data from the ASAS-SN variable star database and catalog (Shappee et al. 2014,
Jayasinghe et al. 2018, 2019), using light-curve analysis and time-series
analysis using the AAVSO VSTAR software package (Benn 2013). Note that VSTAR
provides the semi-amplitude SA, not the full amplitude or range. The ASAS-SN
data and light curves are freely available on-line (asas-sn.osu.edu/variables)
as are the AID data and light curves, and the VSTAR package (aavso.org). The
AAVSO v observations are limited by their low precision – typically 0.2 to 0.3
mag – but are very numerous for many of the stars. The AAVSO V observations
tend to be sparse. The ASAS-SN observations extend over only a few years.

In some cases, a strategy which was helpful for determining the LSP was to bin
the visual observations, in bins equal in length to the pulsation period. This
strategy was less-useful when the SA was less than 0.1 mag, or when the star
was a bimodal pulsator, or when the data were sparse and/or had unusually long
seasonal or other gaps.

3\. Results

Table 1 lists the 37 stars for which I found good evidence for an LSP; Table 2
lists 25 stars for which I found some evidence that an LSP is not present; we
include a rough upper limit to the semi-amplitude that I could detect. In some
cases, the upper limit was too large to be of much significance. Table 3 lists
16 stars which may be bimodal. Table 4 lists 15 stars for which the period in
the General Catalogue of Variable Stars (Samus et al. 2017) is probably not
correct, or is absent. Note that, for some stars in the sample, the GCVS
period is the LSP, not the pulsation period.

For the following stars, I obtained no significant results about possible
bimodal pulsation, or about LSPs, but there may be other useful information in
the Notes, in the Appendix: SS And, V Aql, RX Boo, ST Cam, RS Cap, BZ Car,
$\mu$ Cep, CV Cep, T Crt, RV Cyg, CH Cyg, V539 Cyg, EI Del, S Dra, UU Dra, V
Eri, SU Hya, U Lac, WY Lac, KP Lyr, Y Mic, RV Mon, SX Mon, W Nor, TW Oph, V521
Oph, V574 Oph, RT Ori, V Pav, RT Pav, TT Per, BU Per, SY Vel.

4\. Discussion

The stars in Table 1 have LSP SAs which tend to be small; for almost half,
they are 0.1 mag or less. This is consistent with previous results in which
the SAs are 0.5 mag or less. Three stars stand out, however: V Hya, with
visual SA = 1.11, BU Lac, with visual SA = 0.88, and SV And with visual SA =
1.98. V Hya is a famous variable in which the deep minima are believed to be
due to obscuration by a cloud associated with a companion passing in front of
the red giant – an extreme example of the process proposed by Soszyński et al.
(2021). SV And also has deep minima; my data are extensive enough to show that
the amplitude of the LSP does not vary significantly over 100 years. It would
be interesting to see whether the SA correlated with any other properties of
the star.

The ratios LSP/P peak strongly at 9-10, which suggests that most of the stars
pulsate in the first overtone. This is to be expected, since most of the stars
in the table are shorter-period pulsators, rather than large-amplitude Mira
stars which tend to pulsate in the fundamental mode.

In 33 of the 37 stars in Table 1, my LSP is consistent with Houk’s. The ones
that disagree are SV And, S Dra, RY Lac, and TW Oph.

There are a few stars for which I could find LSPs, but not pulsation periods.
According to the Soszyński et al. (2021) hypothesis, there is no reason why a
non-pulsating star could not have an LSP, though there would have to be some
mechanism by which the companion could become dust-shrouded.

Table 2 lists stars which had adequate data, but in which I was not able to
identify an LSP. For some of them, the upper limit to the SA was well below
0.1 mag. For others, the upper limit was sufficiently high that I cannot rule
out an LSP.

Table 3 lists bimodal stars, and their pulsation period ratios. These cluster
around 0.5, and are consistent with previous observed (Percy and Huang 2015)
and theoretical values. There is more scatter than found by Percy and Huang
(2015), but the trend of decreasing P(short)/P(long) with increasing P(long)
is present in our data,

One star had a second period which did not fit either the LSP or the bimodal
classification: RW Psc with periods of 65 and 154 days. The former could be a
second overtone and the latter a fundamental mode.

Note that Houk (1963) did not include possible bimodal pulsators in her list
of stars – with one exception: RV Cyg, for which she proposed periods of 300
and 470 days. I was not able to obtain results on this star.

5\. Conclusions

Of the 103 red giants listed by Houk (1963) as possibly having LSPs, I have
found evidence for an LSP in 37 of them. In 33, the LSP that I have found is
consistent with the value given by Houk (1963). For a few other stars, I have
put stringent upper limits on the the amplitude of any LSP. For the rest, I do
not have results, or cannot rule out an LSP. I conclude that the photographic
photometrists of the past were rather successful in detecting and measuring
LSPs in pulsating red giants, even when the amplitudes were small.

There are still many unanswered questions about this phenomenon, which occurs
in a large fraction of RGs. The Soszyński et al. (2021) mechanism for
producing LSPs is a fascinating and important one, and suggests that studies
of LSPs should continue. A combination of long-term photographic and visual
photometry, along with more modern techniques has the potential to answer some
of the unanswered questions.

Acknowledgements

This paper is based primarily on data from the AAVSO International Database. I
am grateful to the observers who made the observations, and the AAVSO staff
who archived them. I also thank those who created and who have maintained the
VSTAR time-series analysis package which was used throughout this paper. I
also made use of ASAS-SN photometric data; I thank the ASAS-SN project team
for this remarkable contribution to stellar astronomy, and for making the data
freely available on-line. The Dunlap Institute is funded through an endowment
established by the David Dunlap Family and the University of Toronto.

References

Benn, D. 2013, VSTAR data analysis software (http://www.aavso.org/node/803)

Houk, N. 1963, Astron. J., 68, 253.

Jayasinghe, T. et al. 2018, Mon. Not. Roy. Astron. Soc., 477, 3145.

Jayasinghe, T. et al. 2019, Mon. Not. Roy. Astron. Soc., 486, 1907.

O’Connell, D.J.K. 1933, Harvard College Observatory Bulletin, 893, 19.

Payne-Gaposchkin, C. 1954, Ann. Harvard College Observatory, 113, 189.

Percy, J.R. and Huang, D.J. 2015, J. Amer. Assoc. Var. Star Obs., 43, 118.

Percy, J.R. and Deibert, E. 2016, J. Amer. Assoc. Var. Star Obs. 44, 94.

Percy, J.R. and Leung, H. W.-H. 2017, J. Amer. Assoc. Var. Star Obs., 45, 30.

Percy, J.R. and Fenaux, L. 2019, J. Amer. Assoc. Var. Star Obs., 47, 202.

Samus, N.N. et al. 2017, Astron. Reports, 61, 80.

Shappee, B.J. et al. 2014, Astrophys. J., 788, 48.

Soszyński, I., et al. 2021, Astrophys.J. Lett., 911, L22.

Table 1: Stars with long secondary periods. The columns list: the star name, the periods given by Houk (1963), the pulsation period P, the LSP, the LSP SA, and LSP/P – the last four determined in the present paper. Star | Houk Periods (d) | P (d) | LSP (d) | SA(LSP) | LSP/P
---|---|---|---|---|---
SV And | 315.96, 930 | 314 | 2283 | 0.84 | 7.3
TW Aur | 150:, 1370 | 144 | 1353 | 0.18 | 9.4
U Cam | 400:, 3000 | 215 | 3000$\pm$200 | 0.13 | 14.0
RS Cam | 88.6, 960 | 90.5 | 999 0.15 | 11.0 |
RS Cnc | 120, 1700 | 239 | 2050$\pm$100 | 0.07 | 8.6
RT Cnc | 90$\pm$, 542 | 93$\pm$3 | 650$\pm$100 | 0.06 | 7.0
Y CVn | 158, 2000-2200 | 160 | 2009 | 0.06 | 12.6
TZ Car | 69, 1183 | 70: | 1213 | 0.25 | 17.3
VY Cas | 100, 600 | 93.4 | 800$\pm$200 | 0.09 | 8.6
AA Cas | 115, 850 | 75$\pm$5 | 870 | 0.05 | 11.6
SS Cep | 90, 950 | 101.1 | 955 | 0.11 | 9.4
RS CrB | 69.5, 331 | 70 | 332 | 0.24 | 4.7
AF Cyg | 94.1, 960 | 93 | 913: | 0.08 | 9.8
AW Cyg | 220, 2200 | 205: | 2290: | 0.08 | 11.2
S Dra | 136, 1319 | 181.7 | 1000 | 0.08 | 5.5
TX Dra | 78, 654 | 77 | 700$\pm$15 | 0.14 | 9.1
Z Eri | 80, 746 | 78 | 710$\pm$20 | 0.10 | 9.1
X Her | 95.0, 746 | 100 | 700 | 0.08 | 7.0
30 g Her | 70$\pm$, 900 | 88 | 875 | 0.16 | 9.9
V Hya | 533, 6500 | 531 | 6500$\pm$500 | 1.11 | 12.2
RV Lac | 67, 550-700 | 70 | 625$\pm$8 | 0.32 | 8.9
RY Lac | 38-91, 300-420 | 80 | 700$\pm$100 | 0.05 | 8.8
BU Lac | 200, 2000 | 250: | 2200$\pm$200 | 0.88 | 8.8
S Lep | 90, 875-900 | 90: | 867$\pm$10 | 0.23 | 9.6
T Mus | 93, 1021 | 93: | 1020 | 0.23 | 11.6
TW Oph | 165:, 2000 | 376.1 | 3225 | 0.23 | 8.6
W Ori | 212, 2450 | 211 | 2400$\pm$50 | 0.20 | 11.4
RX Peg | 110, 629 | 110 | 650 | 0.12 | 5.9
TW Peg | 90, 956.4 | 100 | 952$\pm$4 | 0.11 | 9.5
rho Per | 33-55, 1100 | 54.65 | 1000$\pm$300 | 0.04 | 18.3
T Per | 326, 2800 | 340 | 2500$\pm$500 | 0.06 | 7.4
U Per | 320.63, 2500 | 320 | 2500$\pm$100 | 0.31 | 7.8
UZ Per | 91, 927 | 90 | 895 | 0.29 | 9.9
RT Psc | 70, 533 | 70: | 515$\pm$5 | 0.08 | 7.4
Y Tau | 240.9, 1750 | 245 | 1733 | 0.04 | 7.1
ST UMa | 81, 590 | 90 | 610$\pm$10 | 0.06 | 6.8
V UMi | 72.0, 760 | 72 | 750$\pm$20 | 0.08 | 10.4
Table 2: Stars possibly lacking long secondary periods. The columns list: the star name, the periods given by Houk (1963), the pulsation period P, in days, and an estimate of the maximum semi-amplitude of the LSP – the last two determined in the present paper. Star | Houk Periods | P (d) | A(max)
---|---|---|---
SS And | 152, 650 | 152: | 0.15
TZ And | 118, 974 | 114.7 | …
UX And | 400$\pm$, 7000 | 402 | 0.10
V Aql | 353, 2270 | 362 | 0.05
UU Aur | 235, 3500 | 445 | 0.07
RX Boo | 78:, 500 | 160 | 0.07
ST Cam | 195, 2100 | 201 | 0.09
RS Cap | 340, 3360 | 130 | 0.15
WZ Cas | 186.0, 4000 | 192 | 0.04
CQ Cas | 110-250, 2300 | 204, 367 | 0.15
CV Cep | 60, 2000 | 60: | 0.18
RR CrB | 60.8, 377 | 55 | 0.04
AW Cyg | 220, 2200 | 204 | 0.04
CH Cyg | 97, 4700 | — | 0.30
V Eri | 97, 1209 | 300 | 0.30
Y Hya | 95, 302.8 | 363 | 0.10
KP Lyr | 146, 1300 | 140 | 0.13
Y Mic | 182, 4650 | 181 | 0.20
SX Mon | 100$\pm$, 1100 | 100: | 0.10
W Nor | 134.7, 1300 | 147 | 0.20
BQ Ori | 110, 795 | 247 | 0.08
Y Per | 252.3, 2400 | 253 | 0.03
BU Per | 365, 2950 | — | 0.10
Z UMa | 198, 1560 | 190 | 0.04
TZ Vir | 134, 6900 | 168 | 0.04
Table 3: Stars which are bimodal. The columns list: the star name, the longer pulsation period Pa in days, the shorter pulsation period Pb in days, and the ratio Pb/Pa. Star | Pa (d) | Pb (d) | Pb/Pa
---|---|---|---
UX And | 402 | 218 | 0.54
UU Aur | 445 | 235 | 0.53
RS Cam | 159 | 89 | 0.56
U Cam | 400 | 215 | 0.54
WZ Cas | 370 | 192: | 0.52:
CQ Cas | 367 | 204 | 0.56
SS Cep | 179 | 100 | 0.56
AF Cyg | 174 | 93 | 0.53
RS Dra | 276 | 142 | 0.51
TX Dra | 134 | 77 | 0.57
RY Lac | 80 | 46.5 | 0.58
BQ Ori | 247 | 120: | 0.49:
TW Oph | 376 | 172 | 0.46
W Ori | 317 | 211 | 0.67
RW Psc | 154 | 65 | 0.42
V UMi | 123 | 72 | 0.59
Table 4: Stars with missing or inaccurate GCVS periods. The columns list: the star name, the GCVS period in days, and the suggested period, based on the present analysis. Star | GCVS period (d) | suggested period (d)
---|---|---
TZ And | none | 114.7
TW Aur | 104 | 144
RS Cnc | 229.155 | 239
Y CVn | 3000 | 160
TZ Car | none | 70
VY Cas | 116 | 93.4
AA Cas | none | 75
SS Cep | 340 | 100
CV Cep | none | 60
V539 Cyg | 160 | not 160
X Her | 95.0 | 101.5
RY Lac | none | 80
BQ Ori | 110 | 247
ST UMa | 110 | 90
TZ Vir | 134 | 168

Appendix: Notes on Individual Stars

The following are notes on individual stars. The symbol v indicates results
from AAVSO visual observations; V indicates results from AAVSO photoelectric
observations; ASAS-SN indicates results from the All-Sky Automated Survey for
Supernovae; PD and PL indicate results from Percy and Deibert (2016) and Percy
and Leung (2017), respectively; GCVS indicates a period from the General
Catalogue of Variable Stars (Samus et al. 2017); numbers in parentheses are
the semi-amplitudes of the periods given; and a colon denotes uncertainty.

SV And: Houk: 316d, 930d. v: 314.96d (1.98); no LSP present. ASAS-SN: 313.0d.

TZ And: Houk: 118d, 974d. v: 114.7d (0.07); no LSP. ASAS-SN: 120:d plus very
slow variation; no evidence for the 974d period. PD: 114.8d, 1355.1d.

UX And: Houk: 400$\pm$d, 7000d. v: 219.5d (0.19), 7000-9000d (0.16). V: 215.8d
(0.19), 401.8d (0.36). ASAS-SN: $\sim$370d. Bimodal?

V Aql: Houk: 353d, 2270d. v: 215.9d (0.06), 362.4d (0.09), LSP not visible. V:
several possible LSPs; none stands out.

TW Aur: Houk: 150:d, 1370d. v: 1353d (0.17). V: 143.6d (0.21). ASAS-SN
suggests $\sim$120d, $\sim$600d. 104d GCVS period not supported.

UU Aur: Houk: 235d, 3500d. v: 235.1d (0.08), 444.6d (0.13), no LSP visible.

RX Boo: Houk: 78:d, 500d. v: 160.4d (0.07), LSP not clear. PD: 160.3d,
2205.1d.

U Cam: Houk: 400:d, 3000d. v: 219.5d (0.09), 2941d (0.12). V: 212.8d (0.13),
2631d (0.23). ASAS-SN: 400d. PD: 219.4d, 2967.4d. Possibly bimodal.

RS Cam: Houk: 88.6d, 960d. v: 89.4d (0.19), 159.4d (0.10), 1019d (0.14). ASAS-
SN: $\sim$90d. PD: 90.5d, 999d. Possibly bimodal.

ST Cam: Houk: 195d, 2100d. v: 201d (0.09), 363d (0.10), 1557d (0.09). ASAS-SN:
170d. Possibly bimodal.

RS Cnc: Houk: 120d, 1700d. v: 238.9d (0.13). V: 239.5d (0.21). PD: 240.8d,
2050d.

RT Cnc: Houk: 90d, 542d. v: 93$\pm$3d (0.06), 750$\pm$100d (0.06). V:
93$\pm$3d (0.17), 600$\pm$50d (weak). ASAS-SN: $\sim$100d. PD: 89.3d, 691.7d.
Evidence for LSP is weak.

Y CVn: Houk: 158d, 2000-2200d. v: 159.2d (0.05); there is an indistinct peak
at 2008d. PD: 160d, 2008.9d.

TZ Car: Houk: 69d, 1183d. v: 1213d (0.24), shorter periods are noisy. ASAS-SN:
$\sim$70d.

BZ Car: Houk: 97d, 1800d. v: 110.7d (0.09). ASAS-SN: $\sim$120d.

VY Cas: Houk: 100d, 600d. v: 93.4d (0.12), 1014d (0.15). ASAS-SN: $\sim$100d,
$\sim$600d. LSP uncertain. 116d GCVS period probably needs adjusting.

WZ Cas: Houk: 186.0d, 4000d. v: 192.1d (0.06), 369.7d (0.15); LSP amplitude
less than 0.04. V: 370d only. ASAS-SN: 200d. Possibly bimodal.

AA Cas: Houk: 70-115d, 850d. v: 74.3d (0.04), 870d (0.04). ASAS-SN: $\sim$75d.
PD: 80.1d, 866.8d. LSP probably present. No GCVS period.

CQ Cas: Houk: 110-250d, 2300d. v: 204.8d (0.33), 367.4d (0.40), not aliases.
V: 204d (0.59). Bimodal, no strong evidence for LSP.

SS Cep: Houk: 90d, 950d. v: 100.3d (0.04), 951.5d (0.10). V: 99.4d (0.10),
179d (0.12), 922.5d (0.33). ASAS-SN: $\sim$100d, $\sim$1000d. PD: 101.1d,
955.2d. Possibly bimodal.

CV Cep: Houk: 60d, 2000d. V: 731d (0.23). ASAS-SN: $\sim$60d and 1000+d.

RR CrB: Houk: 60.8d, 377d. v: 55.3d (0.04), 91.0d (0.03), 392.8d (0.04). ASAS-
SN consistent with $\sim$60d and $\sim$400d. PL: 55.5d, 630d. LSP probably
present. Low amplitudes!

RS CrB: Houk: 69.5d, 331d. v: 330.0d (0.24). ASAS-SN: $\sim$70d, $\sim$300d.
332.2d GCVS period is LSP.

T Crt: Houk: 70$\pm$, 1006. ASAS-SN: $\sim$70d.

AF Cyg: Houk: 94.1d, 960d. v: 93.4$\pm$1d (0.10), 174$\pm$2d (0.08), 913d or
1439d (0.07). V: 92.8d (0.22), 174$\pm$2d (0.34), 92.8d (0.22). ASAS-SN:
$\sim$160d. PL: 94d, 1439d. Probably bimodal with probable LSP.

AW Cyg: Houk: 220d, 2200d. v: 357d (0.06); no sign of a 220d period or LSP. V:
203.8d (0.12), LSP, if present, is weak. ASAS-SN: $\sim$190d. PD: 209:d,
2289:d.

CH Cyg: Houk: 97, 4700. The light curve of this famous variable is dominated
by large irregular eruptions.

V539 Cyg: Houk: 160d, 1500d. v: 355$\pm$10d (0.11). ASAS-SN: $\sim$90d. 160d
GCVS period appears not correct.

S Dra: Houk: 136d, 1319d. v: 181.7d (0.07), 311.7d (0.07), 997d (0.08). ASAS-
SN: $\sim$200d. The Houk and GCVS period of 136d does not seem to be present.

RS Dra: Houk: 141.15d, 280d. v: 142$\pm$1d (0.27), 276$\pm$2d; the first may
be a harmonic. ASAS-SN: $\sim$40d and 300d. Difficult to interpret; if the 40d
ASAS-SN period is real, then one of the longer periods may be an LSP.

TX Dra: Houk: 78d, 654d. v: 77.0d (0.08), 136$\pm$1d (0.07), 712.8d (0.14). V:
77.5$\pm$1d (0.14), 132$\pm$2d (0.24), 868d ((0.17). ASAS-SN: 75d. PD: 77.5d,
711.8d. Bimodal pulsator with LSP.

UU Dra: Houk: 120d, 960d. ASAS-SN: $\sim$120d. No sign of LSP.

Z Eri: Houk: 80d, 746d. v: 78.4d (0.05), 724.6d (0.10). V: 78.5d (0.11),
730.5d (0.13). PD: 78.5d, 692.4d. PL: 118.4d, 725d. Longer period is an LSP.
The 80d GCVS period needs refining.

X Her: Houk: 95.0d, 746d. v: 101.4d (0.06), 700$\pm$30d (0.08). V: 101.5d
(0.19), 700$\pm$30d (0.23). ASAS-SN: $\sim$100d; an 800d period may be
present. PD: 176.6d, 1185.3d. PL: 176.6d, 667d.

30 g Her: Houk: 70$\pm$d, 900d. v: 89d (0.03), 880d (0.16). V: 88d (0.07),
873d (0.16). PD: 87.6d, 877d. LSP is present.

V Hya: Houk: 533d, 6500d. v: 531.9d (0.53), 6329d (1.13). PD: 531.4d, 6907.4d.

Y Hya: Houk: 95d, 302.8d, 1200d. v: 363.0d (0.13), no conspicuous LSP.

SU Hya: Houk: 95$\pm$d, 780d. ASAS-SN: 90d, 500d.

U Lac: Houk: 150d, 550-690d. AAVSO data sparse. ASAS-SN: $\sim$50d, 700d
(dominant).

RV Lac: Houk: 67d, 550-700d. v: 632.9d (0.32). V: 70d (0.19), 98d (0.19),
617.3d (0.40). ASAS-SN: $\sim$60d, 600d. PD: 70d, 632d. LSP present.

RY Lac: Houk: 38-91d, 300-420d. v: 46d (0.03), 81d (0.04), 612d (0.05). V: 47d
(0.10), 80d (0.10), 600-800d (0.13). ASAS-SN: 79.6d, 300-400d. Bimodal; LSP is
not clear.

BU Lac: Houk: 200d, 2000d. GCVS: 2000d. v: 2128d (0.95). V: 2403d (1.25); no
evidence in v or V or ASAS-SN for 200d period.

S Lep: Houk: 90d, 875-890d. v: 857d (0.23), very weak peak at 99d (amplitude
less than 0.05); V: 878d (0.31), no peak near 90d. Rather bright for ASAS-SN.

KP Lyr: Houk: 146d, 1300d. v: 140.4d (0.21) plus several possible LSPs. ASAS-
SN: 130d and possibly $\sim$1000d. No LSP results.

Y Mic: Houk: 182d, 4650d. v: 181$\pm$2d (0.17), no clear LSP.

RV Mon: Houk: 131.5d, 1047d. v: 360.0d (0.13), 982.3d (0.12). LSP is weak.

T Mus: Houk: 93d, 1021d. v: 1022.5d (0.23); 93d period not present in our
data.

W Nor: Houk: 134.7d, 1300d. v: 147.2d (0.19); possible 823d or 2020d LSP.
ASAS-SN is consistent with this pulsation period and an LSP $\geq$ 2000d.

TW Oph: Houk: 165:d, 2000d. v: 171.9d (0.16), 376.1d (0.41). GCVS: 185d.
Bimodal.

W Ori: Houk: 212d, 2450d. v: 210.7d (0.12), 316.2d (0.11), 2358d (0.20). V:
317.3d (0.25). PD: 210.7d, 2335.1d.

BQ Ori: Houk: 110d, 795d. v: 127.3d (0.10), 246.9d (0.17), possibly 709d. V:
118d (0.25), 248d (0.50). ASAS-SN: $\sim$100-250d, irregular. PD: 246.6d,
2147.8d. Bimodal?

V574 Oph: Houk: 71.5d, 500d. ASAS-SN: 70d; possible LSP.

RX Peg: Houk: 110d, 629d. V: $\sim$650d, shorter period possibly present.
ASAS-SN: $\sim$110d, possible LSP $\geq$ 1500d.

TW Peg: Houk: 90d, 956.4d. v: 100.0d (0.04) weak, 948.8d (0.11). ASAS-SN:
$\sim$100d, $\sim$750d.

T Per: Houk: 326d, 2800d. v: 2488d (0.06), 326d not present. V: 342.9d (0.18),
1938d (0.17) uncertain. ASAS-SN: 318.7d, possible 2000-3000d. GCVS: 2430d.

Y Per: Houk: 252.3d, 2400d. v: 252.8d (0.41), LSP amplitude less than 0.04. V:
253.8d (0.37), possible LSP. ASAS-SN: $\sim$200d.

U Per: Houk: 320.63d, 2500d. v: 318.57d (1.07), 2531.6d (0.31). V:
2500$\pm$10d. ASAS-SN: 320.3d, slow (could be 2500d).

TT Per: Houk: 82d, 843d. v: nothing stands out. ASAS-SN: $\sim$90d.

UZ Per: Houk: 91d, 927d. v: 896.9d (0.29), no sign of 91d. V: no sign of 91d.
ASAS-SN: 90d, $\sim$1000d. PD: 186.3d, 893.4d.

rho Per: Houk: 33-55d, 1100d. V: 54.65d (0.05), 1169d (0.04). PL: 55d, 723d.

RT Psc: Houk: 70d, 533d. v: 511.2d (0.19), no sign of shorter period. V:
517.6d (0.27), shorter period signal is noisy.

RW Psc: Houk: 31-53d, 154d. v: 154.2d (0.15). ASAS-SN: 65$\pm$15d, 150d.
Unusual period ratio.

Y Tau: Houk: 240.9d, 1750d. v: 244.7d (0.10), no LSP. V: 1733d (0.36). ASAS-
SN: $\sim$250d.

Z UMa: Houk: 198d, 1560d. v: 192$\pm$3d (0.38), no LSP. V: 188.5d (0.92),
4184d (0.30). ASAS-SN: $\sim$200d, no evidence for 1560d period.

ST UMa: Houk: 81d, 590d. v: 90.2d (0.04), 610.5d (0.06). V: 89.5d (0.13), 600d
(0.14). ASAS-SN: 80$\pm$10d, $\sim$600d possibly present. PD: 90.3d, 623.1d.

V UMi: Houk: 72.0d, 760d. v: 72.88d (0.10), 121.1d (0.04), 754,7d (0.08). V:
71.2d (0.16), 124.8d (0.15), 730d (0.22). ASAS-SN: $\sim$75d. PD: 72.9d,
757.3d. Bimodal.

TZ Vir: Houk: 134d, 6900d. v: 167.7d (0.24), 8928d (0.25). ASAS-SN: consistent
with 160d period.


# Towards 3D VR-Sketch to 3D Shape Retrieval

Ling Luo1,2 Yulia Gryaditskaya1,2 Yongxin Yang1,2 Tao Xiang1,2 Yi-Zhe Song1,2
1SketchX, CVSSP, University of Surrey 2iFlyTek-Surrey Joint Research Centre on
Artificial Intelligence

###### Abstract

Growing free online 3D shapes collections dictated research on 3D retrieval.
Active debate has however been had on (i) what the best input modality is to
trigger retrieval, and (ii) the ultimate usage scenario for such retrieval. In
this paper, we offer a different perspective towards answering these questions
– we study the use of 3D sketches as an input modality and advocate a VR-
scenario where retrieval is conducted. Thus, the ultimate vision is that users
can freely retrieve a 3D model by air-doodling in a VR environment. As a first
stab at this new 3D VR-sketch to 3D shape retrieval problem, we make four
contributions. First, we code a VR utility to collect 3D VR-sketches and
conduct retrieval. Second, we collect the first set of $167$ 3D VR-sketches on
two shape categories from ModelNet. Third, we propose a novel approach to
generate a synthetic dataset of human-like 3D sketches of different abstract
levels to train deep networks. At last, we compare the common multi-view and
volumetric approaches: We show that, in contrast to 3D shape to 3D shape
retrieval, volumetric point-based approaches exhibit superior performance on
3D sketch to 3D shape retrieval due to the sparse and abstract nature of 3D
VR-sketches. We believe these contributions will collectively serve as
enablers for future attempts at this problem. The VR interface, code and
datasets are available at https://tinyurl.com/3DSketch3DV.

##  1 Introduction

3D model retrieval has become an important topic due to a growing number of
free online 3D repositories. It finds applications in CAD design, 3D printing,
3D animation and movies production, where the time required to model a 3D
shape can be strongly reduced by relying on retrieval from existing 3D shape
collections. Various input modalities have been tried – images [15] and rough
2D sketch [33] at first, with latest research focusing on 3D to 3D, i.e.,
using 3D scans [1, 5] or existing 3D shapes [31, 25, 22, 7, 10, 11, 13, 24,
26, 32].

Despite great strides made, two salient questions still remain (i) what
constitutes the best input modality to initiate 3D retrieval, and (ii) under
what usage scenario can such retrieval be best facilitated. For the former, 2D
sketches and images are both in 2D, therefore can not reach the level of
details desired, and the 2D-3D domain gap can also be counter-intuitive. The
3D-based paradigm on the other hand dictates existing 3D models to be readily
available, which to some degree forms a “chicken-and-egg” problem (i.e.,
where/how to source the input model at the first place). As for the latter,
apart from 2D sketches which are freely defined by the user, all other usage
scenarios do not offer flexibility in terms of the input desired – one can not
easily alter an image, not to mention a 3D model/scan.

In this paper, we offer a new perspective on 3D shape retrieval – we advocate
the use of 3D sketches as a new input modality. This new 3D-VR sketch to 3D
shape paradigm not only enables detailed retrieval as per the common 3D model
to 3D model setting, but also simultaneously offers a degree of flexibility
found elsewhere in 2D sketch-based retrieval. Our ultimate vision is as
follows: with a specific 3D model in mind, one emerges into a VR environment,
roughly sketch out the model using handheld controllers, press retrieve and
then relevant models would start populating the VR environment – for any
downstream tasks the user may desire (e.g., VR shopping, 3D printing).

Our first contribution is therefore coding a VR environment for the said
purpose. With this VR environment, we collected the first human VR-sketch
dataset. $10$ users with no artistic background were hired to produce a total
of $167$ 3D VR-sketches from the two categories from ModelNet: chairs and
bathtubs. Some examples are shown in Figure 1. Since the collection process is
both time- and cost-sensitive, we additionally propose the first 3D model to
3D sketch generator, and construct a synthetic dataset of 3D sketches. A key
trait of our generator is that it can produce 3D sketches of different
abstraction levels, effectively simulating that found in real-human sketches.
Through training a series of deep 3D VR-sketch to 3D retrievals model using
the real and synthetic datasets, we drive out a few important insights (i)
retrieval performance drops with an increasing abstraction level, and (ii)
models trained with synthetic sketches can already reach a decent performance
level when tested on human sketches.

Figure 1: Our collected VR sketches exhibit variability of sketching styles
and levels of details.

At last, we experiment with different shape/sketch representations, and state-
of-the-art losses commonly used in 3D model retrieval [14, 13, 32] but re-
purposed for our problem. We focus on investigating the domain gap between 3D
shapes and 3D sketches (see Figure 1 for an example), due to the two key
properties of sketches: _sparsity_ : full 3D models versus sparse sketched
lines, and _abstraction_ : 3D models are geometrically perfect, while sketches
are subject to deformations. In particular, we examine multi-view versus
point-based 3D representations, and show that the later is more robust to both
sparsity and an increased level of abstractness. We further propose an
architecture with a reconstruction path to explicitly allow for change in
abstractness.

In summary, our contributions include: (i) a new perspective on 3D model
retrieval, where 3D VR-sketches are used for the first time to conduct
retrieval, (ii) a dataset of human 3D VR-sketches, collected using a purpose-
built VR environment, (iii) an approach to generate a synthetic 3D sketch with
a variable level of abstractness, (iv) comprehensive evaluations using recent
3D shape retrieval models re-purposed to the task of 3D sketch-based
retrieval, to drive out insights, plus a novel regularization track to address
the sketch-model domain gap.

##  2 Related work

##### 3D shape representation.

When dealing with 3D shape retrieval from a single image or a 3D shape,
existing work is divided into two large groups based on the shape
representation used: view-based [31, 25, 22, 7, 10, 11, 13] or volumetric. The
volumetric representation can be further broken down to point-cloud [29, 24,
26, 32] or voxel-based [36, 23, 2, 19]. In this work we show that on the task
of 3D sketch to 3D model retrieval point-cloud representation beats multi-view
approaches due to better handling of the sparsity of 3D sketches.

##### 2D image- and 3D shape-based retrieval.

The retrieval problem in a multi-class setting is closely related to the shape
classification problem, where the intermediate embedding of an image or a 3D
shape is used for retrieval. A vanilla approach for multi-class shape
classification is to use a softmax cross-entropy loss [13, 20]. Others works
specifically tackle retrieval [31, 14], where the triplet loss and its
variants [34, 38, 35, 14] have become a common standard. Amongst these, [14]
represents the state-of-the-art for 3D model retrieval. It combines triplet
and center losses [35], to solve for their respective drawbacks by using the
class center as a positive sample. In this work we evaluate both the triplet
and the triplet center losses on the task of 3D sketch-based retrieval and
combine them with an additional Reconstruction Loss, tailored to the 3D
sketch-based retrieval problem.

Due to a domain gap between images and 3D shapes, or target and query 3D
models being from different distributions, Siamese [38, 27] or Heterogeneous
[37, 15, 20] network architectures can be more beneficial for a certain
problem. In this work, we compare these two types of architectures for our
multi-view baseline, and show that due to a strong domain gap between a 3D
sketch and a 3D shape, the Heterogeneous architecture by far beats the Siamese
one.

##### 3D sketch-based 3D model retrieval.

So far there was very little work on 3D sketch-based retrieval [17, 16, 37,
8], especially in recent years. Most of them work with sketches collected
using Microsoft Kinect, which not only has limited tracking accuracy, but the
collection interface is also counter-intuitive, where 3D sketching is
performed while visualizing 2D projections. As a result, sketches collected
mostly have low fidelity and exhibit less details. Our VR-sketches are
completely different: (i) visualization and sketching are both performed in
3D, and (ii) the use of the latest VR technology offers a high precision.
Together, they ensure our VR-sketches are of high-fidelity and rich in
details, thus more fitting for retrieval. A notable exception is the work by
Giunchi et al. [8], yet they address a different problem of 3D model retrieval
for dense-color VR sketches (and optionally base 3D shapes), while we target
at a more simplistic and abstract shape representation – a sparse set of
single-color lines. Note that we can not find the said dataset [17, 16, 37] in
an open access, thus can not offer a direct comparison. Our NGVNN [13]
baseline is however already superior to the state-of-the-art used by [37] and
[8].

##### Non-photorealistic rendering (NPR).

NPR is an old graphics and vision problem, for a detailed overview of existing
methods for generating 2D NPR rendering we refer the interested reader to a
recent report [3]. The only attempt to produce the representation that
resembles a 3D sketch automatically form a 3D shape was proposed by Li et al.
[17] as a concatenation of the shape views from six canonical viewpoints. They
use this representation to show that the performance of their non-learning
method on outline to 3D model retrieval task significantly outperforms the
performance of the 3D sketch to 3D model retrieval. This experiment indicates
that such simple shape representation is not sufficient as training data for
3D sketch-based 3D shape retrieval. Our experiments support that: The network
trained on sketches with higher abstractness values significantly outperforms
the network trained on clean sketches. To the best of our knowledge, we are
the first to propose a method to generate abstract 3D sketches from 3D models.

##  3 Datasets

Collecting a full dataset of 3D human sketches is a labor intensive task. We
collect a small dataset of human sketches that we use to guide the synthetic
dataset generation. We as well use it as a test set to validate that the
network trained on the proposed synthetic data generalizes well to human
sketches. To obtain the training data, we reside to a common strategy of
generating a synthetic training data.

###  3.1 Selected shapes

For training and testing our sketch-based retrieval models we use the repaired
clean manifold meshes111https://github.com/lei65537/Visual_Driven_Mesh_Repair
[4] from ModelNet10. We use all 10 classes from ModelNet10, with the total of
3958 shapes. We split the dataset into train, validation and test sets, which
contain 2847, 317 and 792 shapes, respectively, ensuring a proportional split
of shapes of each class between three sets.

###  3.2 Human 3D sketch dataset

##### Task.

We are targeting 3D sketches created by novices, which can be viewed as an
equivalent of the quick 2D sketches from the QuickDraw dataset [12]. To enable
the usage of the collected sketches for fine-grained retrieval evaluation we
built a dataset of paired sketches and 3D models. We experimented with a
setting, where one can observe a 3D model in VR for an unlimited duration of
time, and then is asked to sketch from memory. We observed that under this
scenario the participants were sometimes omitting features important to
accurately testing fine-grained retrieval, and instead let the participants to
sketch over a reference 3D model (Figure 2). We provide in the supplemental a
qualitative evaluation of the sketches from memory and the retrieval
performance on such inputs. To mimic the level of detail that can be expected
from the 3D sketches from the imagination, we opted to use wide ribbon lines,
but do not pose any constraints on sketching style or level of detail.

##### Participants.

We selected 139 chairs and 28 bathtub shapes from ModelNet10’s test set, and
hired 10 participants, who have no art background or 3D sketch experience. The
shapes were randomly divided into 10 subsets, consisting from 13-14 chairs and
2-3 bathtubs each. Each participant was assigned with one of these subsets.
There is no duplicated shapes between subsets.

Figure 2: Our VR sketching environment allows to load the reference 3D model,
and the user is asked to sketch around it. At any moment the user can hide the
3D model. Figure 3: For a subset of shapes from the two categories from
ModelNet10: chairs and bathtubs (a), we collect human novices sketches (b). We
generate a synthetic dataset of 3D sketches for the 10 shape categories from
ModelNet10. We first generate detailed curve networks with FlowRep [9] (c) and
then apply a set of detail and stroke filtering steps to mimic the appearance
of human sketches, where the appearance is controlled by an abstractness
parameter $l_{a}$ (d), defined in Section 3.3.3.

##### Interface.

Although there are general-purpose VR painting and design softwares that
enable users to draw directly in 3D (such as Google’s Tilt
brush222https://www.tiltbrush.com/, and Facebook’s
Quill333https://www.quill.com/), they do not serve our purpose in full: (i) we
would like to record detailed stroke-based information, (ii) we require the
option of displaying a reference model for data collection, and (iii) we want
a fresh code base for any additional functionalities in the future (e.g.,
sketch-based 3D editing). We implemented our custom 3D sketching environment
based on Oculus Rift platform and Unity engine.

###  3.3 Synthetic training data generation

As a first step towards obtaining a synthetic sketch representation of a 3D
shape, we extract detailed curve-networks with the method of Gori et al. [9].
This method is designed to produce a curve network that preserves well shape
details, which are, though, uncommon for human novices sketches (Figure 3
b,c). We observe that novices not only omit small details, but tend to
represent thin volumetric details with single lines and moreover often omit
subset of feature lines (Figure 3 b). To obtain human sketch appearance, we
focus on two aspects: level of detail and mechanical inaccuracies. We first
perform details filtering and lines consolidation, followed by lines
filtering. We then break the long curve chains into shorter strokes, to which
we apply a set of local and global transformations (Figure 3 d).

####  3.3.1 Curves network extraction.

FlowRep method [9] requires an input to be a curvature-aligned quad-dominant
mesh. For processing efficiency, we simplify original triangular meshes by
decimation before converting them into quad-dominant meshes using Blender, but
the quality of conversion does not always comply with the requirement of
FlowRep. Thus, in this stage, we filtered nearly 20% of original dataset which
failed to be processed by FlowRep. This can be alleviated in the future by
replying on more advanced quad-meshing algorithms [21].

FlowRep allows to control the level of detail of the output curve network by
adjusting the values of the descriptiveness threshold $d_{max}$. We,
nevertheless, found little impact of this parameter on the output networks, as
illustrated in Figure 4. We set this parameter to $20^{\circ}$ to generate our
synthetic dataset, following their default value.

Figure 4: We found little effect of the descriptiveness threshold parameter
$d_{max}$ in FlowRep [9] on most of the shapes from ModelNet10 dataset.

####  3.3.2 Details filtering and consolidation.

The code by Gori et al. [9] returns networks of curves in a form of chained
edges of an input mesh. We first break these chains into several if the angle
between two consequent edges is smaller than 135 degrees. We then remove all
the chains those length is smaller than $10\%$ of the smallest of height and
width of the bounding box of the original shape ($d_{min}$). We re-sample all
the chains using Ramer-Douglas-Peucker algorithm [28] with an accuracy
parameter set to $0.02\,d_{min}$. Finally, we iteratively go over all the
chains and compute for each chain the closest, tangentially aligned chain. If
the distance between such two chains is smaller than $5\%$ of the largest
length of the two chains, the chains are substituted by their aggregate curve.
These steps allow to remove small details and close to each other lines. Yet,
the simplified curve networks contain many more lines than can be found in
majority of 3D sketches by novices.

####  3.3.3 Abstraction.

In this section we describe our approach to obtain a 3D sketch with different
levels of detail and mechanical inaccuracies, which we jointly refer to as a
level of sketch abstractness $l_{a}$. We constraint $l_{a}$ to be between 0
and 1.

##### Level of detail

(I) To reduce the number of lines we first compute the similarity matrix using
a discrete Frechet distance among chains. When computing the Frechet distance
we first align the strokes at one of their end points, by translating one of
the strokes. We then perform the grouping according to this similarity matrix
with Agglomerative Hierarchical Clustering, where the number of cluster
$n_{cluster}$ is a function of $l_{a}$ and the number of chains in the sketch
$n_{chains}$: $n_{cluster}=\max(n_{chains}(1-0.8l_{a})/2,10)$. (II) We next
for each cluster iteratively select a pair of two most distant lines in terms
of their absolute positions and compute the mean distance from all the lines
$d_{mean}$ in the cluster to these two lines. We remove all the lines in a
cluster for which the distance to any of the two selected lines is less than
$d_{mean}$. (III) After getting the reduced set of chains, we break long
chains into several shorter chains, which we refer to as strokes. We split
each chain at vertices, where the curvatures are twice larger than the mean
curvature of an original chain. We further filter out short strokes whose
lengths are smaller than $0.2s_{max}$ to avoid tiny details, where $s_{max}$
is the largest dimension of an input network.

Figure 5: The two diagrams on the left demonstrate the heterogeneous and
siamese network architectures with NGVNN as a backbone. The two diagrams on
the right show the networks based on PointNet++, when trained as described in
Section 4.2 and in Section 4.3.

##### Mechanical inaccuracies

To mimic mechanical and perspective inaccuracies of human sketches we apply to
each stoke a set of global and local deformations. We first apply a global
translation, rotation and scaling, depending on the given level of abstraction
$l_{a}$. To achieve this we deploy an auxiliary parameter $t$, which is
randomly sampled from the range $[0,1.5l_{a}]$. The rotation angle for each
axes is then independently randomly sampled from the interval $[-10t,10t]$
degrees. The scale factor for each axes is independently randomly sampled from
$[1-0.1t,1+0.1t]$. To obtain a translation vector we randomly sample from the
surface of a sphere with its radius value randomly sampled from
$[0,s_{max}t]$, where $s_{max}$ is the largest dimension of an input network,
as before. After a global stroke deformation, we apply a random translation
for each stroke vertex. The translation vector is randomly sampled from the
disk with a radius $r_{v_{i}}$, which lies in the plane orthogonal to the
stroke direction in the vertex ${v_{i}}$. The radius $r_{v_{i}}$ is randomly
sampled from the range $[0,0.1l_{a}l_{stroke}]$, where $l_{stroke}\in[0,1]$ is
the length of the current stroke. We extend both ends of each stroke by $p$
which value is randomly sampled from $[0,0.1s_{max}]$, to reproduce human
strokes appearance.

After global and local stroke deformations, we apply 3D spline interpolation,
which results in smoother strokes, matching the appearance of human strokes.

For our experiments, we generate 5 synthetic datasets with 5 levels of
abstraction, $[0.0,0.25,0.5,0.75,1.0]$, and obtain two mixed datasets by
mixing sketches of all abstraction levels or only the sketches with 3
abstraction levels in the middle. Figure 3 c. shows example sketches obtained
with different settings of $l_{a}$ parameter.

##  4 Networks

###  4.1 Backbones

##### Multi-view representation

We choose View N-Gram Network (NGVNN) [13] as a baseline for a multi-view
based shape retrieval, since it was shown to outperform the alternative
solutions [18, 14, 7, 31] on the 3D-shape based retrieval task on ModelNet40.
We employ VGG-11 with batch normalization [30] as a backbone. The output of
the penultimate layer is used as a feature embedding. We experiment with two
architectures: one where the same network is used to encode the 3D sketch and
3D shape (Siamese), and one where there are two branches those weights are
trained separately (Heterogeneous) (see Figure 5).

##### Volumetric representation

Due to sparsity of a 3D sketch, among the volumetric shape representations we
opted for point-based. In particular, we select PointNet++ [26], which applies
hierarchical feature learning to efficiently capture local structures and to
get a robust point cloud representation. We adopt their multi-scale grouping
and random input dropout during training (MSG+DP) strategy, which is more
robust to a variable sampling density on an input point cloud due to an
adaptively selected neighborhood’s size. We use the same three-level
hierarchical network with three fully connected layers as in the original
paper and use the output of the first fully connected layer as a feature
embedding. We use the Siamese network for this model.

We train both PointNet++ and NGVNN from scratch for a fair comparison. The
feature vector is a 512-dimensional vector in both cases.

###  4.2 Losses

We compare the performance of the two backbone 3D representations with the
optimal sampling and rendering strategies, when trained with the
classification loss $\mathcal{L}_{cls}$, versus training with a combinations
of the classification loss and the triplet loss $\mathcal{L}_{tl}$ [34], or
the combination of the classification loss and the triplet center loss
$\mathcal{L}_{tcl}$ [14]:

$\mathcal{L}=\lambda_{cls}\mathcal{L}_{cls}+\lambda_{tl\backslash
tcl}\mathcal{L}_{tl\backslash tcl},$ (1)

we set $\lambda_{cls}=1$ and $\lambda_{tl}=\lambda_{tcl}=0.1$.

The goal of both triplet loss and triplet center loss is to obtain a better
structured embedding space. The input for the cross-entropy loss
$\mathcal{L}_{cls}$ are the class probabilities, while the input for the
triplet loss and triplet center loss are features extracted by the encoders
(Figure 5).

##### Triplet loss

The goal of the triplet loss is to ensure that the anchor-negative distances
are larger than the anchor-positive distances by a given margin. For the
triplet loss $\mathcal{L}_{tl}$, we use both 3D sketches and 3D shapes as
anchors. 3D shapes and 3D sketches sharing the same class as the anchor are
treated as positive, while those from different classes are negative samples.
The triplet loss for the $i$-th triplet is:

$\mathcal{L}_{tl}^{i}=\max\\{0,d_{pos}^{i}-d_{neg}^{i}+m_{tl}\\},$ (2)

where $d_{pos}^{i}$ is the anchor-positive distance and $d_{neg}^{i}$ is the
anchor-negative distance, $m_{tl}$ is the margin. As a distance $d$ we use the
squared euclidean distance, where the feature space is normalized to a unit
hypersphere. We choose the margin value by calculating the average difference
between $d_{pos}$ and $d_{neg}$ of the validation set of the model trained
with the classification loss only. We set margin for NGVNN baseline to
$m_{tl}=2.0$ and for PointNet++ to $m_{tl}=1.8$.

##### Triplet center loss

Triplet center loss [14] combines the strength of the triplet loss and the
center loss. It avoids the problem of triplet hard mining by taking as a
positive the learned center of the shape’s class. As a negative sample it
takes the closest center of some other class:

$\mathcal{L}_{tcl}^{i}=\max\\{0,d(f^{i},c_{y^{i}})-\min_{j\neq
y^{i}}d(f^{i},c_{j})+m_{tcl}\\},$ (3)

where $y^{i}$ is a class label of the anchor 3D sketch/shape, $f^{i}$ is the
embedding of an anchor and $c_{k}$ is the center of the class $k$ in the
embedded space.

Given the data imbalance among classes of the used 3D shape dataset, we employ
a balanced batch sampler. For each batch, we first randomly select $k=8$
classes from all $K$ classes, then we select $p=1$ 3D sketch-shape pairs for
each of the $k$ class. For the efficiency, we use an on-the-fly hard negative
triplet mining to calculate the triplet loss rather than generating the
triplets beforehand.

The final loss $\mathcal{L}_{tl/tcl}$ is the mean of all triplet (center)
losses within a batch: $\mathcal{L}_{tl\backslash
tcl}=\frac{1}{N}\sum_{i=1}^{N}\mathcal{L}_{tl\backslash tcl}^{i},$ where $N$
is the number of all valid triplets.

###  4.3 Regularization branch

To account for a variation of human styles, we aim at pushing the 3D sketch
embedding towards style invariance by introducing a regularization branch. We
propose the architecture with an additional decoder branch (Figure 5), based
on the reconstruction point-based architecture (PSGN) by Fan et al. [6], and
only use 2 fully connected layers to form the reconstruction branch. As a
reconstruction loss, we use the Chamfer distance $\mathcal{L}_{Ch}$ between an
input 3D sketch with a given level of abstraction and the 3D curve network
obtained as described in Section 3.3.2. The full loss is

$\mathcal{L}=\lambda_{cls}\mathcal{L}_{cls}+\lambda_{tl\backslash
tcl}\mathcal{L}_{tl\backslash tcl}+\lambda_{Ch}\mathcal{L}_{Ch},$ (4)

where $\lambda_{cls}=1$ and $\lambda_{tl\backslash tcl}=0.1$, as before. We
choose $\lambda_{Ch}=8$ for TCL and $\lambda_{Ch}=12$ for TL.

Figure 6: Visual comparison of random and uniform sampling strategies. To
clearly show the difference, we set the number of points to 256 for this
visualization. Figure 7: mAP values and Top-1 scores using models trained with
the dataset of 3D sketches with $l_{a}=0.5$. When we train PointNet++ with
$m_{tcl}=75$, the value that optimizes the validations synthetic dataset, then
CL+TCL+Rec gives the highest mAP over all the baselines on all synthetic
datasets, but has slightly lower mAP on human dataset than the network trained
with CL+TCL. When $m_{tcl}=50$, the value that optimizes the mAP on a human
dataset of the network trained with CL+TCL, the reconstruction loss (Rec)
boosts the mAP on human dataset from $0.969$ to $0.983$. ’:network’ means that
the network, obtained as described in Section 3.3.2, is contributing to the
triplets constriction.

##  5 Evaluations

We adopt the following common evaluation measures to evaluate the retrieval
performance: Mean Average Precision (mAP), Normalized Discounted Cumulative
Gain (NDCG), Nearest Neighbor (NN), which evaluate the ability of a model to
discriminate between shape classes, and Top-k accuracy, which measure how many
of the retrieval tasks have a ground-truth within top k retrieved results.

###  5.1 Sampling and rendering strategies

We experiment with two rendering styles for NGVNN [13] and two sampling
strategies for PointNet++ [26].

##### 3D sketch rendering for multi-view network.

We generate 12 $224\times 224$ orthographic views of each 3D shape and 3D
sketch by placing 12 virtual cameras around it every 30 degrees, as was
proposed by Lee et al. [15]. The cameras are elevated 30 degrees from the
ground plane. For both 3D shapes and 3D sketches we experiment with two types
of rendering styles: Phong Shading and depth maps. For 3D sketches we
represent each line as a 3D tube.

##### Point cloud sampling.

To get the point cloud representation, we first sample 10000 points from
shapes and 3D sketches. For shapes we use Monte-Carlo
sampling444https://github.com/fwilliams/point-cloud-utils and for sketches we
use equidistant sampling. We then adopt two types of sampling from the initial
10000 points to obtain a sparse set of 1024 points: random sampling or uniform
sampling. The uniform sampling is obtained with the farthest point sampling.
The sparse sets are obtained on-the-fly and thus might differ from one
iteration to another. Figure 6 shows the visualization of random and uniform
sampling.

###  5.2 Effect of sampling and rendering.

We compare different sampling methods for PointNet++ and rendering styles for
NGVNN, described in Section 5.1, when training with the classification loss
and the triplet loss on the pairs of 3D shapes and 3D sketches rendered with
$l_{a}$ set to 0.5. As shown in Figure 8, the uniform sampling for PointNet++
and depth rendering for NGVNN perform best, so we use these settings for the
rest of the experiments.

It can be seen from Figure 8 that point-based representation (dashed lines) by
far outperforms the multi-view representation (solid lines) when we use the
siamese architecture for NGVNN. We thus for the rest of experiments use the
heterogeneous architecture for NGVNN.

Figure 8: Comparison of sampling strategies and rendering styles, evaluated on
synthetic datasets generated with different levels of abstractness $l_{a}$.
All the baselines in this figure were trained using a combination: CL + TL.

###  5.3 Triplet loss vs triplet center loss

Figure 9: Evaluation of the performance of PointNet++, trained with CL+TCL, as
a function of the different values of $m_{tcl}$, see Equation 3.

The performance of the triplet (TL) and triplet center losses (TCL) is highly
dependent on the selected values of the margin. For the triplet loss we select
the margin $m_{tl}$ value as described in Section 4.2. To select the optimal
value of $m_{tcl}$, we train PointNet++ with CL and TCL on the synthetic
dataset with $l_{a}=0.5$ and different values of $m_{tcl}$ (Figure 9).
Different values optimize the mAP on the synthetic, $m_{tcl}=75$, and human,
$m_{tcl}=50$, test data. Figure 7 shows that the mAP of TL and TCL on our
problem is comparable, while the Top-1 accuracy is much higher for the TL,
showing that TL learns better inner-class variance and is more suitable for a
fine-grained retrieval.

Figure 10: Example retrieval results with TL+Rec, the shapes matching the
ground-truth are marked with orange boxes. Please see the supplemental for
more examples. Figure 11: Comparison of the training datasets (Section 5.5).
Each line corresponds to a different training dataset. For the dataset which
consists of all abstractness levels for each shape we randomly selected one of
the abstractness levels.

###  5.4 Regularization branch

The reconstruction branch improves the mAp values for the TCL, while the mAP
values for TL are similar with and without the reconstruction branch, both
resulting in top performance. In Top-1 scores the regression branch
significantly boosts the performance of the TL loss, resulting in the top
performance on all synthetic and human datasets (Figure 7). Consistent with
the observation from the previous section, TCL optimizes the inter-class
variance, which is further improved with the reconstruction loss; TL optimizes
the inner-class variance and strongly benefits from our reconstruction branch
(Table 1).Some visual results of the retrieval are provided in Figure 10.

###  5.5 Study of data abstraction level

In this section, we use different levels of abstraction to train the
PointNet++ with the combination of the classification and triplet losses
(Figure 11). It can be seen that training on the cleanest sketches
($l_{a}=0.0$), yellow dashed lines, generalizes the worst to the other
abstraction levels and human sketches, while training with the most abstract
sketches ($l_{a}=1.0$) results in a stable performance, though lower than the
optimal for each of abstractness levels. It gives nearly optimal performance
on human sketches. Interestingly, the performance on humans sketches when
training on sketches with $l_{a}=0.25$ outperforms the training on sketches
with $l_{a}=0.5$ and $l_{a}=0.75$, what might indicate that human participants
either did detailed and clean sketches or highly abstract ones. Drawing more
certain conclusions would though require a larger set of human sketches, which
includes more participants.

Model | mAP | NDCG | NN | Top-1
---|---|---|---|---
CL+TL: 3D$\rightarrow$3D NGVNN (2-branch) | 0.934 | 0.966 | 0.934 | 0.018
CL+TL: PointNet++ | 0.937 | 0.968 | 0.946 | 0.258
CL+TL+Rec: PointNet++ | 0.961 | 0.979 | 0.964 | 0.329
CL+TCL: PointNet++ ($m_{tcl}$=75) | 0.964 | 0.979 | 0.964 | 0.060
CL+TCL+Rec: PointNet++ ($m_{tcl}$=75) | 0.961 | 0.978 | 0.958 | 0.090
CL+TCL: PointNet++ ($m_{tcl}$=50) | 0.969 | 0.981 | 0.964 | 0.084
CL+TCL+Rec: PointNet++ ($m_{tcl}$=50) | 0.983 | 0.990 | 0.982 | 0.138

Table 1: Comparison of all the models on human sketches.

###  5.6 Ablation study: 2D sketch vs 3D sketch

Figure 12: Comparison of the 2D sketch-based retrieval versus 3D sketch-based
retrieval (Section 5.6). The baselines are trained on the dataset with
$l_{a}=0.5$.

We additionally compare the performance of the 3D sketch retrieval models to a
performance of a 2D sketch retrieval model. We use NGVNN as a backbone to
build the 2D-sketch based shape retrieval baseline (2D$\rightarrow$3D NGVNN),
where we use NGVNN to encode 3D shape and additional branch to encode a 2D
sketch, which is a common practice for 2D sketch/image based retrieval [13].
The 2D branch uses the same VGG11 backbone as NGVNN. The input for the 2D
sketch branch is the view rendered with the camera azimuth angle equal to
$45^{\circ}$. It can be seen in Figure 12 that the 2D sketch-based retrieval
is outperformed by all the models for 3D sketch-based retrieval, advocating
for the 3D sketch-based retrieval.

##  6 Conclusion

In this work we proposed the problem of 3D VR-sketch to 3D model retrieval. We
first introduced a purpose-built VR environment to collect VR-sketches and
conduct retrieval. We then collected a set of 3D human sketches for a subset
of two shape classes from ModelNet10. We further proposed an approach for
generating synthetic 3D sketches with different levels of abstraction, and
demonstrated that the methods trained on our synthetic data generalize well to
human sketches. Via a series of comprehensive evaluations, we find that
compared to 3D shape-based retrieval, point-based shape representation is
advantageous over the multi-view representation. At last, we propose an
architecture with an additional sketch regularizing branch that leads to a
superior performance over all the considered baselines, demonstrating the
benefit of directly tackling the abstract nature of VR-sketches. We hope to
have offered a valid first stab at this new problem.

## References

  * [1] A. Avetisyan, A. Dai, and M. Nießner.  End-to-end cad model retrieval and 9dof alignment in 3d scans.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2551–2560, 2019.
  * [2] A. Brock, T. Lim, J. M. Ritchie, and N. Weston.  Generative and discriminative voxel modeling with convolutional neural networks.  arXiv preprint arXiv:1608.04236, 2016.
  * [3] P. Bénard and A. Hertzmann.  Line drawings from 3d models: A tutorial.  Foundations and Trends® in Computer Graphics and Vision, 11(1-2):1–159, 2019.
  * [4] L. Chu, H. Pan, Y. Liu, and W. Wang.  Repairing man-made meshes via visual driven global optimization with minimum intrusion.  ACM Trans. Graph. (SIGGRAPH ASIA), 38(6):158:1–158:18, 2019.
  * [5] M. Dahnert, A. Dai, L. J. Guibas, and M. Nießner.  Joint embedding of 3d scan and cad objects.  In Proceedings of the IEEE International Conference on Computer Vision, pages 8749–8758, 2019.
  * [6] H. Fan, H. Su, and L. J. Guibas.  A point set generation network for 3d object reconstruction from a single image.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 605–613, 2017.
  * [7] Y. Feng, Z. Zhang, X. Zhao, R. Ji, and Y. Gao.  Gvcnn: Group-view convolutional neural networks for 3d shape recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 264–272, 2018.
  * [8] D. Giunchi, S. James, and A. Steed.  3d sketching for interactive model retrieval in virtual reality.  In Proceedings of the Joint Symposium on Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering, pages 1–12, 2018.
  * [9] G. Gori, A. Sheffer, N. Vining, E. Rosales, N. Carr, and T. Ju.  Flowrep: Descriptive curve networks for free-form design shapes.  ACM Transactions on Graphics (TOG), 36(4):1–14, 2017.
  * [10] A. Grabner, P. M. Roth, and V. Lepetit.  3d pose estimation and 3d model retrieval for objects in the wild.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3022–3031, 2018.
  * [11] A. Grabner, P. M. Roth, and V. Lepetit.  Location field descriptors: Single image 3d model retrieval in the wild.  In 2019 International Conference on 3D Vision (3DV), pages 583–593. IEEE, 2019.
  * [12] D. Ha and D. Eck.  A neural representation of sketch drawings.  In International Conference on Learning Representations (ICLR), 2018\.
  * [13] X. He, T. Huang, S. Bai, and X. Bai.  View n-gram network for 3d object retrieval.  In Proceedings of the IEEE International Conference on Computer Vision, pages 7515–7524, 2019.
  * [14] X. He, Y. Zhou, Z. Zhou, S. Bai, and X. Bai.  Triplet-center loss for multi-view 3d object retrieval.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1945–1954, 2018.
  * [15] T. Lee, Y.-L. Lin, H. Chiang, M.-W. Chiu, W. Hsu, and P. Huang.  Cross-domain image-based 3d shape retrieval by view sequence learning.  In 2018 International Conference on 3D Vision (3DV), pages 258–266. IEEE, 2018.
  * [16] B. Li, Y. Lu, F. Duan, S. Dong, Y. Fan, L. Qian, H. Laga, H. Li, Y. Li, P. Lui, et al.  Shrec’16 track: 3d sketch-based 3d shape retrieval.  2016\.
  * [17] B. Li, Y. Lu, A. Ghumman, B. Strylowski, M. Gutierrez, S. Sadiq, S. Forster, N. Feola, and T. Bugerin.  3d sketch-based 3d model retrieval.  In Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, pages 555–558, 2015.
  * [18] Z. Li, C. Xu, and B. Leng.  Angular triplet-center loss for multi-view 3d shape retrieval.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8682–8689, 2019.
  * [19] K. Z. Lin, W. Xu, Q. Sun, C. Theobalt, and T.-S. Chua.  Learning a disentangled embedding for monocular 3d shape retrieval and pose estimation.  arXiv preprint arXiv:1812.09899, 2018.
  * [20] A. Liu, S. Xiang, W. Li, W. Nie, and Y. Su.  Cross-domain 3d model retrieval via visual domain adaptation.  In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 828–834. AAAI Press, 2018.
  * [21] M. Lyon, M. Campen, D. Bommes, and L. Kobbelt.  Parametrization quantization with free boundaries for trimmed quad meshing.  ACM Transactions on Graphics (TOG), 38(4):1–14, 2019.
  * [22] Y. Ma, B. Zheng, Y. Guo, Y. Lei, and J. Zhang.  Boosting multi-view convolutional neural networks for 3d object recognition via view saliency.  In Chinese Conference on Image and Graphics Technologies, pages 199–209. Springer, 2017.
  * [23] D. Maturana and S. Scherer.  Voxnet: A 3d convolutional neural network for real-time object recognition.  In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 922–928. IEEE, 2015.
  * [24] C. R. Qi, H. Su, K. Mo, and L. J. Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017.
  * [25] C. R. Qi, H. Su, M. Nießner, A. Dai, M. Yan, and L. J. Guibas.  Volumetric and multi-view cnns for object classification on 3d data.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5648–5656, 2016.
  * [26] C. R. Qi, L. Yi, H. Su, and L. J. Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  In Advances in neural information processing systems, pages 5099–5108, 2017.
  * [27] Y. Qi, Y.-Z. Song, H. Zhang, and J. Liu.  Sketch-based image retrieval via siamese convolutional neural network.  In 2016 IEEE International Conference on Image Processing (ICIP), pages 2460–2464. IEEE, 2016.
  * [28] N.-p. Ramer-Douglas-Peucker.  Ramer-douglas-peucker algorithm.  1972\.
  * [29] S. Ravanbakhsh, J. Schneider, and B. Poczos.  Deep learning with sets and point clouds.  arXiv preprint arXiv:1611.04500, 2016.
  * [30] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [31] H. Su, S. Maji, E. Kalogerakis, and E. Learned-Miller.  Multi-view convolutional neural networks for 3d shape recognition.  In Proceedings of the IEEE international conference on computer vision, pages 945–953, 2015.
  * [32] M. A. Uy, J. Huang, M. Sung, T. Birdal, and L. Guibas.  Deformation-aware 3d model embedding and retrieval.  arXiv preprint arXiv:2004.01228, 2020.
  * [33] F. Wang, L. Kang, and Y. Li.  Sketch-based 3d shape retrieval using convolutional neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1875–1883, 2015.
  * [34] J. Wang, Y. Song, T. Leung, C. Rosenberg, J. Wang, J. Philbin, B. Chen, and Y. Wu.  Learning fine-grained image similarity with deep ranking.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1386–1393, 2014.
  * [35] Y. Wen, K. Zhang, Z. Li, and Y. Qiao.  A discriminative feature learning approach for deep face recognition.  In European conference on computer vision, pages 499–515. Springer, 2016.
  * [36] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao.  3d shapenets: A deep representation for volumetric shapes.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912–1920, 2015.
  * [37] Y. Ye, B. Li, and Y. Lu.  3d sketch-based 3d model retrieval with convolutional neural network.  In 2016 23rd International Conference on Pattern Recognition (ICPR), pages 2936–2941. IEEE, 2016.
  * [38] Q. Yu, F. Liu, Y.-Z. Song, T. Xiang, T. M. Hospedales, and C.-C. Loy.  Sketch me that shoe.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 799–807, 2016.