BCL [10] | 53.97$\pm$0.10 | 18.31$\pm$0.05 | 58.14$\pm$0.48 | 23.10$\pm$0.27 | 92.59$\pm$0.26 | 90.97$\pm$0.31 | 65.99$\pm$0.21 | 45.56$\pm$0.16
Ours | 53.75$\pm$0.28 | 18.63$\pm$1.56 | 58.13$\pm$0.27 | 23.42$\pm$1.46 | 92.43$\pm$0.29 | 91.39$\pm$1.43 | 66.87$\pm$0.19 | 45.86$\pm$1.58
Ours (FFHQ) | 57.00$\pm$0.89 | 31.24$\pm$1.95 | 61.79$\pm$0.97 | 35.90$\pm$1.85 | 95.90$\pm$0.90 | 95.69$\pm$1.74 | 68.48$\pm$1.11 | 58.45$\pm$2.50
Ours (Synthetic) | 57.40$\pm$1.07 | 34.46$\pm$1.03 | 61.33$\pm$0.94 | 37.24$\pm$1.31 | 96.06$\pm$1.10 | 95.22$\pm$1.21 | 71.18$\pm$0.89 | 62.84$\pm$1.02
Method | HeavyMakeup | HighCheekbone | MouthSlightlyOpen | NarrowEyes
Unbiased | Bias-conflicting | Unbiased | Bias-conflicting | Unbiased | Bias-conflicting | Unbiased | Bias-conflicting
Baseline | 59.50$\pm$0.12 | 30.09$\pm$0.53 | 54.17$\pm$0.95 | 10.12$\pm$1.50 | 62.86$\pm$0.91 | 26.49$\pm$1.35 | 55.42$\pm$0.87 | 8.05$\pm$1.41
LNL [5] | 63.64$\pm$0.00 | 26.82$\pm$0.91 | 50.89$\pm$0.36 | 7.38$\pm$0.58 | 61.65$\pm$0.36 | 23.77$\pm$0.57 | 50.18$\pm$0.24 | 5.26$\pm$0.52
DI [6] | 59.77$\pm$0.17 | 30.26$\pm$0.08 | 55.75$\pm$0.46 | 10.59$\pm$0.48 | 65.60$\pm$0.49 | 27.10$\pm$0.45 | 54.20$\pm$0.35 | 8.54$\pm$0.34
LfF [7] | 62.84$\pm$0.95 | 28.54$\pm$3.65 | 51.18$\pm$0.29 | 9.13$\pm$1.44 | 59.94$\pm$0.30 | 25.44$\pm$1.52 | 50.53$\pm$0.35 | 6.92$\pm$1.48
EnD [8] | 62.32$\pm$1.36 | 29.47$\pm$2.54 | 53.32$\pm$0.21 | 9.14$\pm$0.92 | 62.14$\pm$0.44 | 25.44$\pm$0.82 | 52.75$\pm$0.54 | 6.85$\pm$0.84
CSAD [9] | 63.35$\pm$1.02 | 30.89$\pm$1.66 | 49.80$\pm$1.37 | 7.65$\pm$2.04 | 58.52$\pm$1.47 | 23.93$\pm$2.09 | 49.02$\pm$1.34 | 5.44$\pm$2.10
BCL [10] | 54.55$\pm$3.21 | 31.32$\pm$2.83 | 55.32$\pm$0.58 | 10.78$\pm$0.52 | 65.08$\pm$0.57 | 27.14$\pm$0.31 | 53.72$\pm$0.57 | 8.58$\pm$0.55
Ours | 64.91$\pm$1.32 | 31.36$\pm$2.43 | 55.17$\pm$0.25 | 11.10$\pm$1.57 | 63.89$\pm$0.30 | 27.46$\pm$1.47 | 55.63$\pm$0.25 | 8.99$\pm$1.43
Ours (FFHQ) | 71.59$\pm$0.81 | 38.64$\pm$1.57 | 58.49$\pm$0.94 | 24.85$\pm$1.91 | 67.15$\pm$0.98 | 40.01$\pm$2.85 | 58.08$\pm$1.09 | 21.50$\pm$1.93
Ours (Synthetic) | 69.32$\pm$0.95 | 40.91$\pm$1.54 | 58.83$\pm$1.13 | 23.60$\pm$1.01 | 67.66$\pm$0.94 | 41.36$\pm$1.10 | 57.77$\pm$0.92 | 22.67$\pm$0.49
Method | PaleSkin | PointyNose | Smiling | StraightHair
Unbiased | Bias-conflicting | Unbiased | Bias-conflicting | Unbiased | Bias-conflicting | Unbiased | Bias-conflicting
Baseline | 61.20$\pm$0.92 | 21.62$\pm$1.34 | 52.23$\pm$1.02 | 5.47$\pm$1.41 | 61.54$\pm$0.91 | 24.76$\pm$1.45 | 53.83$\pm$1.01 | 4.99$\pm$1.35
LNL [5] | 59.06$\pm$0.36 | 18.89$\pm$0.48 | 49.08$\pm$0.26 | 2.89$\pm$0.50 | 58.36$\pm$0.40 | 22.05$\pm$0.51 | 48.67$\pm$0.35 | 2.59$\pm$0.45
DI [6] | 63.05$\pm$0.49 | 22.09$\pm$0.41 | 52.87$\pm$0.35 | 6.14$\pm$0.50 | 62.22$\pm$0.40 | 25.27$\pm$0.33 | 52.54$\pm$0.48 | 5.79$\pm$0.36
LfF [7] | 58.35$\pm$0.40 | 20.54$\pm$1.53 | 49.29$\pm$0.27 | 4.39$\pm$1.58 | 58.58$\pm$0.44 | 23.70$\pm$1.50 | 49.09$\pm$0.31 | 4.03$\pm$1.53
EnD [8] | 60.52$\pm$0.52 | 20.70$\pm$0.86 | 51.42$\pm$0.50 | 4.60$\pm$0.89 | 60.72$\pm$0.47 | 23.76$\pm$0.98 | 51.22$\pm$0.48 | 4.03$\pm$0.89
CSAD [9] | 56.93$\pm$1.45 | 19.16$\pm$2.10 | 47.83$\pm$1.40 | 2.98$\pm$2.05 | 57.18$\pm$1.52 | 22.22$\pm$2.00 | 47.54$\pm$1.47 | 2.38$\pm$2.12
BCL [10] | 62.46$\pm$0.59 | 22.33$\pm$0.52 | 53.50$\pm$0.57 | 6.10$\pm$0.54 | 61.82$\pm$1.59 | 25.45$\pm$0.07 | 52.12$\pm$0.59 | 5.87$\pm$0.54
Ours | 62.44$\pm$0.20 | 22.66$\pm$1.60 | 53.21$\pm$0.20 | 6.53$\pm$1.49 | 62.66$\pm$0.24 | 25.78$\pm$1.48 | 53.01$\pm$0.20 | 6.00$\pm$1.53
Ours (FFHQ) | 65.94$\pm$1.02 | 35.14$\pm$1.85 | 56.58$\pm$0.90 | 18.95$\pm$2.30 | 65.78$\pm$1.06 | 38.27$\pm$2.01 | 56.29$\pm$0.99 | 18.45$\pm$2.31
Ours (Synthetic) | 65.63$\pm$0.89 | 36.40$\pm$0.57 | 57.03$\pm$1.09 | 20.32$\pm$1.31 | 66.20$\pm$1.07 | 39.66$\pm$1.28 | 58.59$\pm$1.05 | 21.89$\pm$1.25
Method | WavyHair | WearingHat | Young | Average
Unbiased | Bias-conflicting | Unbiased | Bias-conflicting | Unbiased | Bias-conflicting | Unbiased | Bias-conflicting
Baseline | 52.81$\pm$0.89 | 6.08$\pm$1.43 | 77.45$\pm$0.96 | 52.49$\pm$1.39 | 50.61$\pm$2.56 | 15.51$\pm$3.66 | 59.03$\pm$0.96 | 21.53$\pm$1.42
LNL [5] | 49.60$\pm$0.22 | 3.47$\pm$0.54 | 72.38$\pm$0.23 | 49.78$\pm$0.46 | 42.66$\pm$1.01 | 11.30$\pm$0.44 | 55.84$\pm$0.31 | 18.81$\pm$0.53
DI [6] | 53.58$\pm$0.34 | 6.63$\pm$0.41 | 76.17$\pm$0.41 | 53.06$\pm$0.44 | 52.47$\pm$0.76 | 13.37$\pm$0.72 | 59.73$\pm$0.43 | 22.03$\pm$0.42
LfF [7] | 49.99$\pm$0.27 | 5.04$\pm$1.59 | 72.73$\pm$0.41 | 51.52$\pm$1.64 | 47.20$\pm$0.19 | 16.80$\pm$0.67 | 56.12$\pm$0.35 | 20.45$\pm$1.54
EnD [8] | 52.13$\pm$0.37 | 5.17$\pm$0.85 | 74.83$\pm$0.39 | 51.45$\pm$0.80 | 52.65$\pm$0.58 | 14.93$\pm$0.40 | 58.32$\pm$0.47 | 20.48$\pm$0.89
CSAD [9] | 48.42$\pm$1.44 | 3.52$\pm$2.01 | 71.08$\pm$1.51 | 49.91$\pm$3.14 | 43.37$\pm$0.64 | 11.24$\pm$1.46 | 54.65$\pm$1.43 | 18.93$\pm$2.07
BCL [10] | 53.06$\pm$0.56 | 6.75$\pm$0.16 | 75.74$\pm$0.21 | 53.24$\pm$0.16 | 50.74$\pm$0.10 | 12.99$\pm$1.65 | 59.28$\pm$0.58 | 22.16$\pm$0.53
Ours | 53.93$\pm$0.26 | 7.15$\pm$1.56 | 76.72$\pm$0.18 | 53.47$\pm$1.53 | 51.21$\pm$0.13 | 17.60$\pm$1.72 | 60.13$\pm$0.27 | 22.45$\pm$1.52
Ours (FFHQ) | 57.55$\pm$1.02 | 20.83$\pm$1.27 | 79.85$\pm$0.97 | 65.95$\pm$1.83 | 52.47$\pm$0.87 | 35.58$\pm$2.28 | 63.43$\pm$0.98 | 34.98$\pm$1.93
Ours (Synthetic) | 57.28$\pm$0.94 | 19.66$\pm$1.98 | 82.24$\pm$1.31 | 69.25$\pm$1.29 | 55.01$\pm$1.47 | 39.89$\pm$0.29 | 63.76$\pm$1.03 | 36.29$\pm$1.24

TABLE XVI: Performance of attribute bias removal methods under _moderate_
attribute bias in CelebA (_TrainOri_ training set). Bold for the best results.

Method | BagsUnderEyes | Bangs | BigLips | BigNose
---|---|---|---|---
Unbiased | Bias-conflicting | Unbiased | Bias-conflicting | Unbiased | Bias-conflicting | Unbiased | Bias-conflicting
Baseline | 76.87$\pm$0.80 | 71.97$\pm$0.99 | 91.06$\pm$0.78 | 91.39$\pm$1.11 | 62.00$\pm$0.78 | 58.22$\pm$1.02 | 81.66$\pm$0.73 | 70.04$\pm$0.98
LNL [5] | 71.85$\pm$0.66 | 76.85$\pm$1.17 | 92.28$\pm$0.82 | 90.33$\pm$1.27 | 63.16$\pm$0.80 | 58.13$\pm$1.29 | 81.22$\pm$0.78 | 71.68$\pm$1.21
DI [6] | 74.24$\pm$0.55 | 78.27$\pm$0.52 | 94.72$\pm$0.55 | 91.78$\pm$0.44 | 65.57$\pm$0.57 | 59.58$\pm$0.35 | 83.64$\pm$0.45 | 73.09$\pm$0.45
LfF [7] | 78.13$\pm$0.66 | 72.85$\pm$1.38 | 93.25$\pm$0.66 | 93.27$\pm$1.52 | 62.17$\pm$0.60 | 61.03$\pm$1.77 | 81.77$\pm$0.62 | 73.03$\pm$1.65
EnD [8] | 78.85$\pm$1.60 | 73.94$\pm$2.68 | 93.18$\pm$1.54 | 94.54$\pm$2.73 | 66.02$\pm$1.64 | 62.16$\pm$2.69 | 83.71$\pm$1.65 | 74.16$\pm$2.70
CSAD [9] | 78.27$\pm$1.76 | 73.23$\pm$2.89 | 94.33$\pm$1.81 | 93.66$\pm$2.93 | 63.36$\pm$1.79 | 61.49$\pm$2.92 | 82.91$\pm$1.81 | 73.40$\pm$2.91
BCL [10] | 74.42$\pm$0.13 | 79.67$\pm$1.84 | 94.88$\pm$0.10 | 93.14$\pm$1.82 | 65.78$\pm$0.11 | 60.92$\pm$1.84 | 83.85$\pm$0.14 | 74.47$\pm$1.81
Ours | 79.31$\pm$1.50 | 75.73$\pm$2.85 | 93.60$\pm$1.48 | 96.10$\pm$2.79 | 65.09$\pm$1.36 | 63.91$\pm$2.80 | 84.09$\pm$1.47 | 75.81$\pm$2.82
Ours (FFHQ) | 80.38$\pm$1.13 | 76.68$\pm$1.02 | 93.61$\pm$1.02 | 97.14$\pm$1.61 | 65.18$\pm$1.04 | 65.40$\pm$1.78 | 84.17$\pm$1.16 | 78.37$\pm$1.92
Ours (Synthetic) | 80.95$\pm$0.97 | 77.25$\pm$1.90 | 95.23$\pm$1.04 | 97.70$\pm$1.80 | 65.49$\pm$1.19 | 64.99$\pm$1.48 | 85.81$\pm$0.94 | 76.88$\pm$1.48
Method | BlackHair | BlondHair | Blurry | BrownHair
Unbiased | Bias-conflicting | Unbiased | Bias-conflicting | Unbiased | Bias-conflicting | Unbiased | Bias-conflicting
Baseline | 86.47$\pm$0.91 | 85.37$\pm$1.02 | 75.92$\pm$0.35 | 52.52$\pm$0.19 | 72.79$\pm$0.83 | 71.79$\pm$1.05 | 78.99$\pm$0.72 | 76.28$\pm$0.97
LNL [5] | 81.22$\pm$0.78 | 81.68$\pm$1.21 | 83.78$\pm$0.21 | 80.54$\pm$0.45 | 74.00$\pm$0.81 | 71.76$\pm$1.20 | 81.12$\pm$0.65 | 78.17$\pm$1.34
DI [6] | 83.64$\pm$0.45 | 83.09$\pm$0.45 | 89.75$\pm$0.56 | 86.38$\pm$0.53 | 76.39$\pm$0.48 | 73.18$\pm$0.18 | 82.58$\pm$0.63 | 77.61$\pm$0.47
LfF [7] | 86.59$\pm$0.67 | 88.16$\pm$1.68 | 84.24$\pm$0.37 | 81.24$\pm$1.38 | 73.05$\pm$0.70 | 74.72$\pm$1.68 | 81.22$\pm$0.60 | 79.14$\pm$1.66
EnD [8] | 88.33$\pm$1.63 | 89.32$\pm$2.68 | 91.21$\pm$0.22 | 87.45$\pm$1.06 | 74.70$\pm$1.68 | 75.92$\pm$2.82 | 81.06$\pm$1.64 | 80.33$\pm$2.77
CSAD [9] | 87.71$\pm$1.75 | 86.56$\pm$2.89 | 89.36$\pm$1.72 | 87.53$\pm$1.69 | 74.05$\pm$1.88 | 75.10$\pm$2.90 | 82.28$\pm$1.73 | 79.45$\pm$3.02
BCL [10] | 83.85$\pm$0.04 | 84.47$\pm$1.81 | 91.27$\pm$0.42 | 87.61$\pm$2.32 | 74.59$\pm$0.01 | 74.57$\pm$1.85 | 82.71$\pm$0.11 | 80.94$\pm$1.82
Ours | 89.85$\pm$1.44 | 91.04$\pm$2.89 | 89.81$\pm$0.45 | 85.29$\pm$1.54 | 75.25$\pm$1.38 | 77.62$\pm$3.33 | 81.37$\pm$1.38 | 82.12$\pm$2.90
Ours (FFHQ) | 90.54$\pm$1.12 | 91.09$\pm$1.61 | 90.86$\pm$0.87 | 88.06$\pm$0.91 | 75.93$\pm$0.96 | 78.14$\pm$1.06 | 81.51$\pm$1.14 | 81.57$\pm$1.77
Ours (Synthetic) | 89.83$\pm$1.10 | 92.51$\pm$1.92 | 90.01$\pm$0.65 | 88.72$\pm$1.16 | 76.33$\pm$1.05 | 78.63$\pm$1.55 | 83.01$\pm$0.94 | 83.11$\pm$1.54
Method | Chubby | DoubleChin | Eyeglasses | GrayHair
Unbiased | Bias-conflicting | Unbiased | Bias-conflicting | Unbiased | Bias-conflicting | Unbiased | Bias-conflicting
Baseline | 78.37$\pm$0.78 | 72.76$\pm$1.09 | 74.91$\pm$0.75 | 67.94$\pm$0.96 | 99.24$\pm$0.89 | 98.48$\pm$1.01 | 85.23$\pm$0.90 | 79.59$\pm$1.14
LNL [5] | 73.85$\pm$0.77 | 78.40$\pm$1.21 | 69.16$\pm$0.77 | 74.93$\pm$1.32 | 99.96$\pm$0.74 | 98.37$\pm$1.37 | 78.82$\pm$0.75 | 85.13$\pm$1.21
DI [6] | 76.20$\pm$0.48 | 79.79$\pm$0.34 | 71.54$\pm$0.45 | 76.34$\pm$0.47 | 99.97$\pm$0.59 | 99.79$\pm$0.47 | 81.21$\pm$0.44 | 86.55$\pm$0.46
LfF [7] | 78.64$\pm$0.74 | 75.46$\pm$1.54 | 75.23$\pm$0.61 | 70.81$\pm$1.76 | 99.31$\pm$0.64 | 98.33$\pm$1.78 | 85.36$\pm$0.75 | 80.49$\pm$1.74
EnD [8] | 80.32$\pm$1.51 | 76.81$\pm$2.75 | 74.99$\pm$1.60 | 72.11$\pm$2.67 | 99.21$\pm$1.68 | 99.57$\pm$2.67 | 87.16$\pm$1.54 | 81.87$\pm$2.75
CSAD [9] | 79.79$\pm$1.78 | 75.99$\pm$2.86 | 76.26$\pm$1.75 | 69.24$\pm$2.99 | 99.46$\pm$1.73 | 98.60$\pm$2.92 | 86.38$\pm$1.83 | 80.98$\pm$2.99
BCL [10] | 80.40$\pm$0.01 | 78.19$\pm$1.82 | 76.71$\pm$0.11 | 73.71$\pm$1.82 | 99.97$\pm$0.12 | 99.97$\pm$1.81 | 81.40$\pm$0.13 | 87.91$\pm$1.82
Ours | 80.77$\pm$1.49 | 78.45$\pm$2.79 | 77.51$\pm$1.39 | 73.78$\pm$2.75 | 99.23$\pm$1.08 | 99.17$\pm$2.77 | 87.61$\pm$1.37 | 83.59$\pm$2.76
Ours (FFHQ) | 81.51$\pm$0.99 | 78.55$\pm$1.62 | 78.42$\pm$1.09 | 73.89$\pm$1.57 | 99.63$\pm$1.17 | 99.32$\pm$1.45 | 88.65$\pm$1.07 | 85.02$\pm$1.84
Ours (Synthetic) | 81.83$\pm$1.05 | 80.04$\pm$1.88 | 78.96$\pm$0.96 | 75.32$\pm$1.84 | 99.72$\pm$1.51 | 99.80$\pm$1.88 | 89.24$\pm$1.04 | 84.57$\pm$1.55
Method | HeavyMakeup | HighCheekbone | MouthSlightlyOpen | NarrowEyes
Unbiased | Bias-conflicting | Unbiased | Bias-conflicting | Unbiased | Bias-conflicting | Unbiased | Bias-conflicting
Baseline | 62.00$\pm$0.02 | 33.75$\pm$0.28 | 84.99$\pm$0.84 | 80.69$\pm$1.02 | 94.17$\pm$0.89 | 94.58$\pm$1.04 | 68.93$\pm$0.83 | 65.74$\pm$1.13
LNL [5] | 66.73$\pm$0.09 | 45.37$\pm$0.37 | 86.13$\pm$0.82 | 80.71$\pm$1.26 | 96.13$\pm$0.82 | 90.71$\pm$1.26 | 68.92$\pm$0.75 | 69.75$\pm$1.31
DI [6] | 74.89$\pm$0.41 | 53.31$\pm$0.62 | 88.49$\pm$0.62 | 82.18$\pm$0.35 | 98.49$\pm$0.62 | 92.18$\pm$0.35 | 71.32$\pm$0.50 | 70.17$\pm$0.45
LfF [7] | 66.20$\pm$1.21 | 45.48$\pm$4.33 | 85.08$\pm$0.67 | 84.57$\pm$1.76 | 94.33$\pm$0.68 | 96.58$\pm$1.75 | 69.02$\pm$0.74 | 68.61$\pm$1.81
EnD [8] | 75.93$\pm$1.31 | 53.70$\pm$5.24 | 86.96$\pm$1.59 | 82.90$\pm$2.72 | 96.11$\pm$1.71 | 95.79$\pm$2.69 | 69.72$\pm$1.66 | 67.79$\pm$2.73
CSAD [9] | 67.88$\pm$2.54 | 53.44$\pm$3.49 | 86.16$\pm$1.82 | 83.98$\pm$2.95 | 95.50$\pm$1.90 | 95.96$\pm$2.87 | 70.12$\pm$1.74 | 67.95$\pm$3.01
BCL [10] | 77.26$\pm$0.48 | 54.15$\pm$0.89 | 88.68$\pm$0.11 | 85.52$\pm$1.83 | 98.68$\pm$0.11 | 96.52$\pm$1.83 | 71.50$\pm$0.11 | 72.54$\pm$1.81
Ours | 76.82$\pm$1.83 | 54.09$\pm$3.04 | 87.47$\pm$1.37 | 86.62$\pm$2.78 | 96.68$\pm$1.38 | 96.52$\pm$2.76 | 71.24$\pm$1.50 | 71.55$\pm$2.76
Ours (FFHQ) | 79.59$\pm$1.73 | 57.58$\pm$1.83 | 87.99$\pm$1.01 | 86.70$\pm$1.05 | 97.57$\pm$1.21 | 98.99$\pm$3.05 | 71.33$\pm$1.05 | 72.57$\pm$1.49
Ours (Synthetic) | 80.73$\pm$0.71 | 58.27$\pm$1.67 | 88.42$\pm$1.04 | 88.08$\pm$2.75 | 98.35$\pm$1.05 | 97.96$\pm$1.93 | 72.81$\pm$1.04 | 73.16$\pm$1.85
Method | PaleSkin | PointyNose | Smiling | StraightHair
Unbiased | Bias-conflicting | Unbiased | Bias-conflicting | Unbiased | Bias-conflicting | Unbiased | Bias-conflicting
Baseline | 75.97$\pm$0.86 | 66.73$\pm$1.11 | 62.70$\pm$0.85 | 56.05$\pm$1.13 | 92.76$\pm$0.84 | 92.28$\pm$0.96 | 72.74$\pm$0.76 | 68.23$\pm$0.96
LNL [5] | 77.01$\pm$0.83 | 66.70$\pm$1.22 | 63.90$\pm$0.72 | 55.92$\pm$1.33 | 93.71$\pm$0.83 | 91.27$\pm$1.27 | 69.33$\pm$0.68 | 72.67$\pm$1.26
DI [6] | 79.41$\pm$0.51 | 68.13$\pm$0.44 | 66.31$\pm$0.47 | 57.37$\pm$0.41 | 96.16$\pm$0.57 | 92.71$\pm$0.50 | 71.72$\pm$0.57 | 74.06$\pm$0.33
LfF [7] | 76.12$\pm$0.62 | 69.61$\pm$1.65 | 62.77$\pm$0.58 | 58.75$\pm$1.75 | 92.79$\pm$0.60 | 94.13$\pm$1.71 | 72.82$\pm$0.68 | 71.03$\pm$1.74
EnD [8] | 77.95$\pm$1.54 | 70.94$\pm$2.72 | 64.60$\pm$1.62 | 60.06$\pm$2.73 | 94.69$\pm$1.71 | 95.37$\pm$2.65 | 74.71$\pm$1.61 | 72.35$\pm$2.74
CSAD [9] | 77.20$\pm$1.75 | 70.10$\pm$3.01 | 63.78$\pm$1.86 | 59.23$\pm$2.94 | 93.97$\pm$1.90 | 94.65$\pm$2.98 | 74.04$\pm$1.73 | 71.49$\pm$2.85
BCL [10] | 78.61$\pm$0.11 | 70.48$\pm$1.85 | 66.51$\pm$0.01 | 58.72$\pm$1.82 | 96.34$\pm$0.12 | 94.03$\pm$1.83 | 71.91$\pm$0.11 | 75.46$\pm$1.83
Ours | 78.43$\pm$1.48 | 72.56$\pm$2.80 | 65.15$\pm$1.54 | 62.79$\pm$2.91 | 95.10$\pm$1.41 | 97.19$\pm$2.86 | 75.08$\pm$1.37 | 73.99$\pm$2.80
Ours (FFHQ) | 78.93$\pm$1.13 | 73.50$\pm$1.45 | 66.08$\pm$1.09 | 63.21$\pm$1.56 | 95.10$\pm$1.14 | 98.12$\pm$1.45 | 75.24$\pm$1.10 | 74.05$\pm$1.52
Ours (Synthetic) | 79.41$\pm$1.16 | 74.13$\pm$1.76 | 66.59$\pm$0.94 | 62.91$\pm$1.99 | 96.85$\pm$1.07 | 98.50$\pm$1.88 | 76.68$\pm$1.02 | 75.52$\pm$1.83
Method | WavyHair | WearingHat | Young | Average
Unbiased | Bias-conflicting | Unbiased | Bias-conflicting | Unbiased | Bias-conflicting | Unbiased | Bias-conflicting
Baseline | 73.94$\pm$0.82 | 66.47$\pm$0.95 | 95.05$\pm$0.85 | 93.51$\pm$0.94 | 49.08$\pm$2.08 | 38.17$\pm$2.87 | 78.08$\pm$0.82 | 71.85$\pm$1.04
LNL [5] | 75.13$\pm$0.80 | 65.48$\pm$1.30 | 97.66$\pm$0.78 | 95.88$\pm$1.28 | 58.82$\pm$1.59 | 45.26$\pm$3.10 | 78.43$\pm$0.75 | 75.03$\pm$1.27
DI [6] | 76.54$\pm$0.59 | 66.86$\pm$0.49 | 97.03$\pm$0.49 | 94.34$\pm$0.38 | 59.28$\pm$0.87 | 41.59$\pm$0.21 | 80.83$\pm$0.54 | 76.45$\pm$0.42
LfF [7] | 74.16$\pm$0.66 | 68.41$\pm$1.71 | 95.04$\pm$0.77 | 96.38$\pm$1.94 | 56.84$\pm$0.81 | 41.78$\pm$0.10 | 79.31$\pm$0.68 | 75.82$\pm$1.73
EnD [8] | 73.96$\pm$1.69 | 70.63$\pm$2.82 | 95.94$\pm$1.70 | 94.55$\pm$2.83 | 56.91$\pm$2.96 | 39.43$\pm$1.95 | 81.14$\pm$1.61 | 77.03$\pm$2.73
CSAD [9] | 75.31$\pm$1.84 | 66.75$\pm$2.86 | 96.28$\pm$1.85 | 95.80$\pm$2.95 | 55.95$\pm$1.59 | 38.02$\pm$3.80 | 80.45$\pm$1.82 | 76.20$\pm$2.94
BCL [10] | 76.72$\pm$0.12 | 68.21$\pm$1.84 | 97.25$\pm$0.02 | 95.67$\pm$1.85 | 50.17$\pm$0.03 | 42.76$\pm$2.33 | 81.02$\pm$0.12 | 77.81$\pm$1.83
Ours | 76.44$\pm$1.37 | 71.29$\pm$2.90 | 96.47$\pm$1.43 | 97.36$\pm$2.91 | 54.89$\pm$3.11 | 44.91$\pm$3.81 | 81.62$\pm$1.46 | 78.76$\pm$2.84
Ours (FFHQ) | 78.09$\pm$1.06 | 72.83$\pm$1.82 | 97.43$\pm$1.03 | 97.27$\pm$1.54 | 62.52$\pm$1.45 | 46.89$\pm$0.50 | 82.62$\pm$1.12 | 79.78$\pm$1.54
Ours (Synthetic) | 77.39$\pm$1.22 | 72.35$\pm$1.50 | 99.03$\pm$1.12 | 99.71$\pm$1.87 | 61.85$\pm$0.88 | 45.18$\pm$3.31 | 83.24$\pm$1.03 | 80.23$\pm$1.84

##  Appendix J Experimental Evaluation on IMDB

In this section, we present the complementary results on IMDB dataset [55] in
Tab. XVII, and the visualization of the filtered images on IMDB dataset in
Fig. 16. Furthermore, we conduct a comprehensive analysis of the results, both
quantitatively and qualitatively.

###  J-A Experiment Setup

Following the setup in [5], we conduct experiments on IMDB with age as the
protected attribute. It is worth noting that age is a twelve-class random
variable quantified with five-year steps in IMDB, which is different from the
binary protected attribute label (sex) in CelebA. To facilitate our analysis,
following data split in [5], we introduce three sub-datasets: (1) _EB1_
consisting of women aged 0-29 and men aged 40+, (2) _EB2_ consisting of women
aged 40+ and men aged 0-29, and (3) _Test_ which contains 20% of all cleaned
samples aged 0-29 and 40+. Besides, we construct a Universal dataset by
appending the biased dataset with the synthetic dataset generated by the
generative model [26].

TABLE XVII: Performance of attribute bias removal methods in IMDB. Bold for the best results. Trained on | EB1 | EB2
---|---|---
Tested on | EB2 | Test | EB1 | Test
Baseline | 59.9$\pm$0.0 | 84.4$\pm$0.1 | 57.8$\pm$0.0 | 69.8$\pm$0.0
BlindEye [4] | 63.7$\pm$1.8 | 85.6$\pm$1.1 | 57.3$\pm$1.6 | 69.9$\pm$1.3
LNL [5] | 68.0$\pm$2.0 | 86.7$\pm$0.9 | 64.2$\pm$1.1 | 74.5$\pm$1.1
BackMI [15] | 69.1$\pm$0.0 | 87.6$\pm$0.0 | 65.1$\pm$0.0 | 76.2$\pm$0.0
EnD [8] | 65.5$\pm$0.8 | 87.2$\pm$0.3 | 69.4$\pm$2.0 | 78.2$\pm$1.2
CSAD [9] | 70.4$\pm$2.8 | 87.0$\pm$1.1 | 68.1$\pm$1.8 | 78.7$\pm$1.9
Ours | 73.1$\pm$2.3 | 87.6$\pm$0.3 | 69.4$\pm$1.2 | 78.6$\pm$0.7
Ours (Universal) | 75.7$\pm$1.8 | 88.3$\pm$0.5 | 70.3$\pm$2.3 | 79.2$\pm$1.0

###  J-B Quantitatively Analysis

Compared with CelebA results in Tab. I of the main paper, we observe that all
models perform well on IMDB, as shown in Tab. XVII. To better analyze it, we
present detailed statistics in Figs. 22 and 21. According to the age label
division shown in Tab. XIX, we present boxplots of age labels for females and
males in Fig. 21. These boxplots show that _Test_ yields the same attribute
bias as _EB1_ so that the accuracy is high when trained on _EB1_ and tested on
_Test_. Moreover, we observe that the sex distribution of _EB2_ is severely
imbalanced according to the number of female samples (1744) and male samples
(15056) as shown in Fig. 22. Besides, the difference between samples with
closed age is not visually clear when comparing the younger females or younger
males with the older females or older males respectively in Fig. 24.
Furthermore, the majority age group of females is 25-29 in _EB1_ and 40-44 in
_EB2_ , and the majority age group of males is 40-44 in _EB1_ and 25-29 in
_EB2_. Thus, the dominated groups for each sex category in _EB1_ and _EB2_ are
visually indistinguishable. Therefore, the models may compromise the
classification performance with attribute bias removal w.r.t. age since age
information is hard to capture by models, thereby resulting in a good
performance for all methods.

###  J-C Qualitatively Analysis

As shown in Fig. 16, our model filters out age information by blurring the
image and adding wrinkles to images of young individuals, which further closes
the appearance gap between different age groups. Notably, there are several
outliers in the training set hindering the generation of high-quality images.
As shown in Fig. 24a, some samples do not contain an obvious face. Moreover,
in Fig. 24b, some samples contain more than one face. Besides, as identified
in [4], the age labels of IMDB are noisy. Overall, these obstacles negatively
affect the generation of high-quality filtered images for our approach.

##  Appendix K Experimental Evaluation on Waterbirds

We supplement the experiments on Waterbirds [38], where the target is
waterbird or landbird, and the attribute is water background or land
background. Following Group DRO [38], the training set contains more instances
of waterbirds with water backgrounds and landbirds with land backgrounds
compared to other combinations, whereas such a correlation does not exist in
the testing set. The networks of the filter and the assigned hyper-parameters
are the same as the experiments on CelebA, as shown in Tab. XXVI and Tab. XXI,
respectively. To construct the Universal dataset, we ensure an even number of
landbirds and waterbirds on both land and water backgrounds by utilizing
provided pixel-level segmentation masks to extract each bird from its original
background and then placing it onto water background or land background
sourced from the Places dataset [71]. Please see the qualitative results of
our method in Fig. 20 and quantitative comparison in Fig. 20.

##  Appendix L Application on More Modalities (CivilComment-WILDS)

In the main paper, our experiments include modalities beyond images, _e.g_.,
Adult dataset, which consists of census data. For NLP, following JTT [72], we
supplement the results on CivilComments-WILDS, an NLP task that classifies
online comments as toxic or non-toxic, where target labels are spuriously
correlated with mentions of certain demographic identities. We assign hyper-
parameters of the filter ($\lambda_{mi}$, $\lambda_{pred}$, and
$\lambda_{rec}$) to be 5, 5, and 10, respectively. Other hyper-parameters are
configured following [57]. Consequently, we set a maximum of 300 tokens for
the token-type embedding of the original text and ensure the same number of
tokens for both reconstructed text and filtered text. The filter is designed
to remove words associated with specific demographic identities. The filter
networks remain consistent with the Adult experiments, with the exception of
adjusting the input dimension to 300, as shown in Tab. XXV. To construct the
Universal dataset, we build a subset by including the samples from all 16
groups categorized by target (toxic or non-toxic) and all 8 demographic
identities. Please see Fig. 20 for the results.

Figure 19: Average and worst-group test accuracies on Waterbirds and
CivilComments-WILDS.

Model | Waterbirds | CivilComments-WILDS
---|---|---
Avg | Worst-group | Avg | Worst-group
Baseline | 97.26$\pm$0.20 | 72.60$\pm$1.90 | 92.64$\pm$0.52 | 57.42$\pm$0.73
LfF [7] | 91.22$\pm$0.85 | 78.04$\pm$1.83 | 92.52$\pm$0.91 | 58.81$\pm$1.23
Group DRO [38] | 93.50$\pm$0.30 | 91.40$\pm$1.10 | 88.91$\pm$0.28 | 69.94$\pm$0.52
JTT [72] | 93.34$\pm$0.56 | 86.72$\pm$0.97 | 91.14$\pm$0.34 | 69.36$\pm$0.89
Ours | 93.37$\pm$0.81 | 92.06$\pm$1.58 | 91.26$\pm$0.95 | 69.51$\pm$0.71
Ours (Universal) | 94.24$\pm$0.92 | 93.21$\pm$1.43 | 92.42$\pm$1.43 | 70.25$\pm$0.56

Figure 20: For the origin image with a land background, the background of the
filtered image tends to shift towards a blue hue, bridging the gap between
land and water backgrounds, since blue is the predominant color for water
backgrounds. On the other hand, for the original image with a water
background, the background of the filtered image tends to become blurred and
yield the texture of leaves.

##  Appendix M Comparison with Group DRO and DFA

Group DRO [38] emphasizes the role of regularization in distributionally
robust optimization to prevent overfitting and enhance worst-group
performance. Our approach surpasses Group DRO by not demanding target labels
during training, which becomes valuable when obtaining target labels is
challenging. Thus, we believe our approach is more applicable to address
attribute bias. Besides, DFA [73] introduces feature-level data augmentation
to generate diverse bias-conflicting samples by disentangling intrinsic and
bias attributes from bias-aligned samples. In contrast, our approach conducts
in-place image editing, eliminating the need for augmenting the dataset with
additional bias-conflicting samples. This property is advantageous especially
for non-binary bias attributes since generating bias-conflicting samples with
combinations of target and all other bias attributes, can be time-consuming.

We supplement the empirical comparison with them on several datasets in Tab.
XVIII. The implementation follows the same guidelines as our main paper. As
shown in Tab. XVIII, Group DRO trade-off average performance with enhanced
worst-group performance. One possible reason is that the regularization in
Group DRO may overemphasize the worst group during training in extreme bias
where the worst group performance is notably compromised.

TABLE XVIII: Comparison under extreme bias between DFA, Group DRO, GAN-
Debiasing, and our method based on testing accuracy in CelebA.

Method | BlondHair | HeavyMakeup
---|---|---
Unbiased ↑ | Bias-conflicting ↑ | Unbiased ↑ | Bias-conflicting ↑
Baseline | 66.11$\pm$0.32 | 33.89$\pm$0.45 | 59.50$\pm$0.12 | 30.09$\pm$0.53
DFA [73] | 65.93$\pm$0.43 | 32.18$\pm$0.63 | 63.74$\pm$0.43 | 30.69$\pm$0.97
Group DRO [38] | 63.82$\pm$0.81 | 34.23$\pm$0.94 | 61.43$\pm$1.82 | 30.92$\pm$1.62
Ours | 66.31$\pm$0.26 | 32.22$\pm$0.43 | 64.91$\pm$1.32 | 31.36$\pm$2.43
Ours (FFHQ) | 71.53$\pm$0.67 | 47.17$\pm$0.72 | 71.59$\pm$0.81 | 38.64$\pm$1.57
Ours (Synthetic) | 71.37$\pm$0.64 | 48.06$\pm$0.82 | 69.32$\pm$0.95 | 40.91$\pm$1.54

##  Appendix N Related Work about Domain-Invariant Representation Learning
Approaches

In the main paper, we provide related work about mutual information-based
methods [5, 15, 9], which provides a good basis to draw conclusions about
domain shift works. In this section, we cover a more comprehensive list of
related work about domain-invariant representation learning approaches. Ganin
_et al_. [33] propose Domain-Adversarial Neural Networks (DANN) for domain
adaptation, which are trained to be discriminative for the primary task in the
source domain and domain-indiscriminate to handle domain shifts. Zhao _et al_.
[34] introduce an upper bound for conditional shift (the variation in class-
conditional distributions of input features between the source and target
domains). Albuquerque _et al_. [35] present an adversarial domain
generalization method to minimize risk across unseen domains by mapping data
to a space where training distributions are indistinguishable while retaining
task-relevant information.

TABLE XIX: Division of the age label in IMDB.

Age | 0-19 | 20-24 | 25-29 | 30-34 | 35-39 | 40-44 | 45-49 | 50-54 | 55-59 | 60-64 | 65-60 | 70+
---|---|---|---|---|---|---|---|---|---|---|---|---
Label | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11

(a) EB1.

(b) EB2.

(c) Test.

Figure 21: Boxplots of age label distribution across sex in IMDB.

(a) EB1 with 24997 females and 11007 males.

(b) EB2 with 1744 females and 15056 males.

(c) Test with 10753 females and 11715 males.

Figure 22: Statistics for the number of females and males in different age
categories of IMDB.

(a) EB1 dominated by females aged 25-29 and males aged 40-44.

(b) EB2 dominated by females aged 40-44 and males aged 25-29.

Figure 23: Comparison of dominated groups in EB1 and EB2.

(a) Examples taken from distance view.

(b) Examples with two faces.

Figure 24: Noisy examples of IMDB for sex classification.

##  Appendix O Training Details

In this section, we present the implementation details of our filter and the
downstream task classifier on Colored MNIST, CelebA, Adult, and IMDB. The
experiments are conducted on one NVIDIA 1080 Ti GPU. The reference time spent
for each experiment is shown in Tab. XX.

###  O-A Filter Training

The network architectures of our filter on Colored MNIST are shown in Tab.
XXIV, the architectures on CelebA and IMDB are shown in Tab. XXVI, and the
architecture on Adult is shown in Tab. XXV. We optimize our filter by Adam
optimizer [74] with default settings $(\beta_{1}=0,\beta_{2}=0.9)$ in WGAN-GP
[53]. The hyper-parameters are shown in Tab. XXI.

TABLE XX: Time statistics based on one NVIDIA 1080 Ti GPU. The wavy line $\sim$ represents the approximation. Experiment | Resolution | Total image | Filter training | Classifier training
---|---|---|---|---
With filter | Without filter
Colored MNIST | 32$\times$32 | 50000 |  $\sim$ 1h 10min |  $\sim$ 26min |  $\sim$ 21min
CelebA | 224$\times$224 |  $\sim$ 180000 |  $\sim$ 1d 2h |  $\sim$ 6h 52min |  $\sim$ 5h 18min
IMDB | 224$\times$224 |  $\sim$ 50000 |  $\sim$ 7h 31min |  $\sim$ 3h 22min |  $\sim$ 2h 28min
Adult | 108 |  $\sim$ 50000 |  $\sim$ 23min |  $\sim$ 5min |  $\sim$ 4min
TABLE XXI: Hyper-parameters for filter-based approach training. Experiment | $\lambda_{mi}$ | $\lambda_{pred}$ | $\lambda_{rec}$ | Learning rate | Batch size | Optimizer
---|---|---|---|---|---|---
Colored MNIST | 10 | 100 | 100 | 0.0003 | 32 | Adam ($\beta_{1}$=0, $\beta_{2}$=0.9)
CelebA | 50 | 50 | 100 | 0.0003 | 32 | Adam ($\beta_{1}$=0, $\beta_{2}$=0.9)
IMDB | 50 | 50 | 100 | 0.0003 | 32 | Adam ($\beta_{1}$=0, $\beta_{2}$=0.9)
Adult | 5 | 5 | 10 | 0.0003 | 32 | Adam ($\beta_{1}$=0, $\beta_{2}$=0.9)
TABLE XXII: Hyper-parameters for downstream task classifier training. Experiment | Learning rate | Batch size | Optimizer
---|---|---|---
Colored MNIST | 0.001 | 32 | Adam ($\beta_{1}$=0.9, $\beta_{2}$=0.999)
CelebA | 0.001 | 32 | Adam ($\beta_{1}$=0.9, $\beta_{2}$=0.999)
IMDB | 0.001 | 32 | Adam ($\beta_{1}$=0.9, $\beta_{2}$=0.999)
Adult | 0.001 | 32 | Adam ($\beta_{1}$=0.9, $\beta_{2}$=0.999)
TABLE XXIII: Hyper-parameters for SSL-based approach training. Experiment | Temperature | Learning rate | Batch size | Optimizer
---|---|---|---|---
CelebA | 0.07 | 0.001 | 32 | Adam ($\beta_{1}$=0.9, $\beta_{2}$=0.999)

###  O-B Downstream Task Classifier Training

For digit classification on Colored MNIST, we use Lenet-5 [75] as the backbone
classifier. For facial attribute classification on CelebA and binary sex
classification on IMDB, following CSAD [9], we use ResNet-18 [59] as the
backbone classifier. For income prediction on Adult dataset, we use a three-
layer MLP as the backbone classifier. The hyper-parameters are shown in Tab.
XXII.

TABLE XXIV: Networks of the target-agnostic adversarial filter on Colored MNIST. Layer | Kernel | Stride | Padding | Output shape | Activation | Normalization
---|---|---|---|---|---|---
Encoder $G_{enc}$
Input | - | - | - | 3 $\times$ 32 $\times$ 32 | - | -
Convolution | 4$\times$4 | 2$\times$2 | 1 | 64$\times$16$\times$16 | LeakyReLU | BatchNorm
Convolution | 4$\times$4 | 2$\times$2 | 1 | 128$\times$8$\times$8 | LeakyReLU | BatchNorm
Convolution | 4$\times$4 | 2$\times$2 | 1 | 256$\times$4$\times$4 | LeakyReLU | BatchNorm
Decoder $G_{dec}$
TransposedConv | 4$\times$4 | 2$\times$2 | 1 | 256$\times$8$\times$8 | ReLU | BatchNorm
TransposedConv | 4$\times$4 | 2$\times$2 | 1 | 128$\times$16$\times$16 | ReLU | BatchNorm
TransposedConv | 4$\times$4 | 2$\times$2 | 1 | 3$\times$32$\times$32 | Sigmoid | BatchNorm
Critic network $D$
Convolution | 4$\times$4 | 2$\times$2 | 1 | 64$\times$16$\times$16 | LeakyReLU | InstanceNorm
Convolution | 4$\times$4 | 2$\times$2 | 1 | 128$\times$8$\times$8 | LeakyReLU | InstanceNorm
Convolution | 4$\times$4 | 2$\times$2 | 1 | 256$\times$4$\times$4 | LeakyReLU | InstanceNorm
Linear | - | - | - | 256 | ReLU | InstanceNorm
Linear | - | - | - | 1 | - | -
Regressor $R$
Convolution | 5$\times$5 | 1$\times$1 | 0 | 6$\times$28$\times$28 | LeakyReLU | -
AveargePooling | 2$\times$2 | 2$\times$2 | - | 6$\times$14$\times$14 | - | -
Convolution | 5$\times$5 | 1$\times$1 | 0 | 16$\times$10$\times$10 | LeakyReLU | -
AveargePooling | 2$\times$2 | 2$\times$2 | - | 16$\times$5$\times$5 | - | -
Convolution | 5$\times$5 | 1$\times$1 | 0 | 120$\times$1$\times$1 | LeakyReLU | -
Linear | - | - | - | 84 | ReLU | -
Linear | - | - | - | 3 | Sigmoid | -
Auxiliary network $T$
Linear | - | - | - | 1024 | LeakyReLU | -
Linear | - | - | - | 256 | LeakyReLU | -
Linear | - | - | - | 64 | LeakyReLU | -
Linear | - | - | - | 10 | LeakyReLU | -
Linear | - | - | - | 1 | - | -
TABLE XXV: Networks of the target-agnostic adversarial filter on Adult. Layer | Kernel | Stride | Padding | Output shape | Activation | Normalization
---|---|---|---|---|---|---
Encoder $G_{enc}$
Input | - | - | - | 108 | - | -
Linear | - | - | - | 64 | ReLU | -
Linear | - | - | - | 10 | ReLU | -
Decoder $G_{dec}$
Linear | - | - | - | 64 | ReLU | -
Linear | - | - | - | 108 | - | -
Critic network $D$
Linear | - | - | - | 64 | ReLU | -
Linear | - | - | - | 1 | - | -
Regressor $R$
Linear | - | - | - | 64 | ReLU | -
Linear | - | - | - | 1 | Sigmoid | -
Auxiliary network $T$
Linear | - | - | - | 10 | LeakyReLU | -
Linear | - | - | - | 1 | - | -
TABLE XXVI: Networks of the target-agnostic adversarial filter on CelebA, IMDB. Layer | Kernel | Stride | Padding | Output shape | Activation | Normalization
---|---|---|---|---|---|---
Encoder $G_{enc}$
Input | - | - | - | 3$\times$224$\times$224 | - | -
Convolution | 4$\times$4 | 2$\times$2 | 1 | 64$\times$112$\times$112 | LeakyReLU | BatchNorm
Convolution | 4$\times$4 | 2$\times$2 | 1 | 128$\times$56$\times$56 | LeakyReLU | BatchNorm
Convolution | 4$\times$4 | 2$\times$2 | 1 | 256$\times$28$\times$28 | LeakyReLU | BatchNorm
Convolution | 4$\times$4 | 2$\times$2 | 1 | 512$\times$14$\times$14 | LeakyReLU | BatchNorm
Convolution | 4$\times$4 | 2$\times$2 | 1 | 1024$\times$7$\times$7 | LeakyReLU | BatchNorm
Decoder $G_{dec}$
TransposedConv | 4$\times$4 | 2$\times$2 | 1 | 1024$\times$14$\times$14 | ReLU | BatchNorm
TransposedConv | 4$\times$4 | 2$\times$2 | 1 | 512$\times$28$\times$28 | ReLU | BatchNorm
TransposedConv | 4$\times$4 | 2$\times$2 | 1 | 256$\times$56$\times$56 | ReLU | BatchNorm
TransposedConv | 4$\times$4 | 2$\times$2 | 1 | 128$\times$112$\times$112 | ReLU | BatchNorm
TransposedConv | 4$\times$4 | 2$\times$2 | 1 | 3$\times$224$\times$224 | Sigmoid | BatchNorm
Critic network $D$
Convolution | 4$\times$4 | 2$\times$2 | 1 | 64$\times$112$\times$112 | LeakyReLU | InstanceNorm
Convolution | 4$\times$4 | 2$\times$2 | 1 | 128$\times$56$\times$56 | LeakyReLU | InstanceNorm
Convolution | 4$\times$4 | 2$\times$2 | 1 | 256$\times$28$\times$28 | LeakyReLU | InstanceNorm
Convolution | 4$\times$4 | 2$\times$2 | 1 | 512$\times$14$\times$14 | LeakyReLU | InstanceNorm
Convolution | 4$\times$4 | 2$\times$2 | 1 | 1024$\times$7$\times$7 | LeakyReLU | InstanceNorm
Linear | - | - | - | 1024 | ReLU | InstanceNorm
Linear | - | - | - | 1 | - | -
Regressor $R$
ResNet18∗ | - | - | - | 128 | - | -
Linear | - | - | - | 1 | Sigmoid | -
Auxiliary network $T$
Linear | - | - | - | 1024 | LeakyReLU | -
Linear | - | - | - | 256 | LeakyReLU | -
Linear | - | - | - | 64 | LeakyReLU | -
Linear | - | - | - | 10 | LeakyReLU | -
Linear | - | - | - | 1 | - | -
∗We modify the last layer of ResNet18 to be a linear layer with 128 output
channels and ReLU as activation
followed by a dropout with $p=0.5$.
TABLE XXVII: Networks of the approach based on self-supervised learning on CelebA. Layer | Kernel | Stride | Padding | Output shape | Activation | Normalization
---|---|---|---|---|---|---
Encoder $E$
ResNet18 | - | - | - | 1000 | - | -
Mapping network $M$
Linear | - | - | - | 512 | ReLU | -
Linear | - | - | - | 128 | ReLU | -
Regressor $R$
Linear | - | - | - | 512 | ReLU | -
Linear | - | - | - | 64 | ReLU | -
Linear | - | - | - | 10 | ReLU | -
Linear | - | - | - | 1 | - | -


11institutetext: Simon Fraser University

# Extreme Floorplan Reconstruction by Structure-Hallucinating Transformer
Cascades

Sepidehsadat Hosseini  Yasutaka Furukawa

###### Abstract

This paper presents an extreme floorplan reconstruction task, a new benchmark
for the task, and a neural architecture as a solution. Given a partial
floorplan reconstruction inferred or curated from panorama images, the task is
to reconstruct a complete floorplan including invisible architectural
structures. The proposed neural network 1) encodes an input partial floorplan
into a set of latent vectors by convolutional neural networks and a
Transformer; and 2) reconstructs an entire floorplan while hallucinating
invisible rooms and doors by cascading Transformer decoders. Qualitative and
quantitative evaluations demonstrate effectiveness of our approach over the
benchmark of 701 houses, outperforming the state-of-the-art reconstruction
techniques. We will share our code, models, and data.

Figure 1: The paper presents a new task, dubbed extreme floorplan
reconstruction. An input is a partial floorplan reconstruction inferred or
curated from panorama images acquired in a house/apartment. An output is a
complete floorplan. A technical challenge lies in the hallucination of
invisible rooms and doors

##  1 Introduction

Indoor panorama photography has exploded in the last five years. Pioneered by
Ricoh Theta, consumer grade panorama cameras are prevalent on the market,
whose applications range from real-estate, entertainment, surveillance, and
more. The real-estate application is particularly successful, where Ricoh
Theta cameras alone have collected 100 million panoramas from residential
houses. This massive indoor panorama collection allows house
renters/buyers/realtors to browse immersive 360 interior views for tens of
millions of houses, already a great commercial success. However, a panorama
collection per house is extremely sparse (i.e., one panorama per room with
little to no visual overlaps) and even leaves some rooms completely invisible.
The data poses fundamental new challenges to current techniques, such as pose
estimation or structure reconstruction, which could enable more sophisticated
architectural analysis for further applications.

This paper focuses on the task of floorplan reconstruction, where an input is
a partial floorplan inferred or manually curated from panorama images, and an
output is a complete floorplan. Reconstructed floorplans would allow further
applications in the real-estate and construction domains (e.g., building code
verification and property value assessment) as well as virtual and augmented
reality industries (e.g., digital content creation). The technical challenge
lies in the reconstruction or “hallucination” of invisible rooms and doors.
Inferring invisible image or geometry data has been studied in computer vision
in the context of image inpainting [25], illumination inference [15], and
surface reconstruction [22]. To our knowledge, this paper is the first to
tackle the hallucination of architectural components such as rooms and doors
at the scale of an entire house.

The contribution of the paper is three fold: 1) A new extreme floorplan
reconstruction task; 2) A benchmark consisting of 701 houses with ground-truth
vector floorplan images; and 3) A neural architecture, whose cascading
Transformer decoders reconstruct an entire floorplan including invisible
rooms/doors. Qualitative and quantitative evaluations demonstrate the
effectiveness of our approach over the existing techniques. We will share our
code, models, and data.

##  2 Related Work

We review related work in 1) floorplan reconstruction, 2) indoor panorama and
floorplan datasets, 3) content hallucination, and 4) extreme pose estimation.

###  2.1 Floorplan reconstruction

Floorplan reconstruction has a long history in image processing research with
strong ties with real-estate and construction industries. With the success of
commodity depth sensors, the current methods take 3D point-clouds as an input
and reconstructs vector-graphics floorplan images, often with a combination of
deep neural networks and optimization [10, 16]. The emergence of panorama
cameras meet growing demand in floorplan reconstruction from images alone.
Zillow indoor dataset provides panorama images and ground-truth floorplans,
which have been reconstructed with manual interactions [5]. This paper tackles
fully automated floorplan reconstruction, where the data comes from a
production pipeline collected by real users, whose image coverage is
incomplete and sparse.

###  2.2 Indoor panorama and floorplan datasets

SUN360 is a large scale panorama image dataset containing indoor images [21].
The data does not have house-level annotations, sufficient for the analysis of
a single room but not for an entire house. RPLAN is a database of synthetic
floorplans in a vector-graphics image format, created by designers [19].
LIFULL HOME’S dataset offers five million real floorplan images in a raster
format, which can be converted to a vector format by a floorplan vectorization
technique with high accuracy [9]. Structured3D dataset offers synthetic
panorama images with floorplans for 3,500 houses or apartments [26].
Matterport3D dataset provides panorama images with floorplans for 90 real
scenes. However, the panorama coverage is extremely dense, where traditional
structure from motion algorithm suffices to reconstruct a scene. ZIND dataset
by Zillow also offers panorama images with floorplans for 1,500 real houses or
apartments [5], which are much sparser and require human interventions for
camera pose estimation and floorplan reconstruction. Nonetheless, the setup is
still small-scale and highly controlled, ensuring that a panorama(s) is taken
in every room. Our data comes from real uncontrolled crowd-sourcing, where
many rooms are not photographed, posing fundamental new challenges to
floorplan reconstruction algorithms.

###  2.3 Content hallucination

Inference of missing image or geometry data has been a popular research topic.
Image inpainting is a classical example [2, 6], where state-of-the-art
techniques employ deep neural networks [24]. Neural illumination by Song et
al. infers a spherical illumination image from a single perspective image for
augmented reality applications [14]. Implicit neural surface representation
allows inference of a full object surface from a single image [22]. Plan2Scene
takes a house interior model with a few number of photographs and infers
texture maps for all the surfaces even for rooms without input photographs
[18]. This paper infers invisible architectural components structures at a
house scale (i.e., rooms and doors).

###  2.4 Extreme pose estimation

Given two RGBD images with little to no visual overlaps, relative pose
estimation is possible by inferring and aligning complete scene structures
[23]. Partial indoor scene models are registered by connecting room-layout
estimations to complete a floorplan [8]. Extreme structure from motion (SfM)
algorithm estimates the camera poses of panorama images with little to no
visual overlaps by learning spatial arrangements of architectural components
(e.g., bedrooms, kitchens, entrances, and doors) [13]. Instead of camera pose
estimation, this paper tackles an “extreme floorplan reconstruction”, whose
problem is to take incomplete floorplan reconstruction (either inferred or
manually curated from posed panorama images) and infer a complete floorplan
including invisible rooms and doors.

##  3 Extreme Floorplan Reconstruction Problem

The section explains the input, the output, the dataset, and the metrics.

Figure 2: An extreme floorplan reconstruction dataset consists of 701 single floor apartments/houses with ground-truth vector-graphics floorplan images. The data comes from a real production pipeline, where an input partial floorplan reconstruction is either inferred automatically [13] or manually curated. The technical challenge lies in the reconstruction of invisible rooms and doors  Table 1: Dataset statistics. We divide the set into five groups based on the number of rooms (2-5, 6-9, 10-13, 14-17, 18+). In each group, the table reports (Top) the number of samples; (Middle) the ave/std of the number of three types of rooms; and (Bottom) the ave/std of the number of two types of doors  | 2-5 | 6-9 | 10-13 | 14-17 | 18+
---|---|---|---|---|---
# of Samples | 60 | 243 | 298 | 84 | 16
Rooms | Visible | 2.4/0.9 | 3.5/1.2 | 3.7/1.6 | 3.9/1.5 | 3.4/1.0
Invisible direct | 1.6/0.5 | 3.7/0.9 | 5.2/1.6 | 6.4/2.2 | 7.2/2.4
Invisible indirect | 0.7/0.9 | 1.4/1.4 | 3.7/2.2 | 5.0/2.8 | 8.2/3.2
Doors | Visible | 3.0/1.2 | 7.2/1.6 | 8.4/2.5 | 10.6/4.0 | 11.4/2.3
Invisible | 0.8/1.0 | 1.5/1.7 | 4.4/3.3 | 5.9/3.2 | 9.1/3.5

Input: The problem input is a partial floorplan reconstruction, consisting of
rooms and doors, each of which is a segmentation mask as a 14-channel
$800\times 800$ image in a top-down view. There are 10 room types and 4 door
types. The room types are (living room, kitchen, western style room, bathroom,
balcony, corridor, Japanese style room, washroom, toilet, and closet). The
door types are (standard-door, entrance-door, closet-door, and open-portal).
An input data comes from either an extreme Structure from Motion (SfM)
algorithm [13] or human annotators (i.e., ground-truth without errors). A door
is represented as a 2x2 pixel region by the annotators. For the extreme-SfM
reconstructions, we identify the door center and replace by a 2x2 pixel
region.

The raw input data is a set of panorama images in an equirectangular
projection with a camera pose, a room layout, door/window detections, and a
room type information. Panoramas are acquired with a mono-pod and have roughly
the same height from the flat floor, which allows us to produce an input
partial floorplan as a 14-channel image from room layout estimations and
camera poses.

During testing, we fit an axis-aligned bounding box to the room/door masks in
a 14-channel image, uniformly rescale to fit at the center of a $200\times
200$ square, then add a padding of 300 pixels all around, resulting in a
$800\times 800$ image. During training, we use only ground-truth samples. We
uniformly rescale an image to fit at the center of a $100\times 100$ square,
and add a padding of 76 pixels all around to make a $256\times 256$ image. We
then apply the same set of augmentation steps in DETR [3] (i.e., cropping and
resizing), followed by 1) 50% chance of horizontal flipping and 2) 50% chance
of rotation by either 90 or -90 degrees.

Output: The output is a complete floorplan in a similar format as the input
(i.e., a component-wise $800\times 800$ raster segmentation mask in the top-
down view). Figure 2 shows a sample, where the house has 10 rooms and 12
doors. Ten doors are visible and given in the input, leaving two “invisible
doors” to be reconstructed. Four rooms are visible, leaving six “invisible
rooms” to be reconstructed; 1) five of which are adjacent to the input
reconstruction and are dubbed “direct invisible rooms”; and 2) the last of
which is dubbed “indirect invisible room” (e.g., an invisible closet in an
invisible room). Note that our system produces a vector-graphics floorplan.
However, vector reconstruction is challenging, and we define the task output
to be a raster floorplan for evaluation.

Dataset: Extreme floorplan reconstruction dataset consists of 701 houses (or
apartments). Each house has a ground-truth floorplan image, which we converted
to a vector-graphics image by Raster2Vector [9] and manual fixes. The raw
input is a set of 2,355 panorama images, captured by Ricoh Theta camera
series. The number of panorama images per house ranges from 1 to 7.
Specifically, (31, 129, 222, 199, 118, 2) houses contain (1, 2, 3, 4, 5, 7)
panoramas, respectively. Table 1 provides more statistics. Input partial
floorplans are either inferred by extreme Structure-from-Motion system [13] or
created from the GT floorplans (i.e., dropping room masks that do not contain
panorama centers). Randomly sampled 651 houses are used for training while 50
houses are used for testing. In the test set, (9, 19, 15, 7) houses contain
(2, 3, 4, 5) panorama images, respectively. The number of invisible rooms vary
between 1 and 17. The number of invisible doors vary between 0 and 11.

Metrics: We borrow the precision and recall metrics, introduced in a floorplan
reconstruction paper [4]. 111We borrow the room metrics but do not use the
corner and edge metrics, which require a vector-graphics floorplan image and
are too harsh in our problem setting. The section defines the room metrics,
which are the same for the doors. The metric calculation differs when the
input is a ground-truth (i.e., by human annotators) or an output of the
extreme-SfM system.

When the input is a ground-truth, there is no need to align reconstructions.
Given component-wise room segmentation masks as a reconstruction, we match the
reconstructed rooms with the ground-truth, then calculate the precision and
recall. We declare that a reconstructed room matches a GT room, when the room
types match and the intersection over union (IoU) score is above 0.7. A GT
room is matched at most once. In practice, we greedily find matches by: 1)
Identifying the match with the highest IoU score; 2) Removing the matched
pair; and 3) Repeat. The process is exactly the same for the door metrics
except that the IoU threshold is 0.5, because door segments are small.

When the input is a result of the extreme-SfM system with errors, we need to
align the reconstruction with the GT before calculating the metric. We
exhaustively try all possible translations (at the granularity of a pixel in a
top-down image space), then record the result with the best F1 score.

##  4 Neural Extreme Floorplan Reconstruction

Figure 3: System overview. Category-wise CNNs with a Transformer block encode
an input partial reconstruction into a set of embedding vectors. The cascading
Transformer decoders reconstruct invisible rooms and doors in three steps

Our end-to-end neural architecture consists of a CNN/Transformer based encoder
and cascading Transformer decoders. The architecture takes a partial
reconstruction and produces a complete floorplan as component-wise raster
segmentation masks, which is refined to a vector-graphics floorplan (See Fig.
3). The section explains the key ideas and the design choices. We refer to the
supplementary for the full architecture specification.

###  4.1 Visible room/door encoder

Convolutional neural networks (CNNs) with a Transformer encodes an input
partial reconstruction given as a set of $(W\times H\times 14)$ images into a
$(\frac{W}{32}\times\frac{H}{32}\times 256)$ feature map in two steps. During
testing, $W=H=800$. During training, $W$ and $H$ depend on the augmentation
and are around 600.

In the first step, a standard Res-Net [7] (either 50 or 101 layers) processes
input images per architectural component category in three branches (i.e.,
room, door, or both). For example, in the room-branch, $(W\times H\times 14)$
images only for the room instances are the input. We feed each room (image) to
Res-Net, take the last layer of the last conv-block
$(\frac{W}{32}\times\frac{H}{32}\times 2048$), apply a 1x1 convolution to
change the feature dimension to 256, and perform element-wise maximum of the
feature maps across all the rooms, which results in a
$(\frac{W}{32}\times\frac{H}{32}\times 256)$ feature map. We treat the output
as $(\frac{W}{32}\times\frac{H}{32})$ tokens with 256-dim embedding for the
next Transformer block. ResNet produces the same number of tokens from the
other two branches. The both-branch is the same as the room-branch except that
the input $(W\times H\times 14)$ image for each room contains masks of its
incident doors. In the door-branch, the input is a single $(W\times H\times
14)$ image that contains all the door masks (instance unaware), which is
memory efficient and works well.

In the second step, a self-attention block from standard Transformer [17] (6
layers w/ 8 heads) takes the tokens from the three branches. Following the
work by Zhou et al. [27], we add standard frequency position encoding to
distinguish the branch-type and the X/Y position in the feature map:

$\displaystyle\overrightarrow{f_{xy}}\leftarrow\overrightarrow{f_{xy}}+\left[\overrightarrow{P}_{128}(x),\overrightarrow{P}_{128}(y)\right]+\overrightarrow{P}_{256}(type)$
(1)
$\displaystyle\overrightarrow{P}_{d}(t)=\left[\\{\cos{(10^{8i/d}t)}\\},\\{\sin{(10^{8i/d}t)}\\}\right],\
(i=1,2,\cdots d/2).$ (2)

$\overrightarrow{f_{xy}}$ denotes the 256-dim feature vector of a token at
position $(x,y)$, where $x\in\left[1,\frac{W}{32}\right]$ and
$y\in\left[1,\frac{H}{32}\right]$. $\overrightarrow{P}_{d}(t)$ is a
$d$-dimensional standard frequency position encoding. $(type)$ is a scalar,
indicating the branch-type (both=1001, room=1002, and door=1003). The tokens
from the “both” branch are passed to the decoder.

###  4.2 Invisible room/door cascading decoders

Three cascaded transformer decoders reconstruct direct invisible rooms,
indirect invisible rooms, and invisible doors (See Fig. 2).

Direct invisible room decoder: Direct invisible rooms are behind the doors
detected in the input panoramas. Let $N_{vis\cdot d}$ be the number of
detected doors, which is the expected number of direct invisible rooms.
Following DETR [3], we pass $N_{vis\cdot d}$ query tokens with learnable
embeddings to a self-attention block, while feeding the tokens from the
encoder via cross-attention. Each query token will contain an embedding of a
direct invisible room to be reconstructed. The number of detected doors (i.e.,
query embeddings) vary per sample. We prepare 20 embedding vectors, which is
large enough for all the training data.

An output embedding vector at each query token is used to 1) classify the
room-type as a 15-dim vector by a fully connected layer with a soft-max
function (see details below); 2) regress the bounding box parameters (i.e.,
the center, the width, and the height) in the normalized image coordinate
($x,y\in[0,1]$) by a 3-layer MLP with ReLU activation (hidden dimension 256);
and 3) estimate a binary segmentation mask by the panoptic segmentation head
in DETR [3]. Note that the type classification labels consist of 10 room
types, 4 door types, and ”no room/door”. The last label indicates that no
room/door is reconstructed. At testing, we use the category with the greatest
value and keep pixels above the value of 0.0 in the segmentation mask. We do
not use the bounding box for floorplan reconstruction, but for a loss function
during training.

Indirect invisible room decoder: The second cascaded decoder reconstructs
indirect invisible rooms via query tokens while passing the encoder tokens via
cross-attention in the same way as the first cascade. The only difference is
that the self-attention block also incorporates the direct room tokens from
the first cascade. Both the direct and the indirect room tokens are used to
predict the room-type, the bounding box parameters, and the segmentation mask
by exactly the same network modules with the same loss functions. We assume 15
indirect invisible rooms at maximum and pass 15 query tokens with learnable
embedding.

Invisible door decoder: The third cascaded decoder reconstructs invisible
doors via query tokens in the same way as the first two cascades. The
difference is that the self-attention block incorporates all the tokens
(direct rooms, indirect rooms, and doors). The room/door types and the
segmentation masks from the third cascade are used to complete the output
floorplan.

###  4.3 Loss Functions

Our neutral network reconstructs a floorplan in three cascades. In each
cascade, we take the reconstructed components, match with the corresponding
ground-truth components, and inject loss functions. The matching is done in
exactly the same way as DETR [3], based on the type classification and the
bounding box estimation. The first cascade is for invisible direct rooms, and
we match against only the GT invisible direct rooms. For the second cascade,
we match against the GT invisible direct/indirect rooms. For the third
cascade, we match against the GT invisible direct/indirect rooms and invisible
doors.

Without loss of generality, we use the second cascade as an example to define
loss functions, where reconstructed invisible room instances are matched
against the corresponding ground-truth. The loss functions consist of the
three terms for the room-type classification, the bounding box parameter
regression, and the segmentation mask estimation:

$\displaystyle L$ $\displaystyle=$ $\displaystyle L_{type}+L_{bbox}+L_{seg},$
(3) $\displaystyle L_{type}$ $\displaystyle=$
$\displaystyle\frac{1}{|\mathcal{I}_{all}|}\sum_{i\in\mathcal{I}_{all}}-w(i)log(p_{i}),$
(4) $\displaystyle L_{bbox}$ $\displaystyle=$
$\displaystyle\frac{1}{|\mathcal{I}_{match}|}\sum_{i\in\mathcal{I}_{match}}5\left\|b_{i}-\hat{b}_{i}\right\|_{1}-2\
\mbox{IOU}(b_{i},\hat{b}_{i}),$ (5) $\displaystyle L_{seg}$ $\displaystyle=$
$\displaystyle\frac{5}{|\mathcal{I}_{match}|}\sum_{i\in\mathcal{I}_{match}}\left[L_{dice}(m_{i},\hat{m_{i}})+\frac{1}{|\mathcal{P}|}\sum_{p\in\mathcal{P}}L_{focal}(m^{p}_{i},\hat{m^{p}_{i}})\right].$
(6)

$L_{type}$ is a standard softmax loss with per-category weighing.
$\mathcal{I}_{all}$ is a set of indexes of all the reconstructed room
instances. With abuse of notation, $p_{i}$ denotes the classification score of
the GT room type (i.e., the room type of the matched GT). $w(i)$ denotes a
weighing constant associated with the GT room type to compensate for the
imbalance in the training data. Concretely, the weight is set inversely
proportional to the number of samples in the training set, which is (1.135,
3.7. 0.6. 0.63, 0.47, 1.2, 2.2, 0.76, 0.48, 0.18, 0.763, 0.524, 0.33, 0.7, and
0.1) for (living room, kitchen, bedroom, toilet, balcony, corridor, tatami ,
washroom. bathroom, closet, closet door, open door, door, entrance, and no-
room), respectively.

$L_{bbox}$ sums the discrepancies of the bounding box estimation over the
matched instances. The first term in the summation is the L1 norm of the
4-dimensional bounding box parameter vector (the center, the width, and the
height in the normalized image coordinate). $b_{i}$ denotes the estimated
parameter vector and $\hat{b_{i}}$ denotes the corresponding GT vector. The
second term is the intersection over union score between the reconstructed and
the ground-truth bounding boxes.

$L_{seg}$ sums the discrepancy of the segmentation mask over the matched
instances. The first term in the summation is the standard dice loss between
the estimated mask ($m_{i}$) and the GT ($\hat{m_{i}}$). The second term is
the average focal loss of the per-pixel mask value. $\mathcal{P}$ denotes the
set of pixels in the domain. $m^{p}_{i}$ (resp. $\hat{m}^{p}_{i}$) denotes the
estimated (resp. GT) per-pixel mask value.

##  5 Floorplan refinement and vectorization

Cascading decoders reconstruct a floorplan as raster images, where room
boundaries are curvy and door shapes are irregular. We use a floorplan
generative model (House-GAN++[12] by Nauata et al.) to refine our output and
convert to a vector-graphics floorplan image. House-GAN++ is designed to
produce floorplans from noise vectors, but can also be used to refine a design
without changing the overall arrangement by specifying an entire floorplan as
an input constraint. House-GAN++ learns a bias to prefer straight shape
boundaries and we use the same post-processing in the original work to produce
a vector graphics floorplan.

RPLAN dataset [19] was originally used to train House-GAN++, but is synthetic
and may exhibit incompatible database bias. We use LIFULL HOME’S dataset [1]
instead. Since the goal is the refinement without overall arrangement changes,
we use fully-connected graphs both for training and testing. The reconstructed
floorplan from our cascading decoder is specified as the input constraint.

Lastly, House-GAN++ iteratively refines a design. We use the following
heuristics to suppress artifacts after each iteration: 1) If a room (resp.
door) is broken into multiple connected components, we pick one with the
highest IOU with the original decoder reconstruction and removes the rest; 2)
If the IOU score with the original becomes less than 50%, we keep the mask in
the previous iteration without updating; and 3) For each door, we compute its
center of mass, find the closest room-boundary point and snap it (this is only
after the last iteration).

##  6 Experimental Results

We have implemented the proposal system using PyTorch 1.10.0 and Python 3.9.7,
and used a workstation with a 3.70GHz Intel i9-10900X CPU (20 cores) and an
NVIDIA RTX A6000 GPU. Due to memory limitation, we use a batch size of 1
during training, while accumulating gradients over 16 samples for one
parameter update. We use AdamW optimization method, while setting the learning
rate to $10^{-5}$ for the CNN module in the encoder and $10^{-4}$ for the rest
of the network (i.e., a Transformer block in the encoder and the three
cascading Transformer decoders). We divide the learning rate by 10 after every
100 epochs. We train for 240 epochs, which takes roughly 22 hours with the
workstation.

Competing methods: We compare the proposed approach against the three
competing methods. The first one is UNeT with a public implementation [11].
The second one is Mask-RCNN with two different backbones (ResNet-50 and
ResNet-101) with an official implementation from Meta AI Research [20]. The
third one is DETR again with the same two backbones (ResNet-50 and ResNet-101)
with an official implementation by the authors [3].

These methods take a single image as an input, while our input is a set of
14-channel images (instance-aware). We perform pixel-wise max-pooling over the
set of images and produce a single 14-channel image as their input. Our output
is a set of architectural components, each of which is represented by a binary
segmentation mask, a probability distribution over the room/door types, and
the bounding box parameters, which Mask-RCNN and DETR directly produce.

For UNet, the output is a single 14-channel image, which is instance-unaware.
Therefore, we keep pixels above the threshold 0.5, find connected components,
and discard small components whose areas are less than 4 pixels to produce a
floorplan. We choose the value of 0.5 because the average F1 score did not
change much when we vary the threshold from 0.1 to 0.9 with an increment of
0.1. For Mask-RCNN, we set a threshold of 0.6 on the confidence prediction. We
vary the parameter and pick the one with the highest average F1 score. DETR
does not require a threshold as in our approach: A room/door is not generated
when the probability of type “no room/door’ is the greatest. A threshold of
0.0 is used to determine a mask for Mask-RCNN and DETR.

For fair evaluation, we have used 1) the same category based weighing (Eq. 5)
for the type classification loss for Mask-RCNN and DETR; 2) the same parameter
update schedule (once every 16 samples); 3) the same data augmentation steps;
and 4) the same learning rate schedules. 222Our network has two learning rate
schedules, one for Transformer blocks and the other for the CNN module.
Exactly the same schedule is used for DETR. Our schedule for the CNN module is
used for Unet and Mask-RCNN.

Table 2: The main quantitative evaluations. We compare against UNet, Mask-RCNN ($\mbox{M}_{\mbox{rcnn}}$), and DETR with ResNet-50 or ResNet-101 backbone. Input partial floorplans are either ground-truth (left) or outputs from the extreme SfM system [13] (right). The precision, the recall, and the F1 score are reported for the rooms and the doors  | Manual (GT) | Extreme-SfM output
---|---|---
Method | Rooms | Doors | Rooms | Doors
| Pre. | Rec. | F1 | Pre. | Rec. | F1 | Pre. | Rec. | F1 | Pre. | Rec. | F1
UNeT | 6.2 | 8.4 | 7.1 | 0.8 | 0.3 | 0.4 | 4.5 | 6.8 | 5.4 | 1.2 | 0.3 | 0.5
$\mbox{M}_{\mbox{rcnn}}$(50) | 38.1 | 24.2 | 29.6 | 12.0 | 13.4 | 12.7 | 37.5 | 23.2 | 28.6 | 12.3 | 11.4 | 11.8
$\mbox{M}_{\mbox{rcnn}}$(101) | 39.2 | 18.8 | 25.4 | 14.6 | 13.9 | 14.3 | 39.1 | 22.6 | 28.6 | 13.9 | 12.6 | 13.2
DETR(50) | 30.6 | 47.3 | 37.1 | 13.0 | 13.4 | 13.2 | 27.1 | 41.5 | 32.8 | 11.0 | 13.9 | 12.3
DETR(101) | 30.7 | 50.4 | 38.1 | 14.2 | 16.1 | 15.1 | 26.8 | 46.3 | 33.9 | 13.3 | 15.3 | 14.2
Ours(101) | 46.2 | 41.4 | 43.7 | 15.7 | 14.8 | 15.2 | 31.9 | 39.9 | 35.4 | 10.3 | 13.1 | 11.5
Ours(50) | 49.2 | 50.8 | 50.0 | 20.2 | 19.3 | 19.7 | 37.1 | 41.0 | 38.9 | 14.0 | 18.4 | 15.9
Table 3:  Ablation study on the category-wise CNN encoder. The left three columns shows how the input partial reconstructions are passed to each of the three CNN branches. (m) indicates that a branch receives multiple images (instance-aware) as an input. (s) indicates that a branch receives a single image (instance-unaware) that contains masks of all the instances. No-mark indicates that a branch is not used  both |  room |  door | Direct Rooms | Rooms | Rooms | Doors
---|---|---|---|---|---|---
(First Cas.) | (Second Cas.) | (Full Pipeline) | (Full Pipeline)
Pre. | Rec. | F1 | Pre. | Rec. | F1 | Pre. | Rec. | F1 | Pre. | Rec. | F1
s |  |  | 56.3 | 62.6 | 59.3 | 40.8 | 45.1 | 42.8 | 43.8 | 46.5 | 42.8 | 14.8 | 14.5 | 14.6
m |  |  | 67.8 | 63.5 | 65.5 | 34.6 | 43.9 | 38.7 | 35.8 | 48.2 | 41.8 | 6.2 | 7.5 | 6.7
m | m |  | 63.7 | 62.5 | 63.1 | 35.3 | 44.6 | 39.4 | 36.0 | 48.6 | 41.3 | 7.1 | 6.9 | 7.0
m |  | s | 60.7 | 61.1 | 60.8 | 47.6 | 45.5 | 46.3 | 40.6 | 47.6 | 43.9 | 14.1 | 15.6 | 14.8
m | m | m | 66.2 | 64.7 | 65.8 | 46.7 | 44.9 | 45.8 | 48.4 | 49.2 | 48.8 | 10.6 | 11.2 | 10.9
s | s | s | 57.2 | 54.2 | 55.6 | 37.6 | 39.7 | 38.6 | 38.4 | 40.1 | 39.2 | 17.8 | 18.9 | 18.3
m | m | s | 68.2 | 64.2 | 66.1 | 49.6 | 46.6 | 48.0 | 49.2 | 50.8 | 50.0 | 20.2 | 19.3 | 19.7
Table 4: Ablation study on the cascading decoders. We turn on and off each of the three cascades (indicated by the left columns) and retrain the entire model. The table also reports the metrics of reconstructions by the first and the second cascades. Note that the first (resp. second) cascade reconstructs only direct rooms (resp. direct and indirect rooms), respectively, whose metrics are reported  Cascades | Direct Rooms | Rooms | Rooms | Doors
---|---|---|---|---
1st |  2nd |  3rd | (First Cas.) | (Second Cas.) | (Full Pipeline) | (Full Pipeline)
Pre. | Rec. | F1 | Pre. | Rec. | F1 | Pre. | Rec. | F1 | Pre. | Rec. | F1
|  | ✓ | x | x | x | x | x | x | 36.3 | 41.1 | 38.5 | 14.5 | 14.6 | 14.5
| ✓ | ✓ | x | x | x | 39.4 | 46.3 | 42.6 | 43.1 | 47.0 | 44.9 | 14.9 | 15.2 | 15.0
✓ |  | ✓ | 67.5 | 60.3 | 63.6 | x | x | x | 36.2 | 41.4 | 38.6 | 16.3 | 16.1 | 16.2
✓ | ✓ |  | 44.8 | 46.3 | 45.5 | 40.9 | 42.0 | 41.5 | 40.9 | 42.0 | 41.5 | x | x | x
✓ | ✓ | ✓ | 68.2 | 64.2 | 66.1 | 49.6 | 46.6 | 48.2 | 49.2 | 50.8 | 50.0 | 20.2 | 19.3 | 19.7
Figure 4: Main qualitative evaluations. The figure shows reconstruction
results before the final refinement for four houses. For each method, we show
two results, when the input partial reconstruction is from the ground-truth or
the extreme SfM system [13]. At the right, we show both our ground-truth
vector-graphics floorplans as well as the original raster floorplan images by
an architect  Figure 5: Reconstructed floorplans before and after the
refinement by House-GAN++ and several heuristics. The same refinement process
is used for all the methods. The input partial reconstructions are ground-
truth in this figure

Quantitative evaluations: Table 2 provides our main quantitative evaluations,
comparing against three competing methods. The best, the second best, and the
third best results are shown in cyan, orange, and magenta, respectively. Our
system achieves the highest F1 score in all the settings. Note that Mask-RCNN
(101) outperforms us in precision, but its recall is extremely low. Similarly,
DETR (101) outperforms in recall, but its precision is extremely low.
Interestingly, a smaller backbone (ResNet-50) achieves better results in our
system, which is probably because the main learning happens in our Transformer
blocks and large CNNs merely introduce overfitting.

Table 3 provides an ablation study on the category-wise CNN encoder. The
bottom row is our final system. The fifth entry shows a configuration where
the door branch receives multiple images (instance aware), which shows
comparable room metrics but worse performance for doors. This suggests that
doors are already cleanly separated and an instance unaware single image
representation is more efficient and effective. The last two rows suggest that
the room information should be passed as the instance aware multi-image
representation.

Table 4 provide an ablation study on the cascading decoders. The middle three
rows show that dropping any of the cascades downgrades performance. While the
third cascade alone (first row) produces reasonable results, the proposed
system with the three cascades achieves the best result in every metric. The
last two rows show an interesting phenomena where the third cascade improves
the performance of the first two cascades via gradient propagation.

Qualitative evaluation: Figure 4 provides the main qualitative evaluations
against Mask-RCNN and DETR. The input partial reconstructions and the
corresponding ground-truth reveal that this is a challenging reconstruction
task unlike any existing problems. Mask-RCNN is often capable of inferring
closets or balconies that are direct neighbors of the input partial
reconstruction. However, it fails to infer indirect invisible rooms or doors
that are far away from the input reconstruction. With the power of the
Transformer, DETR reconstructs indirect rooms and doors more than Mask-RCNN.
Nonetheless, our approach infers many invisible structures successfully, in
particular when manual (ground-truth) partial reconstructions are given.
Please see the supplementary for more reconstruction examples.

Figure 5 shows the raw floorplan reconstructions by our neural network and the
final floorplan models after the refinement by House-GAN++ [12] and a few
heuristics. The refinement process successfully turns raw floorplan
reconstructions with many artifacts (e.g., curvy room boundaries and gaps
between rooms) into clean floorplan models. For the top two examples, the
refined floorplans are close to the ground-truth. The bottom example exhibits
a few discrepancies (e.g., a few missing closets and an inaccurate corridor
shape) but looks realistic and suffices for a digital content in AR/VR
applications.

Concluding marks: This paper presents a new extreme floorplan reconstruction
task, a new benchmark, and an effective neural architecture as a solution. The
task is challenging and many of our results are not satisfactory. Major
failure modes are 1) missing rooms; 2) inaccurate room shapes, in particular
corridors; and 3) inaccessible rooms without any doors. We hope that this
paper starts an avenue of new research towards an ultimate extreme floorplan
reconstruction system, capable of reconstructing accurate and realistic
floorplans for tens of millions of houses with sparse panorama coverage out
there. We will share all our code, model, and data, except the panorama images
for a privacy concern.

##  7 Acknowledgement

This research is partially supported by NSERC Discovery Grants with
Accelerator Supplements, DND/NSERC Discovery Grant Supplement, and a RICOH
research grant.

## References

  * [1] Liful dataset, https://www.nii.ac.jp/dsc/idr/lifull
  * [2] Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of the 27th annual conference on Computer graphics and interactive techniques. pp. 417–424 (2000)
  * [3] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.M. (eds.) Computer Vision – ECCV 2020. pp. 213–229. Springer International Publishing, Cham (2020)
  * [4] Chen, J., Liu, C., Wu, J., Furukawa, Y.: Floor-sp: Inverse cad for floorplans by sequential room-wise shortest path. In: The IEEE International Conference on Computer Vision (ICCV) (2019)
  * [5] Cruz, S., Hutchcroft, W., Li, Y., Khosravan, N., Boyadzhiev, I., Kang, S.B.: Zillow indoor dataset: Annotated floor plans with 360deg panoramas and 3d room layouts. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2133–2143 (June 2021)
  * [6] Hays, J., Efros, A.A.: Scene completion using millions of photographs. ACM Transactions on Graphics (ToG) 26(3), 4–es (2007)
  * [7] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015), http://arxiv.org/abs/1512.03385
  * [8] Lin, C., Li, C., Wang, W.: Floorplan-jigsaw: Jointly estimating scene layout and aligning partial scans. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5674–5683 (2019)
  * [9] Liu, C., Wu, J., Kohli, P., Furukawa, Y.: Raster-to-vector: Revisiting floorplan transformation. In: 2017 IEEE International Conference on Computer Vision (ICCV). pp. 2214–2222 (2017). https://doi.org/10.1109/ICCV.2017.241
  * [10] Liu, C., Wu, J., Furukawa, Y.: Floornet: A unified framework for floorplan reconstruction from 3d scans. In: Proceedings of the European Conference on Computer Vision (ECCV) (September 2018)
  * [11] MILESI, A..: U-net: Semantic segmentation with pytorch. https://github.com/milesial/Pytorch-UNet (2018)
  * [12] Nauata, N., Hosseini, S., Chang, K.H., Chu, H., Cheng, C.Y., Furukawa, Y.: House-gan++: Generative adversarial layout refinement network towards intelligent computational agent for professional architects. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13632–13641 (2021)
  * [13] Shabani, M.A., Song, W., Odamaki, M., Fujiki, H., Furukawa, Y.: Extreme structure from motion for indoor panoramas without visual overlaps. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5703–5711 (2021)
  * [14] Song, S., Funkhouser, T.: Neural illumination: Lighting prediction for indoor environments. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6918–6926 (2019)
  * [15] Song, S., Funkhouser, T.A.: Neural illumination: Lighting prediction for indoor environments. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 6911–6919 (2019)
  * [16] Stekovic, S., Rad, M., Fraundorfer, F., Lepetit, V.: Montefloor: Extending mcts for reconstructing accurate large-scale floor plans. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 16034–16043 (October 2021)
  * [17] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. CoRR abs/1706.03762 (2017), http://arxiv.org/abs/1706.03762
  * [18] Vidanapathirana, M., Wu, Q., Furukawa, Y., Chang, A.X., Savva, M.: Plan2scene: Converting floorplans to 3d scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10733–10742 (2021)
  * [19] Wu, W., Fu, X.M., Tang, R., Wang, Y., Qi, Y.H., Liu, L.: Data-driven interior plan generation for residential buildings. ACM Transactions on Graphics (SIGGRAPH Asia) 38(6) (2019)
  * [20] Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., Girshick, R.: Detectron2. https://github.com/facebookresearch/detectron2 (2019)
  * [21] Xiao, J., Ehinger, K.A., Oliva, A., Torralba, A.: Recognizing scene viewpoint using panoramic place representation. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition. pp. 2695–2702. IEEE (2012)
  * [22] Xu, Q., Wang, W., Ceylan, D., Mech, R., Neumann, U.: Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. Advances in Neural Information Processing Systems 32 (2019)
  * [23] Yang, Z., Pan, J.Z., Luo, L., Zhou, X., Grauman, K., Huang, Q.: Extreme relative pose estimation for rgb-d scans via scene completion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4531–4540 (2019)
  * [24] Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Free-form image inpainting with gated convolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4471–4480 (2019)
  * [25] Yu, Y., Zhan, F., Wu, R., Pan, J., Cui, K., Lu, S., Ma, F., Xie, X., Miao, C.: Diverse image inpainting with bidirectional and autoregressive transformers. Proceedings of the 29th ACM International Conference on Multimedia (2021)
  * [26] Zheng, J., Zhang, J., Li, J., Tang, R., Gao, S., Zhou, Z.: Structured3d: A large photo-realistic dataset for structured 3d modeling. In: Proceedings of The European Conference on Computer Vision (ECCV) (2020)
  * [27] Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., Zhang, W.: Informer: Beyond efficient transformer for long sequence time-series forecasting. In: The Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Conference. vol. 35, pp. 11106–11115. AAAI Press (2021)