# Associations between pain-management treatments and opioid use disorder risk
among Medicaid patients

Kara E. Rudolph  Department of Epidemiology, Mailman School of Public Health,
Columbia University PhD  Department of Epidemiology, Mailman School of Public
Health, Columbia University Nicholas T. Williams  Department of Epidemiology,
Mailman School of Public Health, Columbia University MPH  Department of
Epidemiology, Mailman School of Public Health, Columbia University Ivan Diaz
New York University Grossman School of Medicine.  PhD  New York University
Grossman School of Medicine.  Sarah Forrest  Department of Epidemiology,
Mailman School of Public Health, Columbia University MPH  Department of
Epidemiology, Mailman School of Public Health, Columbia University Katherine
L. Hoffman  Department of Epidemiology, Mailman School of Public Health,
Columbia University MPH  Department of Epidemiology, Mailman School of Public
Health, Columbia University Hillary Samples  Rutgers Institute for Health,
Rutgers University  PhD  Rutgers Institute for Health, Rutgers University
Mark Olfson  Department of Epidemiology, Mailman School of Public Health,
Columbia University MD  Department of Epidemiology, Mailman School of Public
Health, Columbia University Lisa Doan  New York University Grossman School of
Medicine.  MD  New York University Grossman School of Medicine.  Magdalena
Cerda  New York University Grossman School of Medicine.  DrPH  New York
University Grossman School of Medicine.  Rachael Ross  Department of
Epidemiology, Mailman School of Public Health, Columbia University PhD
Department of Epidemiology, Mailman School of Public Health, Columbia
University

###### Abstract

Introduction: Chronic pain patients are at increased risk of opioid-misuse.
Less is known about the unique risk conferred by each pain-management
treatment, as treatments are typically implemented together, confounding their
independent effects. We estimated the extent to which pain-management
strategies were associated with risk of incident opioid use disorder (OUD) for
those with chronic pain, controlling for baseline demographic and clinical
confounding variables and holding other pain-management treatments at their
observed levels.

Methods: We used data from two chronic pain subgroups within a cohort of non-
pregnant Medicaid patients aged 35-64 years, 2016-2019, from 25 states: 1)
those with a chronic pain condition co-morbid with physical disability
(N=6,133) or 2) those with chronic pain without disability (N=67,438). We
considered 9 pain-management treatments: prescription opioid i) dose and ii)
duration; iii) number of opioid prescribers; opioid co-prescription with iv)
benzodiazepines, v) muscle relaxants, and vi) gabapentinoids; vii) non-opioid
pain prescription, viii) physical therapy, and ix) other pain treatment
modality. Our outcome was incident OUD.

Results: Having an opioid and gabapentin co-prescription or an opioid and
benzodiazepine co-prescription was statistically significantly associated with
a 16-46% increased risk of OUD. Opioid dose and duration also were
significantly associated with increased risk of OUD. Physical therapy was
significantly associated with an 11% decreased risk of OUD in the subgroup
with chronic pain but no disability.

Conclusions: Co-prescription of opioids with either gabapentin or
benzodiazepines may substantially increase risk of OUD. More positively,
physical therapy may be a relatively accessible and safe pain-management
strategy.

## Introduction

Opioid overdoses continue to rise in the United States (US) to levels termed
“staggering” 1. Mortality rates involving opioids have nearly quadrupled over
the past 10 years 2, underscoring the urgency of accelerating opioid misuse
prevention and treatment efforts 3. A large body of research indicates that
individuals with non-cancer chronic pain are at particular risk of opioid
misuse 4, 5, 6, 7, 8, 9, and the subset of chronic pain patients that also
have a physical disability (termed “high-impact chronic pain" 10, 11) may be
at even greater risk 12.

However, identifying subgroups at increased risk of adverse opioid-related
outcomes is of limited use in the absence of evidence-based strategies that
could reduce their risk. Given the centrality of pain management for those
with a chronic pain condition, identifying treatments that are particularly
risky or protective and amenable to intervention would provide evidence-based
targets for prevention efforts. Opioid prescribing continues to be one of the
most common pain-management treatments 4, but opioid prescriptions at higher
doses and longer durations, use of multiple opioid prescribers, and co-
prescriptions of opioids with medications that have respiratory-depressive
effects, like benzodiazepines, muscle relaxants, and gabapentin/pregabalin,
have been implicated in increasing risk of opioid misuse 13, 14, 15, 16, 17,
8, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29. Currently, a multi-modal
approach to pain treatment is recommended that consists of $\geq 2$ of the
following: i) medications (opioid and non-opioid), ii) restorative therapies
(e.g., physical therapy, massage therapy), iii) interventional procedures
(e.g., epidural steroid injections, nerve blocks), iv) behavioral health
(e.g., counseling), and v) complementary/integrative health (e.g.,
acupuncture, yoga) 11. By including non-opioid treatments in pain management,
reliance on opioid-based treatments may be reduced. Although these recommended
non-opioid modalities may effectively treat pain 11, with the exception of
counseling, whether or not they reduce opioid misuse remains an open question
30, 31, 32, 33, 34, 35, 36.

In studying the relationship between pain-management treatments and opioid
misuse outcomes, previous research considered one or a select few pain-
management treatments at a time, leaving others unmeasured and unaccounted
for. However, chronic pain patients may experience multiple treatments
concurrently or close together in time. Consequently, this previous research
was likely unable to isolate the unique contribution of each treatment, as the
treatment being considered was conflated with other, unmeasured treatments. We
sought to address this gap by considering a larger number of pain-management
treatments across the multiple modalities specified above, using a novel
approach to isolate the extent to which each was associated with an increased
or decreased risk of incident opioid use disorder (OUD) for those with i) a
chronic pain condition co-morbid with physical disability or ii) a chronic
pain condition without physical disability. We hypothesized that opioid-
involved pain-management strategies would be associated with more risk and
that non-opioid strategies would be associated with reduced risk. We estimated
the risk associated with each treatment among adult, non-dual-eligible
Medicaid patients without baseline OUD, 2016-2019.

## Methods

The study was approved by the Columbia University Institutional Review Board.
All code to replicate cohort and variable creation and statistical analyses is
given in the GitHub repository: blindedforreview. Additional methodological
details are in Section S1 of the appendix.

### Data and Cohort

We utilized data from Medicaid T-MSIS Analytic Files (TAF): Demographics,
Other Services, Inpatient, and Pharmacy claims, 2016-2019, and included non-
pregnant Medicaid patients aged 35-64 years in 25 states that had implemented
Medicaid expansion under the Affordable Care Act by 2014. Because individuals
receiving Social Security Disability Insurance transition to Medicare after 24
months 37, we included claims for a maximum of 24 months post-enrollment. We
permitted enrollment on or after January 2, 2016 and required 12 months of
continuous enrollment. During the first 6 months of enrollment (months 1-6),
we evaluated eligibility criteria, characterized baseline covariates, and
determined membership in one of the two chronic condition subgroups described
below (if any). We measured the set of pain-management treatment variables in
months 7-12. We measured outcomes during months 13-24 in the primary analysis
and months 13-18 in a secondary analysis. Figure 1 depicts this timeline.

Figure 1: Study measure characterization timeline

Figure S1 illustrates the exclusion/inclusion criteria. Using the initial
6-month period, we excluded those dual-eligible for Medicare, lacking a
Medicaid eligibility code or with undetermined disability status, or who had a
claim indicating OUD during this period.

Patients were censored at the point they became Medicare-eligible, 24 months-
post enrollment, at the end of 2019, or if they had disenrolled by the end of
the study period.

### Measures

#### Covariates

Baseline covariates included: age in years, sex, race/ethnicity, primary
language (English), marriage/partnership status, household size, veteran
status, and whether the beneficiary’s income was likely $>$138% of the Federal
Poverty Level; and clinical measures, including any inpatient or outpatient
diagnosis of bipolar disorder, anxiety disorder, attention deficit
hyperactivity disorder (ADHD), depressive disorder, or other mental disorder
38, and whether or not the beneficiary had a claim for mental health
counseling. In addition to baseline covariates, we controlled for anxiety and
depressive disorder diagnoses occurring in months 7-12, as these conditions
have been shown to be affected by pain and also increase risk of OUD 39, 40,
41, 42, 5, 43, 44, 45, 46, 47, 48, 49, 50, 51. We also controlled for presence
of a claim for mental health counseling during months 7-12, as this could be a
marker for a mental health condition that was otherwise not observed in the
claims diagnostic codes.

#### Chronic condition subgroups

Using claims from months 1-6, we determined whether each patient met criteria
for either: (1) chronic pain and co-occurring physical disability (CPPD), or
(2) chronic pain only (CP). We identified chronic pain status by non-cancer
diagnoses (ICD-10 codes) typically associated with chronic pain 52, occurring
at least two times 90 days apart for the same region of the body 53. We
identified a physical disability by including eligibility codes indicating
eligibility based on disability, while excluding codes indicating common
disabilities which were unlikely to result in periods of acute or chronic
pain, e.g. developmental disabilities, hearing or sight impairments) 12.

#### Exposures

We considered the following variables related to the prescribing of opioids
for pain (excluding opioids used for OUD treatment): 1) maximum daily opioid
dose in morphine milligram equivalents (MME) considering doses of all pain-
related fills, 2) proportion of days covered (out of the 6 month period) for
these opioid prescriptions, 3) number of unique opioid prescribers, and co-
prescription ($>$25% overlap, first prescription $\geq 5$ days supply) of
opioids for pain with 4) benzodiazepines, 5) muscle relaxants, and 6)
gabapentinoids. We also considered the following non-opioid pain management
variables: 7) presence of a non-opioid pain prescription (noting that claims
will poorly capture over-the-counter medications), 8) physical therapy, and 9)
other pain treatment modalities comprising at least one of the following: a)
ablative techniques, b) acupuncture, c) joint and nerve blocks, d) botulinum
toxin injections, e) chiropractic work, f) transcutaneous electrical nerve
stimulation, g) epidural steroids, h) intrathecal drug therapies, i) massage
therapy, and j) trigger point injection.

#### Outcomes

The primary outcome was new OUD diagnosis (which we refer to as "incident
OUD", because those with OUD during the washout period were excluded), defined
as the presence of any of the following: ICD-10 diagnosis codes indicating
opioid abuse or dependence; nonfatal, unintentional opioid overdose ICD-10
diagnosis codes; and medication for OUD treatment (MOUD; methadone,
buprenorphine, or injection naltrexone) 54. As a secondary outcome, we defined
incident OUD diagnosis using only ICD-10 diagnosis codes indicating opioid
abuse or dependence 38, 55. For the primary analysis, outcomes were assessed
months 13-24 post-enrollment, and a censoring indicator distinguished those
who remained enrolled versus did not. However, it is plausible that our
censoring model may be poorly estimated, so to reduce reliance on it, we
examined only a 18-month study period in the secondary analysis.

### Statistical Analysis

We estimated the association between each pain-management treatment of
interest and incident OUD risk, adjusting for covariates, holding other pain-
management treatments at their observed levels, and stratifying by chronic
condition subgroup. We formally define these quantities and the assumptions
under which they can be interpreted as effects in Section S1.3 of the
appendix. For binary pain-management treatments (e.g., presence of opioid co-
prescription with gabapentin), we estimated the expected difference in OUD
risk if everyone in the subgroup received the treatment versus no intervention
on treatment. For non-binary pain-management treatments (e.g., maximum daily
opioid dose), we estimated the expected difference in OUD risk if everyone in
the subgroup had their value increase by 20% (i.e., multiplied each person’s
value by 1.2) versus no intervention on treatment.

The above statistical estimands are considered modified treatment policies and
are especially well-suited for questions that consider a set of interrelated
exposure or treatment variables where interventions can be considered across
the set of variables 56, 57, 58, 59. We estimated these associations using a
doubly robust, nonparametric targeted minimum loss-based estimator 59. We used
a cross-fitted version of this estimator with 2-folds for the CP subgroup and
5-folds for the CPPD subgroup. This estimator fits regressions for the outcome
mechanism, treatment mechanism, and censoring mechanism. These regressions
were fit using an ensemble of machine learning algorithms60 consisting of
intercept-only models, main-terms generalized linear models, gradient boosted
machines61, multivariate adaptive regression splines62, neural networks63, and
random forests64.

Code for all data cleaning and statistical analyses is available at
blindedforreview.

## Results

There were $N=6,133$ Medicaid patients in the CPPD subgroup and $N=67,438$ in
the CP subgroup. Table LABEL:tab:desc gives the descriptive statistics of
these subgroups, which were similar demographically but with generally higher
prevalence of mood disorders in the CPPD subgroup. Over half of patients (61%)
in the CPPD group received an opioid pain prescription in the 7-12 month
period; opioid prescription rates were lower but still common in the CP
subgroup at 40%. Co-prescriptions of opioids with benzodiazepines,
gabapentinoids, and muscle relaxants were common in both subgroups. Of the
non-opioid pain-management treatments, over one-quarter of those in either
group had a claim for physical therapy in the 7-12 month period.

[Table LABEL:tab:desc here]

Figure 2(b) shows the estimated relative risks (as a percent, e.g., 1.2 is 20%
and 0.8 is -20%) of each pain-management treatment on incident OUD by month 18
of follow up (month 24 of cohort enrollment) and their 95% confidence
intervals, controlling for covariates and holding the other pain-management
treatments at their observed values.

Figure 2: Estimated effects of each pain-management treatment on incident OUD
and 95% CIs for the cohort with (A) co-occurring chronic pain and disability,
and (B) chronic pain alone.

(a)

(b)

Several pain treatments were associated with increased OUD risk across both
subgroups. Having an opioid and gabapentin co-prescription was statistically
significantly associated with a 32% (95% CI: 13-53%) and 16% (95% CI: 0-36%)
increased risk of OUD for CP and CPPD subgroups, respectively. Given that
average 12-month risk of OUD is relatively high in these groups—developing in
10% of CPPD patients and 4% of CP patients (Table 1)—these relative risk
estimates correspond to appreciable absolute risk. Opioid and benzodiazepine
co-prescription was also associated with substantial increased OUD risk (24%,
95% CI: 6-45% in the CP group; 45%, 95% CI: 14-86% in the CPPD group). In
addition, increases in the proportion of days covered by an opioid was also
statistically significantly associated with increased risk of OUD in both
groups and increases in opioid dose was associated with increased risk in the
CP group. For example, for CP, increasing the proportion of days covered by
20% was associated with a 10% (95% CI: 9-12%) increased risk of OUD, and
increasing the maximum daily dose by 20% was associated with a 4% (95% CI:
3-5%) increased OUD risk.

On the other hand, physical therapy was associated with a decreased risk of
developing OUD; in the CP subgroup, this association was a statistically
significant 11% decreased risk (95% CI: 1-19%).

[Figure 2(b) here]

We repeated the above analyses defining OUD based only on ICD-10 codes (Figure
S2); results were largely unchanged. We also repeated the above analyses
measuring incident OUD after only 6 months of follow-up (Figure S3); results
were largely unchanged.

## Discussion

In a large, carefully constructed cohort of over 70,000 adult, non-elderly
Medicaid patients with chronic pain, we found robust evidence of which
treatments were associated with the most risk of developing OUD and which were
associated with reduced risk. In particular, our findings caution against the
co-prescription of opioids and gabapentin, which has become a mainstay of pain
treatment, prescribed for an estimated 20% of pain patients, but which we
found was associated with increased risk of incident OUD by up to 32%. On a
positive note, our findings highlight the potential benefits of physical
therapy for those with chronic pain, as it was associated with an estimated
11% statistically significant decreased risk of OUD. While previous literature
found evidence that physical therapy reduced risk of being prescribed an
opioid 65, ours is the first to show that physical therapy—even when used in
conjunction with prescribed opioids—may reduce risk of an opioid misuse
outcome.

In terms of identifying risky pain treatment strategies, our findings largely
corroborated previous research; our unique contributions were in terms of our
careful quantification of such associated risk, holding all other treatments
constant at their observed levels. A large body of previous research
established that higher doses of opioids and opioids taken for longer duration
each conferred increased risk of opioid misuse, including the development of
OUD 13, 14, 15, 18, 17, 8. We too found that these aspects of opioid
prescribing were risky—a 20% increase in maximum daily opioid dose was
associated with a 2-4% increase in OUD risk (which was statistically
significant for the larger CP subgroup), and a 20% increase in proportion of
days covered by an opioid prescription was associated with a 10-15% increase
in OUD risk, which was statistically significant in both subgroups. Also
aligned with previous research 18, 19, 17, 20, 21, 22, 23, 24, 25, 26, 27, 28,
29, we found co-prescribing of opioids with the respiratory depressive
medications gabapentin and benzodiazepines to be risky, associated with a
statistically significant 16%-46% increased OUD risk, depending on medication
and subgroup.

However, we did not find significant associations between OUD and other
aspects of treatment that previous research found to be risky. For example, in
contrast with previous research 16, 18, we did not estimate a statistically
significant association between the total number of opioid prescribers and
OUD. This discrepancy could be because after holding opioid dose, duration,
and co-prescriptions constant (which previous research did not do), there is
little additional risk conferred by the total number of prescribers.
Similarly, we did not estimate a statistically significant association between
the co-prescription of opioid and muscle relaxants and OUD; previous research
found evidence that their co-prescription increased risk of opioid overdose
22, 23, but was again limited by not conditioning on other common aspects of
pain treatment (thus, their associations could have been confounded). Finally,
although we hypothesized that other pain treatment modality would be
associated with decreased risk of OUD, we instead found no significant
association, perhaps because our definition comprised a set of treatments that
were disparate.

This study improved upon previous research in several ways. First, we measured
a larger number of pain-management treatments that may occur concurrently, at
least in part, and used a novel statistical approach to better isolate the
unique contribution of each treatment to OUD risk. Second, we used a large,
policy-relevant cohort of Medicaid patients. An estimated 40% of people with
OUD are covered by Medicaid 66; thus, this population has the potential to be
particularly impacted by policy change. Third, we used a flexible, robust
nonparametric estimator to make our analysis as assumption-lean as possible.
For example, unlike previous analyses attempting to estimate risk of pain
treatments on OUD and other opioid misuse outcomes, we do not assume that we
can correctly specify a parametric model, which would be a tenuous assumption
given the complex web of time-varying demographic, clinical, provider, and
contextual factors that contribute to risk of such outcomes. Our estimator is
also doubly robust, meaning that as long as we can estimate either the outcome
model consistently or the treatment and censoring models consistently, then
our estimate is expected to be unbiased.

This study is nonetheless limited in several respects. In claims data,
information (e.g., diagnoses) may not be recorded at all, may be recorded
inaccurately, or may be recorded at an inaccurate time. One relevant example
is the challenge of measuring OUD in claims; previous research found that
using ICD-10 diagnosis codes in Medicaid claims data captured 84% of OUD cases
identified by record reviews by addiction medicine clinicians 67.
Consequently, for our primary analysis, we used a more expansive definition of
OUD (including opioid poisonings and MOUD prescribing), as has been done
previously 54, 12. Further, we dichotomized or otherwise summarized several of
the pain-management treatment variables. This was necessary to make our
analysis tractable, but it likely results in some loss of information and may
introduce nondifferential measurement error. Finally, we expect the chronic
pain and physical disability subgroups we consider to also be mismeasured,
though a previous analysis found the membership in these groups to be
relatively stable over time 12.

Another limitation related to our use of claims data is that there is
significant attrition of patients over time, 14% and 21% of eligible patients
were censored by 18- and 24-months respectively. We required our cohort to
maintain coverage over a 12-month period so that we could characterize our
subgroups of interest, covariates, and pain treatments. This means that our
inferences apply only to individuals who maintain coverage for at least this
period. Unlike other previous work, we do not require our cohort to maintain
coverage for the entire 24-month study period; instead, we censor them when at
the point that they lose Medicaid as their primary insurer. This approach is
advantageous because it uses more information and the inferences apply to a
larger portion of the Medicaid population.

In conclusion, we estimated the unique contribution of common pain-management
treatments to risk of incident OUD in a large, recent cohort of Medicaid
patients, disentangling each treatment’s contributed risk from the risk
conferred by other, potentially co-occurring treatments and confounding
factors. Our estimation of the risk associated with the co-prescription of
gabapentin with opioids is noteworthy both because of its magnitude and
because of how common this pain treatment is, affecting 9.1%, and 23.4% of
those in the CP and CPPD subgroups, respectively. On a more positive note, our
finding that receipt of physical therapy was associated with reduced risk of
OUD was also noteworthy, because it represents a common and relatively
accessible pain-management strategy.

This work was supported by the National Institute on Drug Abuse (grant number
R01DA053243).

## References

  * 1 Katz J., Sanger-Katz M., Sullivan E.. Some Key Facts About Fentanyl. New York Times, https://www.nytimes.com/2023/10/05/upshot/fentanyl-opioids-mexico-explainer.html.  Accessed: 2023-10-25.
  * 2 National Institute on Drug Abuse . Drug Overdose Death Rates. https://nida.nih.gov/research-topics/trends-statistics/overdose-death-rates 2023.  Accessed: 2023-10-25.
  * 3 Centers for Medicare & Medicaid Services . Ongoing emergencies & disasters 2020\. https://www.cms.gov/About-CMS/Agency-Information/Emergency/EPRO/Current-Emergencies/Ongoing-emergencies Accessed: 2020-07-11.
  * 4 Mikosz C. A., Zhang K., Haegerich T., et al. Indication-specific opioid prescribing for US patients with Medicaid or private insurance, 2017 JAMA Network Open.  2020;3:e204514–e204514.
  * 5 Marshall B., Bland M. K., Hulla R., Gatchel R. J.. Considerations in addressing the opioid epidemic and chronic pain within the USA Pain Management.  2019;9:131–138.
  * 6 Dunn K. M., Saunders K. W., Rutter C. M., et al. Opioid prescriptions for chronic pain and overdose: a cohort study Annals of internal medicine.  2010;152:85–92.
  * 7 Orhurhu V., Olusunmade M., Urits I., et al. Trends of opioid use disorder among hospitalized patients with chronic pain Pain Practice.  2019;19:656–663.
  * 8 Volkow N. D., McLellan A. T.. Opioid abuse in chronic pain—misconceptions and mitigation strategies New England Journal of Medicine.  2016;374:1253–1263.
  * 9 Cerdá M., Krawczyk N., Hamilton L., Rudolph K. E., Friedman S. R., Keyes K. M.. A critical review of the social and behavioral contributions to the overdose epidemic Annual Review of Public Health.  2021;42:95–114.
  * 10 Pitcher M. H., Von Korff M., Bushnell M. C., Porter L.. Prevalence and profile of high-impact chronic pain in the United States The Journal of Pain.  2019;20:146–160.
  * 11 Committee I. P. R. C.. National Pain Strategy: A Comprehensive Population Health-Level Strategy for Pain tech. rep. 2016, url: https://www.iprcc.nih.gov/sites/default/files/, Date accessed: 2020-05-07.
  * 12 Hoffman K. L., Milazzo F., Williams N. T., et al. Independent and joint contributions of physical disability and chronic pain to incident opioid use disorder and opioid overdose among Medicaid patients Psychological Medicine.  2023:In press.
  * 13 Savych B., Neumark D., Lea R.. Do opioids help injured workers recover and get back to work? The impact of opioid prescriptions on duration of temporary disability Industrial Relations: A Journal of Economy and Society.  2019;58:549–590.
  * 14 Edlund M. J., Martin B. C., Russo J. E., DeVries A., Braden J. B., Sullivan M. D.. The role of opioid prescription in incident opioid abuse and dependence among individuals with chronic non-cancer pain: the role of opioid prescription The Clinical journal of pain.  2014;30:557.
  * 15 Glanz J. M., Binswanger I. A., Shetterly S. M., Narwaney K. J., Xu S.. Association between opioid dose variability and opioid overdose among adults prescribed long-term opioid therapy JAMA Network Open.  2019;2:e192613–e192613.
  * 16 Peirce G. L., Smith M. J., Abate M. A., Halverson J.. Doctor and Pharmacy Shopping for Controlled Substances Medical Care.  2012;50:7.
  * 17 Cho J., Spence M. M., Niu F., Hui R. L., Gray P., Steinberg S.. Risk of Overdose with Exposure to Prescription Opioids, Benzodiazepines, and Non-benzodiazepine Sedative-Hypnotics in Adults: a Retrospective Cohort Study Journal of General Internal Medicine.  2020:1–8.
  * 18 Rose A. J., Bernson D., Chui K. K. H., et al. Potentially inappropriate opioid prescribing, overdose, and mortality in Massachusetts, 2011–2015 Journal of general internal medicine.  2018;33:1512–1519.
  * 19 Gressler L. E., Martin B. C., Hudson T. J., Painter J. T.. Relationship between concomitant benzodiazepine-opioid use and adverse outcomes among US veterans Pain.  2018;159:451–459.
  * 20 Ford J. A., Hinojosa M. S., Nicholson H. L.. Disability status and prescription drug misuse among US adults Addictive Behaviors.  2018;85:64–69.
  * 21 Sun E. C., Dixit A., Humphreys K., Darnall B. D., Baker L. C., Mackey S.. Association between concurrent use of prescription opioids and benzodiazepines and overdose: retrospective analysis BMJ.  2017;356.
  * 22 Li Y., Delcher C., Wei Y.-J. J., et al. Risk of opioid overdose associated with concomitant use of opioids and skeletal muscle relaxants: a population-based cohort study Clinical Pharmacology & Therapeutics.  2020;108:81–89.
  * 23 Khan N. F., Bykov K., Barnett M. L., Glynn R. J., Vine S. M., Gagne J. J.. Comparative risk of opioid overdose with concomitant use of prescription opioids and skeletal muscle relaxants Neurology.  2022;99:e1432–e1442.
  * 24 Gomes T., Juurlink D. N., Antoniou T., Mamdani M. M., Paterson J. M., Brink W.. Gabapentin, opioids, and the risk of opioid-related death: a population-based nested case–control study PLoS Medicine.  2017;14:e1002396.
  * 25 Kuehn B. M.. Growing role of gabapentin in opioid-related overdoses highlights misuse potential and off-label prescribing practices JAMA.  2022;328:1283–1285.
  * 26 Corriere M. A., Daniel L. L., Dickson A. L., et al. Concurrent Gabapentin and Opioid Use and Risk of Mortality in Medicare Recipients with Non-Cancer Pain Clinical Pharmacology & Therapeutics.  2023;114:1050–1057.
  * 27 Peckham A. M., Fairman K. A., Sclar D. A.. All-cause and drug-related medical events associated with overuse of gabapentin and/or opioid medications: a retrospective cohort analysis of a commercially insured US population Drug Safety.  2018;41:213–228.
  * 28 Olopoenia A., Camelo-Castillo W., Qato D. M., et al. Adverse outcomes associated with concurrent gabapentin, opioid, and benzodiazepine utilization: A nested case-control study The Lancet Regional Health–Americas.  2022;13.
  * 29 Bykov K., Bateman B. T., Franklin J. M., Vine S. M., Patorno E.. Association of gabapentinoids with the risk of opioid-related adverse events in surgical patients in the United States JAMA Network Open.  2020;3:e2031647–e2031647.
  * 30 Vowles K. E., Witkiewitz K., Sowden G., Ashworth J.. Acceptance and commitment therapy for chronic pain: evidence of mediation and clinically significant change following an abbreviated interdisciplinary program of rehabilitation The Journal of Pain.  2014;15:101–113.
  * 31 Aytur S. A., Ray K. L., Meier S. K., et al. Neural mechanisms of acceptance and commitment therapy for chronic pain: a network-based fMRI approach Frontiers in Human Neuroscience.  2021;15:587018.
  * 32 Roberts R. L., Ledermann K., Garland E. L.. Mindfulness-oriented recovery enhancement improves negative emotion regulation among opioid-treated chronic pain patients by increasing interoceptive awareness Journal of Psychosomatic Research.  2022;152:110677.
  * 33 Thomas E. A., Garland E. L.. Mindfulness is associated with increased hedonic capacity among chronic pain patients receiving extended opioid pharmacotherapy The Clinical Journal of Pain.  2017;33:166.
  * 34 Priddy S. E., Hanley A. W., Riquino M. R., Platt K. A., Baker A. K., Garland E. L.. Dispositional mindfulness and prescription opioid misuse among chronic pain patients: craving and attention to positive information as mediating mechanisms Drug and Alcohol Dependence.  2018;188:86–93.
  * 35 Xie H., Guarino H., Moore S. K., et al. Web-based cognitive behavior therapy for chronic pain patients with aberrant drug-related behavior: How did it work and for whom? Journal of Behavioral Medicine.  2021;44:704–714.
  * 36 Craft W. H., Tegge A. N., Bickel W. K.. Episodic future thinking reduces chronic pain severity: A proof of concept study Drug and Alcohol Dependence.  2020;215:108250.
  * 37 Rupp K., Riley G. F.. Longitudinal patterns of Medicaid and Medicare coverage among disability cash benefit awardees Soc. Sec. Bull..  2012;72:19.
  * 38 Samples H., Williams A. R., Olfson M., Crystal S.. Risk factors for discontinuation of buprenorphine treatment for opioid use disorders in a multi-state sample of Medicaid enrollees Journal of Substance Abuse Treatment.  2018;95:9–17.
  * 39 Cohen S. P., Vase L., Hooten W. M.. Chronic pain: an update on burden, best practices, and new advances The Lancet.  2021;397:2082–2097.
  * 40 Fox M. H., Reichard A.. Disability, health, and multiple chronic conditions among people eligible for both medicare and medicaid, 2005–2010 Preventing Chronic Disease.  2013;10:E157.
  * 41 Whitney D. G., Hurvitz E. A., Peterson M. D.. Cardiometabolic disease, depressive symptoms, and sleep disorders in middle-aged adults with functional disabilities: NHANES 2007–2014 Disability and Rehabilitation.  2018:1–6.
  * 42 Mills S. E., Nicolson K. P., Smith B. H.. Chronic pain: a review of its epidemiology and associated factors in population-based studies British Journal of Anaesthesia.  2019;123:e273–e283.
  * 43 Sullivan M. D., Edlund M. J., Fan M.-Y., DeVries A., Braden J. B., Martin B. C.. Risks for possible and probable opioid misuse among recipients of chronic opioid therapy in commercial and medicaid insurance plans: The TROUP Study Pain.  2010;150:332–339.
  * 44 Segrin C., McNelis M., Pavlich C. A.. Indirect effects of loneliness on substance use through stress Health Communication.  2018;33:513–518.
  * 45 Cance J. D., Saavedra L. M., Wondimu B., Scaglione N. M., Hairgrove S., Graham P. W.. Examining the relationship between social connection and opioid misuse: a systematic review Substance Use & Misuse.  2021;56:1493–1507.
  * 46 Zoorob M. J., Salemi J. L.. Bowling alone, dying together: the role of social capital in mitigating the drug overdose epidemic in the United States. Drug and Alcohol Dependence.  2017;173:1–9.
  * 47 McLean K.. "There’s nothing here": deindustrialization as risk environment for overdose. International Journal of Drug Policy.  2016;29:19–26.
  * 48 Monnat S. M.. Factors associated with county-level differences in us drug-related mortality rates American Journal of Preventive Medicine.  2018;54:611–619.
  * 49 Krueger A. B.. Where have all the workers gone? An inquiry into the decline of the US labor force participation rate Brookings papers on economic activity.  2017;2017:1.
  * 50 Dasgupta N., Beletsky L., Ciccarone D.. Opioid crisis: No easy fix to its social and economic determinants. American Journal of Public Health.  2018;108:182–186.
  * 51 Ledingham E., Adams R. S., Heaphy D., Duarte A., Reif S.. Perspectives of adults with disabilities and opioid misuse: Qualitative findings illuminating experiences with stigma and substance use treatment Disability and Health Journal.  2022;15:101292.
  * 52 Mannes Z. L., Malte C. A., Olfson M., et al. Increasing risk of cannabis use disorder among US veterans with chronic pain: 2005-2019. Pain.  2023\.
  * 53 Miller G. F., Guy Jr G. P., Zhang K., Mikosz C. A., Xu L.. Prevalence of nonopioid and opioid prescriptions among commercially insured patients with chronic pain Pain Medicine.  2019;20:1948–1954.
  * 54 Cochran G., Gordon A. J., Lo-Ciganic W.-H., et al. An examination of claims-based predictors of overdose from a large Medicaid program Medical Care.  2017;55:291.
  * 55 Samples H., Williams A. R., Crystal S., Olfson M.. Psychosocial and behavioral therapy in conjunction with medication for opioid use disorder: Patterns, predictors, and association with buprenorphine treatment outcomes Journal of Substance Abuse Treatment.  2022;139:108774.
  * 56 Haneuse S., Rotnitzky A.. Estimation of the effect of interventions that modify the received treatment Statistics in Medicine.  2013;32:5260–5277.
  * 57 Muñoz I. D., Van Der Laan M.. Population intervention causal effects based on stochastic interventions Biometrics.  2012;68:541–549.
  * 58 Young J. G., Hernán M. A., Robins J. M.. Identification, estimation and approximation of risk under interventions that depend on the natural value of treatment using observational data Epidemiologic Methods.  2014;3:1–19.
  * 59 Díaz I., Williams N., Hoffman K. L., Schenck E. J.. Nonparametric causal effects based on longitudinal modified treatment policies Journal of the American Statistical Association.  2023;118:846–857.
  * 60 van der Laan M. J., Polley E. C., Hubbard A. E.. Super Learner Statistical Applications in Genetics & Molecular Biology.  2007;6:Article 25.
  * 61 Chen T., He T., Benesty M., et al. xgboost: Extreme Gradient Boosting 2022\.  R package version 1.6.0.1.
  * 62 Friedman J. H.. Multivariate Adaptive Regression Splines The Annals of Statistics.  1991;19:1 – 67.
  * 63 Venables W. N., Ripley B. D.. Modern Applied Statistics with S.  New York: Springerfourth ed. 2002.  ISBN 0-387-95457-0.
  * 64 Wright M. N., Ziegler A.. ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R Journal of Statistical Software.  2017;77:1–17.
  * 65 George S. Z., Goode A. P.. Physical therapy and opioid use for musculoskeletal pain management: competitors or companions? Pain Peports.  2020;5:e827.
  * 66 Kaiser Family Foundation . Medicaid’s Role in Addressing the Opioid Epidemic 2019, url: https://www.kff.org/infographic/medicaids-role-in-addressing-opioid-epidemic/.
  * 67 McNeely J., Owens E., Bone E., et al. Sensitivity of paid insurance claims data for identifying hospital patients with opioid use disorder Addiction Health Services Research Conference.  2020:Available: https://osf.io/7pz5b/, Date accessed: 29 Dec 2020.
  * 68 Medicaid Data Quality Atlas, howpublished = https://www.medicaid.gov/dq-atlas/welcome, note = Accessed: 2022-11-10
  * 69 Foundation K. F.. Status of State Medicaid Expansion Decisions: Interactive Map 2020, url: https://www.kff.org/medicaid/issue-brief/status-of-state-medicaid-expansion-decisions-interactive-map/.

Table S0:

Characteristic1 |  Physical Disability & Chronic Pain |
---|---|---
N = 6,133 |  |
|  Chronic Pain |
N = 67,438 |  |
| Number (%) |
Demographics (months 1-6) |  |
Age (median, IQR) | 54 (50, 59) | 50 (44, 56)
Sex |  |
Female | 3,680 (60) | 36,409 (54)
Male | 2,453 (40) | 31,029 (46)
Race/Ethnicity |  |
AIAN, non-Hispanic2 | 67 (1.2) | 1,270 (2.3)
Asian, non-Hispanic | 56 (1.0) | 2,879 (5.2)
Black, non-Hispanic | 1,491 (28) | 7,982 (14)
Hawaiian/Pacific Islander |  32 (0.6) | 451 (0.8)
Hispanic, all races | 513 (9.5) | 9,221 (17)
Multiracial, non-Hispanic |  3 (<0.1) | 52 (<0.1)
White, non-Hispanic | 3,228 (60) | 33,463 (60)
Unknown | 743 | 12,120
Primary Language English | 4,806 (95) | 52,049 (89)
Unknown | 1,054 | 8,645
Married/Partnered | 502 (13) | 6,746 (26)
Unknown | 2,262 | 41,455
High Income ($>138$FPL2) | 107 (1.7) | 1,528 (2.3)
Household Size |  |
1 | 1,347 (73) | 16,436 (72)
2 | 266 (14) | 3,293 (14)
$\geq 3$ | 229 (12) | 3,251 (14)
Unknown | 4,291 | 44,458
Veteran |  6 (0.7) | 177 (0.9)
Unknown | 5,250 | 47,819
TANF Benefits2 | 89 (1.6) | 4,193 (7.6)
Unknown | 650 | 12,315
SSI Benefits2 |  |
Mandatory or Optional | 471 (12) | 157 (0.7)
Not Applicable | 3,453 (88) | 22,932 (99)
Unknown | 2,209 | 44,349
Psychiatric Conditions & Counseling (months 1-6)
Bipolar | 410 (6.7) | 1,399 (2.1)
Anxiety | 1,531 (25) | 14,205 (21)
ADHD2 | 57 (0.9) | 761 (1.1)
Depression | 1,473 (24) | 9,954 (15)
Other Mental Illness | 618 (10) | 4,948 (7.3)
Mental Health Counseling | 1,006 (16) | 5,460 (8.1)
Psychiatric Conditions & Counseling (months 7-12)
Anxiety | 651 (11) | 5,688 (8.4)
Depression | 474 (7.7) | 4,176 (6.2)
Mental Health Counseling | 1,075 (18) | 6,196 (9.2)
Opioid Pain Management (months 7-12)
Opioid Pain Prescription | 3,761 (61) | 27,246 (40)
Hydrocodone | 1,599 (26) | 13,334 (20)
Oxycodone | 1,790 (29) | 10,049 (15)
Fentanyl | 464 (7.6) | 3594 (5.3)
Morphine | 419 (6.8) | 2,421 (3.6)
Hydromorphone | 294 (4.8) | 2,231 (3.3)
Codeine | 283 (4.6) | 2,365 (3.5)
Buprenorphine | 27 (0.4) | 170 (0.3)
Other | 75 (1.2) | 376 (0.6)
Dose, Duration |  |
Max Daily Dose (MME)3 | 25 (0, 60) | 0 (0, 36)
Proportion of opioid days | 0.05 (0, 0.69) | 0.00 (0.00, 0.08)
High-Risk Prescribing Practices |  |
Distinct Prescribers3 | 1 (0, 2) | 0 (0, 1)
Co-prescription3 |  |
Benzodiazepine3 | 809 (13) | 3,176 (4.7)
Stimulant3 | 58 (0.9) | 476 (0.7)
Gabapentinoid3 | 1,437 (23) | 6,135 (9.1)
Muscle Relaxant3 | 1,284 (21) | 6,935 (10)
Non-opioid Pain Management (months 7-12)
Nonopioid Pain Prescription | 5,213 (85) | 45,250 (67)
Antidepressant | 2,973 (49) | 20,846 (31)
Muscle Relaxant | 2,299 (38) | 16,630 (25)
Antiinflammatory & Antirheumatic | 2,886 (53) | 26,825 (40)
Topical | 464 (7.6) | 2,548 (3.8)
Other Analgesic & Antipyretic | 3,382 (55) | 21,541 (32)
Physical Therapy | 1,581 (26) | 17,932 (27)
Multimodal Pain Treatment | 1,554 (25) | 17,975 (27)
Ablative Techniques | 82 (1.3) | 646 (1.0)
Acupuncture | 15 (0.2) | 506 (0.8)
Blocks | 353 (5.8) | 3,436 (5.1)
Botulinum Toxin Injection | 59 (1.0) | 436 (0.6)
Chiropractic | 207 (3.4) | 3,204 (4.8)
Electrical Nerve Stimulation | 133 (2.2) | 594 (0.9)
Epidural Steroid | 296 (4.8) | 2,796 (4.1)
Intrathecal Drug Therapy | 21 (0.3) | 31 (<0.1)
Massage Therapy | 725 (12) | 10,221 (15)
Trigger Point Injection | 144 (2.3) | 1,235 (1.8)
Outcomes (months 13+) |  |
OUD by 18 months2 | 410 (7.0) | 1,894 (3.1)
OUD by 24 months2 | 517 (9.7) | 2,280 (4.1)

1 Median (IQR) for continuous measures; N (%) for categorical measures

2 Abbreviations:

AIAN = American Indian and Alaska Native

TANF = Temporary Assistance for Needy Families

SSI = Supplemental Security Income

FPL = Federal Poverty Level

ADD/ADHD = Attention Deficit (Hyperactivity) Disorder

OUD = Opioid Use Disorder

3 Among patients with any opioid prescription

##  S1 Additional detail on data and cohort

We included data from the following states that implemented Medicaid expansion
by 2014: ND, VT, NH, CA, OR, MI, IA, NV, OH, IL, NY, MD, MA, RI, HI, WV, WA,
KY, DE, AZ, NJ, MN, NM, CT, CO, AR. We excluded MD due to unreliable diagnosis
code data68) 69. We restricted the analysis to expansion states, because they
nearly all low-income ($\leq$138% of the Federal Poverty Limit) non-elderly
adults are eligible for coverage in these states 69. We permitted enrollment
only on or after January 2, 2016, because those with an enrollment date of
January 1, 2016 were likely continuously enrolled from 2015, making their
initial enrollment date unknown. It was important to know the initial
enrollment date, because we limited to the first 24 months of enrollment as
after this point, individuals with SSDI would transition to Medicare as the
primary payer.

###  S1.1 Additional detail on subgroup definitions

We had two analytic subgroups defined by chronic pain and physical disability
status: 1) co-occurring chronic pain and physical disability (CPPD), and 2)
chronic pain without physical disability (CP). Having these subgroups be well-
defined is necessary for our research question and causal estimands of
interest to also be well-defined, and for understanding to whom our inferences
apply.

In the pain literature, those with cancer pain are typically excluded for
reasons related to the above, as pain treatment priorities would likely be
different for this subgroup. Here, we excluded individuals with a cancer
diagnosis or in hospice or palliative care from our cohort. Identifying
beneficiaries with a likely physical disability was impeded by not having
access to the beneficiary’s qualifying disability coupled with extensive
missingness on whether the beneficiary was receiving Medicaid due to SSDI/SSI.
Consequently, we used eligibility codes to identify beneficiaries who likely
qualified for Medicaid due to a disability, then, used a combination of
eligibility codes and exclusion criteria as a means for identifying those with
a likely physical disability. Even though we do not explicitly compare effect
estimates between subgroups, we applied the following exclusion criteria to
the entire cohort to result in more homogeneous, well-defined subgroups who
may be “at risk” of pain management treatments. We excluded those who were
disabled due to hearing, vision, severe speech, dementia, epilepsy, severe
intellectual impairments, dementias, intellectual disabilities, cerebral
palsy, tuberculosis, epilepsy, and schizophrenia and other serious mental
illness (SMI) with psychosis from our cohort.

###  S1.2 Additional detail on covariates

We use the initial 6-month period to characterize the beneficiary’s baseline
covariates: age in years, sex, race/ethnicity (non-Hispanic Asian; non-
Hispanic American Indian and Alaska Native (AIAN) or Hawaiian/Pacific
Islander; non-Hispanic Black; Hispanic, all races; non-Hispanic White; and
non-Hispanic Multiracial or unknown), English as their primary language,
marriage/partnership status (currently partnered vs. not), household size (1,
2, and $>2$), veteran status (yes/no), income likely $>$133% FPL (yes/no,
determined using the latest washout eligibility code and demographic income
information), any inpatient or outpatient DSM-5 diagnosis of: bipolar
disorder, any anxiety disorder, attention deficit disorder/ attention deficit
hyperactivity disorder (ADD/ADHD), any depressive disorder, or other mental
illness (e.g., Anorexia, personality disorders, as operationalized by
samples2018risk 38). All relevant ICD codes and operationalizatons are
detailed in the Github repository.

###  S1.3 Additional detail on statistical analysis

We assume observed data $\mathbf{O}=(\mathbf{W},S,\mathbf{A},\Delta,\Delta
Y),$ where: $\mathbf{W}$ represents the covariates measured during months 1-6
(i.e., during the washout period); $S$ represents the chronic condition group
(i.e., CPPD or CP), characterized during months 1-6; $\mathbf{A}$ represent
pain management variables, measured during months 7-12; $\Delta$ represents an
indicator of remaining uncensored after the initial enrollment date (24 months
for the primary analysis and 18 months for the secondary analysis); finally,
$Y$ represents the outcome of incident OUD, which is observed among those who
remain uncensored.

We estimated the effect of each pain management treatment of interest on
incident OUD, adjusting for covariates, holding other pain management
treatments at their observed levels, and stratifying by chronic condition
subgroup. Using the notation given above, this effect can be written:
$\mathsf{E}(Y^{d_{n}(\mathbf{A}),\Delta=1}-Y^{\mathbf{A},\Delta=1}\mid S=s),$
where $\mathsf{E}(Y^{\mathbf{A},\Delta=1}\mid S=s)$ denotes the expected value
of the counterfactual outcome had the set of pain management treatments
($\mathbf{A}$) not been intervened on (i.e., remained as observed) and had no
one been censored in subgroup $S=s$, and where
$\mathsf{E}(Y^{d_{n}(\mathbf{A}),\Delta=1}\mid S=s)$ denotes the expected
value of the counterfactual outcome had the particular pain management
treatment $A_{n}$ been intervened on as dictated by the function
$d_{n}(\mathbf{A})$ but the remaining pain management treatments stayed as
observed and had no one been censored in subgroup $S=s$. We considered $N=9$
pain management treatments: $\mathbf{A}=(A_{1},...,A_{9})$. For binary pain
management treatments (e.g., presence of co-prescription with gabapentin, non-
opioid pain prescription, physical therapy, other pain treatment), we
estimated the effect if everyone in the subgroup received the treatment versus
observed treatment. So, for example, if $A_{1}$ was a binary treatment, we
would estimate the effect had everyone received treatment $A_{1}$, but the
remaining pain management treatments remained at their observed levels, i.e.,
$d_{1}(\mathbf{A})=(1,A_{2},...,A_{9})$ versus had there been no intervention
on the set of treatments, $\mathbf{A}$. For non-binary pain management
treatments (maximum daily opioid dose, proportion of opioid days, number of
opioid prescribers), we estimated the effect if everyone in the subgroup had
their value increase by 20% (i.e., multiplied each person’s value by 1.2)
versus no intervention on treatment. So, for example, if $A_{2}$ was a non-
binary treatment, we would estimate the effect had everyone had 1.2 times
their observed value but the remaining pain management treatments remained at
their observed levels, i.e., $d_{2}(\mathbf{A})=(A_{1},1.2\times
A_{2},A_{3},...,A_{9}).$

We consider causal effects that isolate the effect of each treatment variable
(holding the other treatment variables at their observed levels and adjusting
for confounding and censoring) on the outcome under identifying assumptions
of: 1) conditional exchangeability, meaning that there is no
unobserved/unmeasured confounding of the relationship between the set of
treatments and outcome conditional on covariates and that there is no
unobserved/unmeasured confounding between censoring and the outcome
conditional on the covariates and treatment; and 2) positivity, meaning that
if it is possible to find a beneficiary with observed data
$(\mathbf{a},c,\mathbf{w})$, then it is possible to find a beneficiary with
data $(d_{n}(\mathbf{a}),1,\mathbf{w})$ for $n\in\\{1:9\\}$. In other words,
this states that: 1) if it is possible to find a beneficiary with covariates
$\mathbf{w}$ and treatment set $\mathbf{a}$, it is possible to find a
beneficiary with covariates $\mathbf{w}$ and treatment set $d_{n}(\mathbf{a})$
for $n\in\\{1:9\\}$, and 2) for every observed $(\mathbf{a},\mathbf{w})$,
there is a nonzero probability of not being censored.


# Unsupervised Part Segmentation through Disentangling
Appearance and Shape

Shilong Liu1 Lei Zhang2 Xiao Yang1 Hang Su1 Jun Zhu1
1 Dept. of Comp. Sci. and Tech Corresponding author BNRist Center  Institute
for AI  Tsinghua-Bosch Joint ML Center
1 Tsinghua University  Beijing  100084  China 2 Microsoft Corporation
{liusl20<EMAIL_ADDRESS><EMAIL_ADDRESS>{suhangss,
<EMAIL_ADDRESS>

###### Abstract

We study the problem of unsupervised discovery and segmentation of object
parts, which, as an intermediate local representation, are capable of finding
intrinsic object structure and providing more explainable recognition results.
Recent unsupervised methods have greatly relaxed the dependency on annotated
data which are costly to obtain, but still rely on additional information such
as object segmentation mask or saliency map. To remove such a dependency and
further improve the part segmentation performance, we develop a novel approach
by disentangling the appearance and shape representations of object parts
followed with reconstruction losses without using additional object mask
information. To avoid degenerated solutions, a bottleneck block is designed to
squeeze and expand the appearance representation, leading to a more effective
disentanglement between geometry and appearance. Combined with a self-
supervised part classification loss and an improved geometry concentration
constraint, we can segment more consistent parts with semantic meanings.
Comprehensive experiments on a wide variety of objects such as face, bird, and
PASCAL VOC objects demonstrate the effectiveness of the proposed method.

Figure 1: Semantic consistency of object parts. Segmentation regions of the
same part should be semantically consistent across object instances and robust
to appearance and shape changes. Best view in colors.

##  1 Introduction

Object parts and landmarks are two widely used intermediate representations of
object local structures, which have received increasing attention recently for
their robustness to the variations of viewpoint and appearance [12]. While
there have been many related works on object landmark detection by either
supervised learning [41, 25] or unsupervised discovery [13, 40] , semantic
part segmentation is relatively less studied [12] due to the costly efforts
required for data annotation.

This paper primarily focuses on unsupervised semantic part segmentation, as
object parts provide richer information about intrinsic object structures
which are complementary to landmark points. To alleviate the efforts in
annotations, we investigate the unsupervised learning strategy for automatic
semantic part segmentation [12]. It has also shown an encouraging impact on
single-view 3D object reconstruction by using self-supervised learned semantic
parts which enforces semantic consistency between the reconstructed 3D mesh
and original 2D images [18].

A good object part segmentation should satisfy two essential constraints on
semantics and geometry, which correspond to object appearance and shape,
respectively. The semantic constraint means that segmentation regions of the
same part should be semantically consistent across object instances and robust
to appearance and shape changes. The geometric constraint means a part
segmentation region should be locally connected and the union of all parts
should entirely cover the corresponding object. These two constraints can be
seen in Fig. 1, in which the parts with the same semantic information (e.g.,
heads of bird) are covered by the same segmentation color (e.g., red) in
different images, and the union of all parts provides a good coverage over an
object. Hung et al. [12] also discuss similar constraints including geometric
concentration, robustness to variations, semantic consistency, and objects as
union of parts, and accordingly develop an unsupervised solution for semantic
part segmentation. However, for the objects-as-union-of-parts constraint,
their solution lacks a principled way to enforce such a constraint and has to
take a dependency on saliency map, which is not always reliable and has to be
replaced with object segmentation mask in some cases 111For example, for CUB-
Birds, the open-source implementation of SCOPS uses ground truth silhouettes
rather than saliency maps and crops birds w.r.t bounding boxes rather than
using the original images. See https://github.com/NVlabs/SCOPS#scops-on-
caltech-ucsd-birds. This dependency makes it hard to scale up with more data
or to other object categories.

To relax the dependency on saliency map (or object segmentation mask) and
further improve the performance of part segmentation, we develop a novel
unsupervised learning approach by disentangling the appearance and shape
representations of semantic parts, followed with reconstruction losses without
using any additional object segmentation mask or saliency map 222Note that we
still assume the availability of object bounding boxes, which are used in all
unsupervised object landmark/part learning works to bound the learning problem
to single object images.. The idea of disentanglement and synthesis is
inspired by [13], despite that it was proposed for automatic landmark point
discovery whereas this work is for semantic part segmentation. By
disentangling the appearance and shape representations of a target object
category, we can effectively add semantic and geometric constraints on the
appearance part and the shape part, respectively.

More specifically, our framework is an encoder-decoder based deep neural
network. The network firstly takes a pair of images which only differ by
geometric transformations, and then uses two encoders to extract the
appearance and shape representations of the two images. After exchanging the
appearance representations between two images, a decoder is used to
reconstruct the input images to close the loop for unsupervised learning. To
avoid degenerated solutions, we design a bottleneck block to squeeze and
expand the appearance representation, leading to a more complete
disentanglement between appearance and shape, which in return yields a better
shape representation for part segmentation. On the squeezed appearance
representation, we apply a classification loss to learn distinctive part
features while enforcing cross-instance semantic consistency. On the
disentangled shape representation, we use an improved geometric concentration
loss with a special treatment for background pixels. This concentration loss,
together with the reconstruction loss, essentially ensures the geometric
constraint. To validate the effectiveness of the proposed framework, we
perform comprehensive experiments on a wide variety of objects such as face,
bird, and PASCAL VOC objects, and obtain superior part segmentation results
compared to previous works.

Our contribution can be summarized as follows:

  * •

To our best knowledge, this is the first attempt that can unsupervised learn
semantic part segmentation without using any addition object segmentation mask
(or saliency map).

  * •

By utilizing a squeeze-and-expand bottleneck block, the proposed method can
effectively disentangle the appearance and shape representations of objects
and remove the dependency of using additional segmentation masks.

##  2 Related Work

###  2.1 Unsupervised Landmark Discovery

There have been many efforts to learn landmarks in a supervised manner [43,
42, 39, 37, 30, 3, 25, 35]. However, supervised landmark detection is largely
limited by the availability of annotated data and mainly works for popular
categories, such as face [21, 23], bird [32], etc. To avoid relying massive
annotated data, unsupervised landmark discovery has attracted much more
attention in recent years. Thewlis et al. [29] proposed to address landmark
discovery by using the equivariance constraint and then expanding the
formulation to a dense situation [28]. Jakab et al. [13] proposed to learn
landmarks by first disentangling pose and appearance of an image and then
recombining them for synthesizing a new image. We term this approach as a
disentanglement and synthesis paradigm. This work shows that combining
disentanglement with synthesis can effectively lead to the desired
equivariance property, which has inspired many follow-up works, including this
paper. Many subsequent works suggest improvement upon [13]. For example,
Thewlis et al. [27] proposed to add a vector exchanging procedure for better
semantic consistency across instances. Jakab et al. [14] then tried to learn
human-interpretable landmarks from unpaired labels and Xu et al. [38] designed
a cycle framework to learn from unpaired data. Nevertheless, such works are
limited to landmark discovery, whereas our work is to obtain part segmentation
from unlabeled data, which is a more challenging problem.

###  2.2 Part Segmentation

Our work is closely related to weakly supervised object co-segmentation. Co-
segmentation aims at identifying pixels of interested foreground objects from
background. There have been many related works on this subject [11, 16, 1, 26,
2, 24, 10], but they mainly deal with semantic information at the object level
rather than at the part level. Besides, most works need to take a whole
dataset as input and perform co-segmentation, while we are more interested in
learning a model that can predict part level semantic information, shown as a
part segmentation map, for each image.

Several recent works have attempted to learn part-level information in an
unsupervised manner. Hung et al. [12] proposed to learn part features that are
semantically consistent across images and have achieved good results in their
paper. However, it takes a dependency on object saliency map, which is not
always reliable and has to be replaced with object segmentation mask in some
cases, to distinguish a foreground object from background. In this work, we
remove this dependency and make semantic part segmentation applicable to more
object categories. Lorenz et al. [22] proposed to disentangle object shape and
appearance to obtain a part modeling result, which has a similar goal to this
work. However they make an elliptical assumption for each part and do not
assign a part label for each pixel. Thus this approach does not work well for
images whose main object occupies a large area, like human faces, as it cannot
generate a complete segmentation map.

Figure 2: The overall framework of our model. The network firstly takes a pair
of images which only differ by geometric transformations and then uses two
encoders to extract the appearance $\mathbf{A}^{(i)},i=1,2$ and shape
representation $\mathbf{S}^{(i)},i=1,2$ of the two images. In the middle are
two bottleneck blocks, each for squeezing the appearance representation and
combining it with the shape representation of another image by an expansion
operation. After exchanging the appearance representations between two images,
a decoder is used to reconstruct the input images to close the loop for
unsupervised learning.

##  3 Methodology

As discussed in Sec. 1, we hope the learned part segmentation results could
meet both the semantic constraint and geometry constraint without additional
saliency map or segmentation mask. To apply the two constraints on the
appearance and shape features separately, we leverage an encoder-decoder
neural network with a squeeze-and-expand bottleneck block in the middle to
disentangle the appearance and shape representations, which will be described
in Sec. 3.1. To ensure the semantic constraint, we use a reconstruction loss
and a classification loss, which will be detailed in Sec. 3.2 and Sec. 3.3
respectively. For the geometry constraint, we improve the concentration loss
and describe the details in Sec. 3.4.

###  3.1 Overall Framework

The overall framework is an encoder-decoder based deep neural network, which
includes three modules: a part segmentation encoder ${E}_{s}$, an appearance
encoder ${E}_{a}$, and a decoder $D$ for image generation. Between the two
encoders and the decoder, a bottleneck block is designed to squeeze and expand
the appearance representation, for a more effective disentanglement between
appearance and shape. The whole framework is shown in Fig. 2.

To learn such a neural network, we assume an image collection of the
interested object category is available as training data. Given an image from
the training dataset, we firstly construct a pair of images which only differ
by geometric transformations. That is, the transformed objects included in the
two images have the same appearance but different shape representations. As
suggested in previous works [13, 22], we use the thin-plate-spline
transformation (TPS) [7, 33] to obtain two transformed images
$x^{(1)},x^{(2)}$. Using $x^{(1)}$ and $x^{(2)}$ as inputs, we can extract an
appearance feature map $\mathbf{A}\in\mathbb{R}^{L\times H\times W}$ and a
part segmentation map $\mathbf{S}\in\mathbb{R}^{(K+1)\times H\times W}$
(corresponding to shape) by $E_{a}$ and $E_{s}$ for each image as

$\displaystyle\mathbf{A}^{(i)}=E_{a}(x^{(i)}),\ \ \
\mathbf{S}^{(i)}=E_{s}(x^{(i)}),\ i=1,2$ (1)

where $K$ denotes the number of object parts, $L$ denotes the dimension of
appearance features, and $H\times W$ is the resolution of two output tensors,
including both the appearance feature map and the part segmentation map. The
$0$-th channel is used for representing the background region and the $1$-st
to $K$-th channels are used for representing $K$ object parts. For notational
convenience, we denote $K^{\prime}=K+1$. Moreover, each part segmentation map
$\mathbf{S}^{(i)},i=1,2$ is normalized along the channel dimension by
$\operatorname{softmax}$.

Appearance Transfer Bottleneck. To disentangle the appearance and shape
representations, reconstruction is normally performed to provide a self-
supervised loss 333In the paper, we mainly use “self-supervised” to refer to a
specific loss, but use “unsupervised” to refer to the overall part
segmentation problem which does not require supervised annotations. after
exchanging the shape (or appearance) representations between two input
samples. However, directly performing reconstruction will lead to a
degenerated solution. For example, the part segmentation map $\mathbf{S}$ may
contain some appearance information as well as shape information, making
itself capable of reconstructing the original image, which is an undesired
trivial solution. Inspired by [13] and [22], we design a simple appearance
transfer bottleneck to squeeze and expand the part appearance features, aiming
at generating a rendered appearance feature map that contains appearance
feature from only one image and shape information from another image.

In the squeeze step (Eq. (2)), each channel in the part segmentation map can
be regarded as an attention map, which is used to weighted average the learned
appearance features over all pixels. This step reduces the dimension of the
appearance feature and squeezes out the shape information of $x^{(1)}$. In the
expand step (Eq. (3)), the squeezed part appearance feature of $x^{(1)}$ is
expanded according to the part segmentation map of $x^{(2)}$, treating each
response channel as an attention map to generate a rendered appearance feature
map. This appearance transfer bottleneck, as well as the reconstruction,
effectively helps avoid degenerated disentanglement learning.

For the sake of simplicity, we only write out the process of transferring the
appearance of $x^{(1)}$ to combine with the shape of $x^{(2)}$, and it is
straightforward to generalize to the inverse direction.

First, we reshape the part segmentation map $\mathbf{S}^{(1)}$ to
$\bar{\mathbf{S}}^{(1)}\in\mathbb{R}^{K^{\prime}\times(HW)}$ and the
appearance feature map $\mathbf{A}^{(1)}$ to
$\bar{\mathbf{A}}^{(1)}\in\mathbb{R}^{L\times(HW)}$ for notational
convenience. Then, we squeeze the part appearance feature by using the part
segmentation map to obtain $\mathbf{F}^{(1)}\in\mathbb{R}^{K^{\prime}\times
L}$ as follows to push out any shape information:

$\displaystyle\mathbf{D}$
$\displaystyle=\operatorname{diag}(\bar{\mathbf{S}}^{(1)}\mathbf{1}_{HW}),$
(2) $\displaystyle\mathbf{F}^{(1)}$
$\displaystyle=\mathbf{D}^{-1}\bar{\mathbf{S}}^{(1)}(\bar{\mathbf{A}}^{(1)})^{T},$

where $\mathbf{1}_{HW}$ is a vector of ones with dimension $HW$ and
$\mathbf{D}^{-1}$ is to perform row normalization on $\bar{\mathbf{S}}^{(1)}$.

For the next step, we expand the squeezed part appearance feature
$\mathbf{F}^{(1)}$ of $x^{(1)}$ according to the shape (part segmentation map)
of the other image $x^{(2)}$. After reshaping $\mathbf{S}^{(2)}$ to
$\bar{\mathbf{S}}^{(2)}\in\mathbb{R}^{K^{\prime}\times(HW)}$, we can obtain
the rendered appearance feature map $\bar{\mathbf{A}}^{1\to
2}\in\mathbb{R}^{L\times(HW)}$ as

$\bar{\mathbf{A}}^{1\to 2}=(\mathbf{F}^{(1)})^{T}\bar{\mathbf{S}}^{(2)}.$ (3)

By reshaping $\bar{\mathbf{A}}^{1\to 2}$ back to ${\mathbf{A}}^{1\to
2}\in\mathbb{R}^{L\times H\times W}$, we get the rendered appearance feature
map ${\mathbf{A}}^{1\to 2}$ which contains the appearance information of image
$x^{(1)}$ only and the shape information of $x^{(2)}$ alone, as we expected.
The inverse direction for ${\mathbf{A}}^{2\to 1}$ can be derived in a similar
way.

Image Generation. Using the rendered appearance feature maps $\mathbf{A}^{2\to
1}$ and $\mathbf{A}^{1\to 2}$ as inputs, we can obtain two reconstructed
images using the decoder $D$, as shown in the right part of Fig. 2.

$\hat{x}^{(1)}=D(\mathbf{A}^{2\to 1}),\ \ \ \hat{x}^{(2)}=D(\mathbf{A}^{1\to
2}).$ (4)

###  3.2 Reconstruction Loss

To ensure the semantic constraint being applied effectively on the
disentangled appearance and shape representations, we use reconstruction as
self-supervised signals. Some previous works [13, 22] have shown that the
disentangling and synthesis paradigm could help learn features with the
desired semantic consistency property, which is also termed as equivariance.
Our reconstruction loss is defined as

$\mathcal{L}_{rec}=\mathcal{P}(x^{(1)},\hat{x}^{(1)})+\mathcal{P}(x^{(2)},\hat{x}^{(2)}),$
(5)

where $\mathcal{P}$ represents the perceptual loss [15, 4] as in previous
works [13, 38].

###  3.3 Part Classification Loss

Similar to semantic segmentation, part segmentation can also be considered as
a multi-class classification problem to assign a class label to each pixel.
Although we do not have part labels for each pixel, we can perform
classification on the squeezed part appearance feature
$\mathbf{F}\in\mathbb{R}^{K^{\prime}\times L}$. That is, the $k$-th part
feature $\mathbf{F}_{k}\in\mathbb{R}^{L}$ should be classified correctly to
the $k$-th class.

Such a self-supervised classification loss not only helps learn distinctive
features for different parts, but also implies semantic consistency across
object instances, which leads to the desired feature invariance property.
Inspired by recent progress in face recognition feature learning [36, 19, 34,
9, 6], to further reduce the intra-class variance, we adopt the ArcFace loss
[6], which can be formulated as:

$\mathcal{L}_{cls}=\frac{1}{K}\sum_{i=1}^{K}\log\frac{e^{s(\cos(\theta_{i,i}+m))}}{e^{s(\cos(\theta_{i,i}+m))}+\sum_{j=1,j\neq
i}^{K}e^{s\cos\theta_{j,i}}}.$ (6)

The background channel is excluded from the loss for its inconsistency across
instances. To implement this loss function, we add a linear classification
layer in the model, which has parameters $W\in\mathbb{R}^{K\times L}$
representing $K$ feature basis of dimension $L$. In the loss function,
$\theta_{j,i}$ is the angle between part feature $F_{i}\in\mathbb{R}^{L}$ and
basis $W_{j}\in\mathbb{R}^{L}$, and $s$ and $m$ are two constants representing
radius scaling and angular margin penalty.

Unlike previous image-level constraints, we apply self-supervised
classification constraints directly on object parts. To our best knowledge,
this is the first time such a constraint is used in unsupervised keypoint
discovery or part segmentation. The use of this constraint effectively leads
to distinctive part features while keeping cross-instance semantic
consistency, which further helps improve the part segmentation performance as
demonstrated in Sec. 2.

###  3.4 Foreground/Background Concentration Loss

As mentioned in Sec. 1, the learned segmentation map should meet the geometry
constraint as well. Our method requires each pixel to be assigned with a part
label. Hence the requirement of an object being entirely covered by the union
of all parts is easy to implement.

For the concentration constraint which requires each object part to be locally
connected, we follow the concentration loss as proposed in [12] for foreground
object pixels but make a special modification for background pixels as they
are scattered around an object and do not follow the concentration constraint.
For foreground pixels:

$\mathcal{L}_{fg}=\frac{1}{K}\sum_{k=1}^{K}\sum_{u,v}\left\lVert\begin{bmatrix}u\\\
v\end{bmatrix}-\begin{bmatrix}c_{u}^{k}\\\
c_{v}^{k}\end{bmatrix}\right\rVert_{2}^{2}\cdot\mathbf{S}_{k,u,v}/z_{k},$ (7)

where $c_{u}^{k}$ and $c_{v}^{k}$ are the coordinates of the $k$-th part
center along axis $u,v$, and $z_{k}$ is a normalization term. $c_{u}^{k}$,
$c_{v}^{k}$, and $z_{k}$ can be calculated as follows:

$c_{u}^{k}=\sum_{u,v}u\cdot\mathbf{S}_{k,u,v}/z_{k},\
c_{v}^{k}=\sum_{u,v}v\cdot\mathbf{S}_{k,u,v}/z_{k},$ (8)

$z_{k}=\min\\{\sum_{u,v}\mathbf{S}_{k,u,v},1\\}.$ (9)

Note that Eq. (9) is slightly different from the definition of $z_{k}$ in
[12], as we found it works better for avoiding degenerated solutions in our
framework.

The foreground concentration loss works well for foreground pixels, but it is
inappropriate for background pixels which are scattered around an object and
lack semantic consistency. [12] is not suffered from this issue because it has
an extra input of saliency map to bound the region of interested object.
However, as we want to remove the dependency on an extra saliency map input,
we propose a background concentration loss as follows to separate out
background pixels. As the channel $0$ of $\mathbf{S}$ is regraded as
background, we can write the loss as:

$\mathcal{L}_{bg}=\sum_{u,v}\left\lVert h(\langle
u,v\rangle)\right\rVert^{2}\cdot\mathbf{S}_{0,u,v}/z_{0},$ (10)

where $h(\langle u,v\rangle)$ is the distance between the pixel $\langle
u,v\rangle$ and its nearest boundary which can be calculated as

$h(\langle u,v\rangle)=\min\\{u,W-u,v,H-v\\}.$ (11)

Compared with the foreground concentration loss as in Eq. (7), this constraint
encourages background pixels to be close to image boundaries. This constraint
is also based on the assumption that the target object has been cropped and
roughly centered with the help of an object bounding box, as mentioned in Sec.
3.1, which is a commonly used assumption in unsupervised object part learning.
Since it is hard to apply a semantic consistency constraint for diverse
background regions, this geometric constraint helps the proposed algorithm
focus on the main object in the center of an image.

###  3.5 Final Loss Function

The overall training loss is to minimize the weighted combination of all the
losses in Eq. (5), (6), (7), and (10):

$\mathcal{L}_{sum}=\lambda_{rec}\mathcal{L}_{rec}+\lambda_{cls}\mathcal{L}_{cls}+\lambda_{fg}\mathcal{L}_{fg}+\lambda_{bg}\mathcal{L}_{bg},$
(12)

where $\lambda_{rec},\lambda_{cls},\lambda_{fg}$ and $\lambda_{bg}$ are hyper
parameters for balancing the final loss.

##  4 Experiments

To evaluate the proposed approach, we conduct experiments on several datasets,
including CelebA [21] and AFLW [23] for human faces, CUB [8] for birds, and
PASCAL VOC [31] for other common object categories. As there is no ground
truth for unsupervised part segmentation, we mainly follow recent works [29,
40, 12] and use landmark regression as a proxy metric to evaluate the
performance of part segmentation for datasets with landmark annotations, e.g.
CelebA, AFLW, and CUB. For PASCAL VOC which does not have landmark
annotations, we evaluate the aggregated part segmentations with the foreground
segmentation using the IOU (intersection over union) metric.

The experiments on human faces (including CelebA and AFLW) and common objects
(including CUB and VOC) are detailed in Sec. 4.2 and Sec. 4.3, respectively.
We also present qualitative results of appearance and shape disentanglement
and transfer on DeepFashion [20] in Sec. 4.4.

###  4.1 Implementation Details

As mentioned in Sec 3, our framework consists of three modules: a part
segmentation encoder $E_{p}$, an appearance encoder $E_{a}$, and a decoder
$D$, all of which are implemented as deep convolutional networks using
PyTorch.

We use all layers before $\operatorname{layer4}$ (as denoted in the PyTorch
implementation) of an unpretrained ResNet34 as the appearance encoder, except
that we set the stride of $\operatorname{layer3}$ to $1$. For the part
segmentation encoder, we replace the backbone model with ResNet18 (using all
layers before $\operatorname{layer4}$ and setting the stride of
$\operatorname{layer3}$ to $1$ as well) and add an extra $1\times 1$
convolution layer at the end of the backbone. We resize all the images to
$128\times 128$ as input resolution, and the size of the output tensor is
$H\times W=32\times 32$. For different tasks, we will use different $K$. Our
generator is composed of seven convolution blocks, each containing a
convolution or transposed convolution layer, a batch normalization layer and a
ReLU layer, and an extra $3\times 3$ convolution layer at the end of the
generator. Of these seven blocks, the second and fourth contain $4\times 4$
transposed convolution layers for upsampling, and the rest contains $3\times
3$ convolution layers.

We initialize all model weights with random Gaussion noise
($\operatorname{sigma}=0.01$), and use Adam [17] as our optimizer, with
$(\beta_{1},\beta_{2})=(0.9,0.999)$, a weight decay of $5\times 10^{-4}$, and
a learning rate of $1\times 10^{-4}$. For the ArcFace loss, we set the
constants $s$ to 20 and $m$ to 0.5. More implementation details are available
in the supplementary materials.

###  4.2 Evaluation on Human Faces

We first test our method on two commonly used human face dataset: CelebA [21]
and AFLW [23]. CelebA is a large-scale face dataset containing more than 200K
images of celebrities collected from Internet. Each image in CelebA has 40
binary annotations, a bounding box and five landmark coordinates for each
face. Our setting is the same as that in [12]. We use the wild image for
training and testing, and filter out the images with face covering less than
$30\%$ of the pixel area. Since unsupervised results cannot be measured
directly, most previous works [28, 13, 22] use proxy tasks for evaluation,
such as landmark regression. Following [12], we first convert part
segmentations into landmarks by taking part centers as in Eq. (8) and then
learn a linear regressor to map the converted landmarks to ground truth
landmarks and evaluate the regression error on test data. The result can be
found in Table 1.

The result shows that our method, without using any additional information,
outperforms all other methods, including SCOPS which uses an additional
saliency map. Although this is a proxy evaluation metric, it in some ways
illustrates the good semantic consistency of our approach. To visualize the
part segmentation result, we show some resulting images from CelebA in Fig. 3.

Method | K=4 | K=8
---|---|---
ULD [29] | - | 31.30
Zhang et al. [40] | - | 40.82
IMM [13] | 19.42 | 8.74
Lorenz et al. [22] | 15.49 | 11.41
SCOPS [12](w/o saliency) | 46.62 | 22.11
SCOPS [12](with saliency) | 21.76 | 15.01
Ours | 15.39 | 12.26
Table 1: Landmark regression results on wild CelebA. We report the landmark
regression error in terms of mean L2 distance normalized by inter-ocular
distance. Note that the first three methods are specially designed for
landmarks discovery and Lorenzet et al.is not for precise part segmentation
for its elliptical shape assumption. We list them for reference.  Figure 3:
Visualization and ablation studies on wild CelebA. It shows that all the
$\mathcal{L}_{rec}$, $\mathcal{L}_{fg}$ and $\mathcal{L}_{bg}$ are essential
for avoiding a degenerated solution and the classification loss further
improves the distinctiveness of parts.

In addition to CelebA, we also test our method on another human face dataset:
AFLW [23]. AFLW contains 25K faces, each of which is annotated with 21
landmarks. AFLW is a more challenging dataset because the variation across
pictures is even larger. Some other works [27] provide a subset of AFLW and
select five face annotations for each face. We uses their unaligned dataset
for testing. We compare our method with relevant works in Table 2. In the
setting of $K=8$, we achieve a mean error of $13.13$, whereas SCOPS achieves
$16.05$ even with a saliency prior. The comparisons on two different datasets
demonstrate the superiority of our proposed method in terms of semantic
consistency of object parts.

Method | Mean Error
---|---
IMM [13] | 13.31
Lorenzet al. [22] | 13.60
SCOPS [12](w/o saliency) | 56.08
SCOPS [12](with saliency) | 16.05
Ours | 13.13
Table 2: Landmark regression results on unaligned AFLW. All methods are in the
setting of $K=8$.

Ablation Study. To validate the effectiveness of each loss function, we
conduct ablation studies on CelebA. Table 3 shows the contribution of each
loss function by excluding one loss function each time. For a more intuitive
understanding, the visual results of each ablation study are shown in the Fig.
3. The results show that the combination of the reconstruction loss, the
foreground loss, and the background loss can effectively help avoid a
degenerated result. When any constraint is missing, the regression mean error
increases dramatically. The visualization in Fig. 3 can show the impact of
each loss more intuitively. The reconstruction loss $\mathcal{L}_{rec}$ is the
key to close the loop for unsupervised learning and ensure semantic
consistency. Without $\mathcal{L}_{rec}$, the segmentation results are no
better than uniform squares. Without the foreground loss $\mathcal{L}_{fg}$,
the parts learned are scattered all over an image. If we remove the background
loss $\mathcal{L}_{bg}$, the result leads to another trivial solution, with
one part covering the whole object and others covering the background. The
classification loss $\mathcal{L}_{cls}$ helps further improve the
distinctiveness of the learned part features and enforce cross-instance
semantic consistency. After adding $\mathcal{L}_{cls}$, the mean error is
further reduced from $14.38$ to $12.26$ as in Table 3. Without
$\mathcal{L}_{cls}$, the part learned may cover some semantically unrelated
regions. For example, the facial area contains some pixels from neck and
clothes.

As we discussed in Sec. 3, the whole framework is based on an encoder-decoder
structure, with the reconstruction loss to setup unsupervised learning tasks.
The reconstruction loss helps ensure both the semantic constraint and the
geometry constraint. The squeeze-and-expand bottleneck block makes it possible
to further add the classification loss for improving the semantic consistency
and add the foreground and background losses for satisfying the geometry
constraint. The ablation studies demonstrate that all the four losses are
essential and effective for good segmentation results.

Method | K=4 | K=8
---|---|---
w/o $\mathcal{L}_{rec}$ | 49.32 | 46.01
w/o $\mathcal{L}_{fg}$ | 45.77 | 41.42
w/o $\mathcal{L}_{bg}$ | 31.83 | 29.93
w/o $\mathcal{L}_{cls}$ | 17.20 | 14.38
Full Model | 15.39 | 12.26
Table 3: Ablation study on CelebA. The results shows that all the four losses
are essential for semantically consistent part segmentation.

(a) CUB

(b) PASCAL VOC

Figure 4: Visualization on CUB and PASCAL VOC. Each instance (shown in one
column) has three images: the input image in the first row, the segmentation
map in the last row, and the overlay of segmentation result on the input image
in the second row.

###  4.3 Evaluation on Common Objects

To test the generalization ability of our method, we also evaluate it on other
two datasets: CUB-200-2011 [8], termed as CUB for short, and PASCAL VOC [31],
termed as VOC in the following paragraphs.

CUB [8] is a dataset with $11,788$ photos of $200$ bird species in total. This
dataset is challenging due to the large variation in bird attitudes. In this
experiment, we follow [12] and select the photos from the first $3$ categories
for segmentation. Similar to the previous experiments for human faces, to
quantify the performance of our result, we use the expected coordinates of
each part as a key point position estimation for landmark regression, where we
use the annotated $15$ part locations as the ground truth landmarks. We show
the mean error of landmark regression in Table 4, where CUB-1, CUB-2 and CUB-3
represents the results of three categories of birds. Similarly, our results
are superior to other methods, which indicates that our method keeps a good
semantic consistency for objects with large variations. Visualization of CUB
is shown in Fig. 4a.

Next, we test our method on PASCAL VOC [31] for more object categories. VOC is
a large-scale dataset which has been widely used for object detection and
semantic segmentation tasks. It contains more than $10,000$ images of $20$
categories in total, and about $7,000$ object instances annotated with
segmentation maps. We follow [12] and select seven categories to test our
method. As VOC does not have landmark annotations, we use the ground truth
segmentation masks of object instances and report in Table 5 the
$\operatorname{IOU}$ of our part-aggregated segmentation result compared with
the ground truth mask. The part-aggregated segmentation is simply taken as the
union of all the learned foreground parts. It is worth noting that, as SCOPS
[12] explicitly uses an additional saliency map (or silhouette), this is not a
fair comparison to some extends. Nevertheless, our results still outperforms
previous methods on most categories, and are better than DFF [5] on all
categories. Visualization of VOC is shown in Fig. 4b.

Limitations: Note that while our method performs well in terms of image
transformation and large appearance variance, we find it still struggles with
distinguishing between orientations of near-symmetric objects. Distinguishing
flipped object parts remains a challenging task for self-supervised learning
and deserves dedicated studies in our future work.

Method | CUB-1 | CUB-2 | CUB-3
---|---|---|---
zhang et al. [40] | 30.12 | 29.36 | 28.19
DFF [5] | 22.42 | 21.62 | 21.98
SCOPS [12] | 18.50 | 18.82 | 21.07
Ours | 18.15 | 17.54 | 19.40
Table 4: Landmark regression results on CUB. We report the mean error of
landmark regression, where the horizontal and vertical coordinates are
normalized by the width and height of ground truth bounding box, respectively.

category | sheep | bus | motor | horse | cow | areo | car
---|---|---|---|---|---|---|---
DFF [5] | 51.03 | 58.63 | 54.80 | 49.51 | 56.39 | 48.38 | 56.48
SCOPS [12] | 56.95 | 73.82 | 58.53 | 55.76 | 60.79 | 69.02 | 65.18
Ours | 64.87 | 78.17 | 59.98 | 53.65 | 65.60 | 52.82 | 70.49

Table 5: Evaluation results on PASCAL VOC. We report $\operatorname{IOU}$ of
part-aggregated segmentation compared with the ground truth mask.

###  4.4 Disentanglement on DeepFashion

To illustrate the effect of disentanglement, which is a key component in our
framework, we evaluation our method on DeepFashion [20] dataset, which
contains over $800,000$ diverse images. In our experiment, we select full-body
images of their “in-shop clothes retrieval benchmark” for learning.

We visualize the part segmentation results and appearance transfer results in
Fig. 5. Every generated image is synthesized by the appearance taking from one
image in the leftmost column and the shape taking from one image in the
topmost row. The visualization is mainly to show if the appearance and shape
representations are well disentangled, rather than showing high-quality
synthesis images.

Figure 5: Appearance and shape disentanglement on DeepFashion. Images in the
same column have the same posture, but their appearances come from the
leftmost images.

##  5 Conclusion

In this paper, we have presented an unsupervised method that can learn part
segmentation without supervised annotations by disentangling appearance and
shape. We considered two essential constraints: semantic constraint and
geometry constraint for a good part segmentation result, and accordingly
developed an encoder-decoder neural network with a squeeze-and-expand
bottleneck in the middle for disentangling the appearance and shape
appearances. This design effectively removes the dependency on a saliency map
or object segmentation mask, yet leads to an improved result. Comprehensive
experiments on a wide variety of objects such as human faces, birds, and
PASCAL VOC objects verified the effectiveness of the proposed method in terms
of both segmentation quality and disentanglement result.

##  6 Acknowledgment

This work was supported by the National Key Research and Development Program
of China (No. 2017YFA0700904, No.2020AAA0104304), NSFC Projects (Nos.
61620106010, 62076147, U19A2081, U19B2034, U1811461), Beijing Academy of
Artificial Intelligence (BAAI), Tsinghua-Huawei Joint Research Program, a
grant from Tsinghua Institute for Guo Qiang, Tiangong Institute for
Intelligent Computing, and the NVIDIA NVAIL Program with GPU/DGX Acceleration.

## References

  * [1] Dhruv Batra, Adarsh Kowdle, Devi Parikh, Jiebo Luo, and Tsuhan Chen.  icoseg: Interactive co-segmentation with intelligent scribble guidance.  In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3169–3176, 2010.
  * [2] Yuning Chai, Victor Lempitsky, and Andrew Zisserman.  Bicos: A bi-level co-segmentation method for image classification.  In 2011 International Conference on Computer Vision, pages 2579–2586, 2011.
  * [3] Lisha Chen, Hui Su, and Qiang Ji.  Deep structured prediction for facial landmark detection.  In Advances in Neural Information Processing Systems, pages 2450–2460, 2019.
  * [4] Qifeng Chen and Vladlen Koltun.  Photographic image synthesis with cascaded refinement networks.  In 2017 IEEE International Conference on Computer Vision (ICCV), pages 1520–1529, 2017.
  * [5] Edo Collins, Radhakrishna Achanta, and Sabine Susstrunk.  Deep feature factorization for concept discovery.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 336–352, 2018.
  * [6] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4690–4699, 2019.
  * [7] Jean Duchon.  Splines minimizing rotation-invariant semi-norms in sobolev spaces.  In Constructive theory of functions of several variables, pages 85–100. Springer, 1977.
  * [8] Mark Everingham, Luc Gool, Christopher K. Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  International Journal of Computer Vision, 88(2):303–338, 2010.
  * [9] Yandong Guo and Lei Zhang.  One-shot face recognition by promoting underrepresented classes, 2018\.
  * [10] Junwei Han, Rong Quan, Dingwen Zhang, and Feiping Nie.  Robust object co-segmentation using background prior.  IEEE Transactions on Image Processing, 27(4):1639–1651, 2018.
  * [11] Dorit S. Hochbaum and Vikas Singh.  An efficient algorithm for co-segmentation.  In 2009 IEEE 12th International Conference on Computer Vision, pages 269–276, 2009.
  * [12] Wei-Chih Hung, Varun Jampani, Sifei Liu, Pavlo Molchanov, Ming-Hsuan Yang, and Jan Kautz.  Scops: Self-supervised co-part segmentation.  In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [13] Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi.  Unsupervised learning of object landmarks through conditional image generation.  In NIPS 2018: The 32nd Annual Conference on Neural Information Processing Systems, pages 4016–4027, 2018.
  * [14] Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi.  Self-supervised learning of interpretable keypoints from unlabelled videos.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8787–8797, 2020.
  * [15] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In European Conference on Computer Vision (ECCV), pages 694–711, 2016.
  * [16] Armand Joulin, Francis Bach, and Jean Ponce.  Discriminative clustering for image co-segmentation.  In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 1943–1950, 2010.
  * [17] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [18] Xueting Li, Sifei Liu, Kihwan Kim, Shalini De Mello, Varun Jampani, Ming-Hsuan Yang, and Jan Kautz.  Self-supervised single-view 3d reconstruction via semantic consistency.  arXiv preprint arXiv:2003.06473, 2020.
  * [19] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song.  Sphereface: Deep hypersphere embedding for face recognition.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6738–6746, 2017.
  * [20] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang.  Deepfashion: Powering robust clothes recognition and retrieval with rich annotations.  In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [21] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
  * [22] Dominik Lorenz, Leonard Bereska, Timo Milbich, and Bjorn Ommer.  Unsupervised part-based disentangling of object shape and appearance.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10955–10964, 2019.
  * [23] Peter M. Roth Martin Koestinger, Paul Wohlhart and Horst Bischof.  Annotated Facial Landmarks in the Wild: A Large-scale, Real-world Database for Facial Landmark Localization.  In Proc. First IEEE International Workshop on Benchmarking Facial Image Analysis Technologies, 2011.
  * [24] Fanman Meng, Hongliang Li, Guanghui Liu, and King Ngi Ngan.  Object co-segmentation based on shortest path algorithm and saliency model.  IEEE Transactions on Multimedia, 14(5):1429–1441, 2012.
  * [25] Rajeev Ranjan, Vishal M. Patel, and Rama Chellappa.  Hyperface: A deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(1):121–135, 2019.
  * [26] Oana Sidi, Oliver van Kaick, Yanir Kleiman, Hao Zhang, and Daniel Cohen-Or.  Unsupervised co-segmentation of a set of shapes via descriptor-space spectral clustering.  In Proceedings of the 2011 SIGGRAPH Asia Conference on, volume 30, page 126, 2011.
  * [27] J. Thewlis, S. Albanie, H. Bilen, and A. Vedaldi.  Unsupervised learning of landmarks by exchanging descriptor vectors.  In International Conference on Computer Vision.
  * [28] James Thewlis, Hakan Bilen, and Andrea Vedaldi.  Unsupervised learning of object frames by dense equivariant image labelling.  In NIPS’17 Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 844–855, 2017.
  * [29] James Thewlis, Hakan Bilen, and Andrea Vedaldi.  Unsupervised learning of object landmarks by factorized spatial embeddings.  In 2017 IEEE International Conference on Computer Vision (ICCV), pages 3229–3238, 2017.
  * [30] Nikolai Ufer and Bjorn Ommer.  Deep semantic feature matching.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5929–5938, 2017.
  * [31] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.  The caltech-ucsd birds-200-2011 dataset.  2011\.
  * [32] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  The Caltech-UCSD Birds-200-2011 Dataset.  Technical Report CNS-TR-2011-001, California Institute of Technology, 2011\.
  * [33] Grace Wahba.  Spline models for observational data.  SIAM, 1990.
  * [34] Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu.  Cosface: Large margin cosine loss for deep face recognition.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5265–5274, 2018.
  * [35] Rongsheng Wang, Jinzhao Tian, and Changlong Jin.  Joint face detection and alignment using focal loss-based multi-task convolutional neural networks.  In Chinese Conference on Biometric Recognition, pages 266–273, 2019\.
  * [36] Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao.  A discriminative feature learning approach for deep face recognition.  In European Conference on Computer Vision, pages 499–515, 2016\.
  * [37] Shengtao Xiao, Jiashi Feng, Luoqi Liu, Xuecheng Nie, Wei Wang, Shuicheng Yan, and Ashraf Kassim.  Recurrent 3d-2d dual learning for large-pose facial landmark detection.  In 2017 IEEE International Conference on Computer Vision (ICCV), pages 1642–1651, 2017.
  * [38] Yinghao Xu, Ceyuan Yang, Ziwei Liu, Bo Dai, and Bolei Zhou.  Unsupervised landmark learning from unpaired data.  arXiv preprint arXiv:2007.01053, 2020.
  * [39] Xiang Yu, Feng Zhou, and Manmohan Chandraker.  Deep deformation network for object landmark localization, 2016.
  * [40] Yuting Zhang, Yijie Guo, Yixin Jin, Yijun Luo, Zhiyuan He, and Honglak Lee.  Unsupervised discovery of object landmarks as structural representations.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2694–2703, 2018.
  * [41] Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang.  Facial landmark detection by deep multi-task learning.  In European Conference on Computer Vision, pages 94–108, 2014.
  * [42] Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang.  Learning deep representation for face alignment with auxiliary attributes.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(5):918–930, 2016.
  * [43] Shizhan Zhu, Cheng Li, Chen Change Loy, and Xiaoou Tang.  Face alignment by coarse-to-fine shape searching.  In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4998–5006, 2015.