# A Synopsis of Stent Graft Technology Development ††thanks: Citation:
Authors. Title. Pages…. DOI:000000/11111.

Umme Hafsa Momy
Florida International University
Miami
<EMAIL_ADDRESS>
###### Abstract

Coronary artery disease (CAD) is the predominant cause of mortality and
morbidity across the globe. Over the past few decades, treatments for CAD have
witnessed dramatic evolution, with percutaneous coronary intervention (PCI)
with stenting taking precedence over bypass surgery as the primary
revascularization strategy. This paper delivers an extensive overview of the
significant progress in coronary stent technology, tracing back to the
inaugural coronary angioplasty in 1977. Early trailblazers like Werner
Forssmann, Charles Dotter, and Andreas Gruentzig laid the groundwork for
interventional cardiology. The introduction of bare metal stents (BMS) in the
late 1980s offered solutions to the limitations of balloon angioplasty, such
as acute vessel closure and restenosis. However, BMS had its own set of
challenges. Consequently, the early 2000s saw the emergence of first-
generation drug-eluting stents (DES), utilizing sirolimus and paclitaxel,
offering significant reductions in restenosis compared to BMS. Despite their
success, safety concerns such as very late stent thrombosis arose. Innovations
continued with the second-generation DES, featuring advanced stent platforms
and biocompatible polymers, ensuring enhanced long-term results. The most
recent advancement has been the bioresorbable vascular scaffolds (BVS), which
are designed to resorb over time, eliminating the need for a long-term
metallic implant. Throughout this journey, clinical trials played a pivotal
role in validating the efficacy of each stent generation. While there have
been remarkable improvements in reducing restenosis and other adverse events,
challenges like optimizing regulatory approval pathways, stent selection, and
minimizing risks associated with thrombosis and restenosis persist. The future
holds promise for more individualized stenting strategies, tailored to
specific patient and lesion profiles. This review not only traces the rapid
evolution of coronary stent technology but also underscores its transformative
impact on patient care and outcomes.

##  1 keywords

Coronary artery disease, Percutaneous coronary intervention, Coronary
stenting, Bare metal stents,Angioplasty, Revascularization Drug-eluting
stents, Bioresorbable scaffolds, Restenosis, Stent thrombosis.

##  2 Introduction

Coronary artery disease (CAD) is a leading cause of mortality worldwide,
responsible for over 9 million deaths in 2016 [1]. Percutaneous coronary
intervention (PCI) with stenting has become the most commonly performed
revascularization procedure for obstructive CAD [2]. Since the first coronary
balloon angioplasty was performed in 1977 [3], there has been rapid
development and evolution of stent technology over the past decades. Bare
metal stents (BMS) were introduced in the late 1980s to overcome limitations
of balloon angioplasty like acute vessel closure and restenosis [4, 5]. First-
generation drug-eluting stents (DES) emerged in the early 2000s to further
reduce restenosis rates compared to BMS [6, 7]. However, concerns emerged
regarding very late stent thrombosis with first-generation DES [8]. This
spurred development of second-generation DES with novel stent platforms and
polymers to improve long-term safety and efficacy [9]. Most recently,
bioresorbable vascular scaffolds (BVS) have been introduced as a transient
scaffold to provide short-term vessel support and drug delivery without
leaving a permanent metallic implant [10].

In this paper, we provide a comprehensive overview of the major developments
in coronary stent technology over the past 30 years since the introduction of
BMS. We summarize the evolution of stent designs and materials from early BMS
to modern second-generation DES and BVS. We review clinical data from landmark
trials comparing different stent generations and technologies. Finally, we
discuss future directions for coronary stenting with a focus on optimizing
patient outcomes and minimizing adverse events like restenosis and stent
thrombosis. This synopsis of the rapid growth of stent technology illuminates
how each new stent generation aimed to incrementally improve upon limitations
of its predecessors.

##  3 Method

###  3.1 Werner Forssmann (1904–1979)

Werner Forssmann was the first to venture into the coronary arteries, because
the human heart has always fascinated him. He invented the notion of giving
drugs to the heart with a catheter in 1929, which was a completely unknown
concept. Even though no institution would allow him to attempt such a thing,
he was undeterred and made the decision to proceed anyway without giving much
thought to the likely outcomes. He asked the nurse to numb his own left elbow
while she quickly cut, a urethral catheter was then inserted into his own arm
after the vein was unlocked. After more spins, he had Dozen escort him to the
radiology lab in the basement, where he used the catheter inserted into his
heart to take photographs [11, 12, 13].

Although Forssmann was unable to continue his research on cardiac
catheterization, At Columbia University in New York, Andre F. Cournand and
Dickinson W. Richards took his idea and developed it further decades later, in
the late 1930s. They improved the method and used it to take meaningful
measurements inside the heart. The catheter technique quickly replaced other
techniques as the accepted method of measuring intracardiac pressure. However,
it would be some time before the crown tree and its tangled branches could be
located

###  3.2 F. Mason Sones (1919–1985)

In 1950, F. Mason Sones signed on with the Cleveland Clinic. He was
intelligent, tenacious, and stayed in the hospital for the majority of his
time.

Sones instructed his colleague to inject a dye shot into the aorta to
illuminate it. In his laboratory, Sones placed a diagnostic catheter into the
ascending aorta of a young patient on October 30, 1958.The oxygen-free
angiographic dye was intended to prevent oxygen delivery and cause ventricular
fibrillation, but a small catheter suddenly began to whip around like an
uncontrollable garden hose was used to inject all of the dye deeply into the
patient’s right coronary artery. Nothing like this had ever been tried before.
The patient was unharmed throughout, and the procedure produced a precise
image of the coronary arteries. Sones proclaimed with triumph, He claimed that
they had "just revolutionized cardiology”, after all. Using diagnostic
catheters to create incredibly detailed images, he was able to successfully
place aortic trunk arteries [14].

Sones began developing unique tapered-tip catheters with an open end and a
mesh just beginning to prevent the catheter from clogging the vessel’s shaft
and was inspired by the unexpected incident and outcome [11]. It did not take
long for coronary angiography to become established as a safe and common test
for CAD. A special J-shaped catheter was developed by Melvin P. Judkins, who
modified the method to make catheterization of the coronary arteries easier
and require less effort.However, it requires extensive practice to get a
catheter into the tiny openings of the coronary arteries [15].

###  3.3 Charles T. Dotter (1920–1985)

The Director of radiology at the University of Oregon in Portland is the
brilliant Charles T. Dotter., developed numerous methods for detecting and
treating vascular disease [16, 17].

Sven-Ivar Selinger, known for his method of inserting a catheter inside blood
vessels placing a catheter inside blood vessels— "Catheter over the wire,
needle in, wire in, needle out" had spent some time with him.10 He conducted
research using various materials, including Piano wires, guitar strings, and
other cables and he produced personal catheters because he believed that
catheter technology could be used for more than just diagnosing problems.

Dotter identified a chance and provided her with a novel procedure never done
before. 83 years of age diabetic lady with a non-healing toe with gangrene and
a foot sore underwent an arteriogram of her left leg on January 16, 1964. Her
surgeons had insisted on amputating the foot, believing she was beyond help
with her poor circulation. but the lady vehemently declined. He first passed a
guidewire through the plaque obstruction before inserting a small-caliber
catheter and then wedged the plaque by inserting larger and larger catheters
through it. In an instant, the lady’s frozen leg warmed up and became
hyperemic, seemingly by magic. The woman’s ulcer was healed after a few weeks,
and the pain was subsided, according to X-rays that showed improved
circulation [18, 19].

###  3.4 Andrew

In 1969, Andreas Gruentzig joined University Hospital of Zurich after
completing his medical undergraduate studies in Heidelberg. Gruentzig was
fascinated by the "Doddering" technique and had learned it from a lecture, but
he also believed it needed improvement because there was a high potential for
vascular harm, plaque falling off and acute distal occlusion due to
embolization.

Gruentzig developed the concept of opening a blood vessel with a balloon
attached to the catheter’s tip. in his wild and revolutionary imagination
Except for Michaela and Maria Schlumpf, no one else expressed much interest in
or support for his idea in 1972 because it was so outlandish. Since they
lacked a laboratory and his research funding, using his kitchen as a a
workspace for the following two years, working on his catheters almost every
evening with his wife and Schlumpf.

It was difficult to get the catheter’s tip fitted with a balloon so that he
could blow it up to open a locked container. He could blow it up to unlock a
locked container was no easy task, and they encountered many technology
issues, including air leaks and numerous balloons that expand asymmetrically
or lose their structural integrity. However, he persisted and adjusted. As a
result, he experimented with different materials, shapes, and designs, but
after hundreds of failures and successes, he began to see a few encouraging
outcomes.4

He experimented with his design on diseased arteries taken from cadavers and
animal models, and as his methods for making balloons got better, he felt
ready to use it on a patient.

It was February 1974, Gruentzig pushed his catheter and inflated the balloon
in a man in his 67s whose lower extremity pain rendered him unable. As a
result, patient’s pain resolved immediately. After that, he moved his next
challenge. While awaiting a chance to use his catheters in a human body, he
met with numerous production companies, engineers and the layout of his device
was constantly being improved.

##  4  A Simple Angioplasty is Insufficient

On September 16, 1977, Bachmann didn’t show any signs of pain The artery was
visible as being open on radiographic images. A significant accomplishment in
medicine, it was a big success. In addition to other cases, Gruentzig was
successful in reproducing the outcomes, and he made the news. On February 7,
1978, the front page of a Swiss newspaper carried the headline "Medical
Sensation: Balloon Treatment Against Heart Attacks [11].

Figure 1: Pathophysiology of In-Stent Restenosis and Thrombosis.

Geoffrey O. Hartzler was one of the many followers Gruentzig had trained over
the years [20].He shocked the medical community in 1980 by using angioplasty
to try to destroy a myocardial infarction (MI) as it developed, which kind of
signaled a fresh approach for the procedure. In addition, Hartzler started
modifying the stiff end of the catheter to have better curves so that it would
slide into place more easily as he grew impatient with Gruentzig’ s
invention’s flaws.

Hartzler has pushed the boundaries further, making the bold claim that failure
of angioplasty is the only cause for bypass surgery Despite the fact that
claim merely partially accurate, initial studies contrasting using surgical
techniques for angioplasty, for instance the Coronary Angioplasty vs Bypass
Revascularization Investigation (CABRI) [21], the Bypass Angioplasty
Revascularization Investigation (BARI) [22],Emory Angioplasty vs Surgery Trial
(EAST) [22], the German Angioplasty Bypass Surgery Investigation (GABI) [22],
the Randomized intervention Treatment of Angina (RITA) [22], showed this in
selected patients.

By extruding plaque, simple balloon angioplasty can temporarily increase lumen
diameter, but elastic recoil quickly eliminates this gain. Plaque dissection
can produce plastic, more permanent changes, but there is a possibility of
acute vascular occlusion with this method [4].

The inventors of coronary angioplasty were forced to perform these procedures
in an active surgical standby mode due to an abrupt occlusion. When balloon-
induced intimal denudation and medial tearing occur, the subendothelial matrix
is exposed to the blood, which promotes platelet aggregation and thrombosis in
the acute phase and chronic negative changes in vascular remodeling (late
recoil) and neointimal hyperplasia in the chronic phase. In the first 6 to 9
months, 30 percent to 40 percent of patients experienced an almost total loss
of therapeutic benefit due to insufficient initial reinforcement and
restenosis (Fig. 1) [5].

##  5 Key Preclinical Studies

Preclinical studies in animal models like porcine coronary arteries provided
key insights into optimal stent design features. For example, Saxon et al.
compared different stent materials and configurations in a pig model. They
found tantalum wire stents achieved a larger acute lumen than stainless steel,
with less thrombosis risk [23]. Schwarzacher et al. tested antithrombotic
stent coatings like carbon and heparin in a sheep model, finding reduced acute
thrombosis compared to uncoated stents [24]. These studies guided early stent
material and coating selections prior to clinical use. Later preclinical work
focused on evaluating safety of stent polymer coatings and kinetic drug
release profiles for developing drug-eluting stents [25].

##  6 Stent Development

In 1986, Ulrich Siegwart of Switzerland and Jacques Puel of France implanted
the first stents[26]. Coil stents and slotted tube designs were first
implanted after these self-expanding mesh designs in 1987 at Emory University
Hospital and So Paulo, Brazil, respectively. With the spread worldwide
Angioplasty procedures revealed that the arteries of numerous patients
gradually narrowed hours, days, or even months following the operation. As
techniques for angioplasty spread across the globe, it became apparent that
many patients’ arteries gradually narrowed weeks or months after the
procedure[27, 28, 29]. Early research revealed rates of restenosis ranging
between twelve and forty-eight percent [30].Stents were created that would
keep vessels conserve after angioplasty to address these problems. Studies
have exhibited improved fast outcomes and longer period without events
stenting associated to standard inflatable angioplasty. Restenosis levels were
declined by about 10 percent overall [5, 4].

##  7 Classification of Stent

The most significant development within transcutaneous coronary
revascularization is the development of stents. Restenosis within a stent. is
no longer a significant issue with coronary intervention to make stents that
can be infused with drugs. Metal stents covered with a polytetrafluoroethylene
(PTFE) membrane, known as covered stent-grafts, were developed to prevent
restenosis caused by the growth of tissue through the mesh of a stent. An
intriguing idea for preventing intraluminal proliferation, sealing degenerated
vein grafts and covering coronary artery perforations is the use of stent
grafts, which integrate a membrane into a coronary stent.

After balloon dilatation, arterial recoil and restenosis should be avoided by
using coronary stents. Bare metal stents (BMS), drug eluting stents (DES), and
bioresorbable vascular scaffolds (BRS) are the three major types of stents.

###  7.1 Bare metal stents (BMS)

Bare metal stents (BMS) were the first stents used. These stents can be made
thinner, have great mechanical strength and poor flexibility. However, bare
metal stents can cause restenosis and can lead to peripheral embolism after
being implanted in old vein grafts. A lower restenosis rate, as confirmed by
two historical trials issued in 1993 [4, 5].Observational studies indicate
that the rate of cardiovascular events is increased when stent grafts are used
voluntarily in native vessels. However, the strong mechanical support also
contributes to neo-intimal hyperplasia. Intravascular ultrasound studies
showed that stents required high pressures to fully expand, leading to the
development of Dual antiplatelet therapy (DAPT) combining ticlopidine,
clopidogrel, and aspirin. In-stent restenosis (ISR), observed in mid- and
long-term follow-up, 15 percent to 30 percent of treated lesions was still
significantly risky with these stents [31].

###  7.2 Drug-Eluting Stents (DES)

Drug-Eluting Stents (DES) the next creation of stents. The drug eluted was an
antimitotic agent that prevented the growth of SMCs. In the history of
interventional cardiology, there has been a third revolutionary paradigm
shift. was signaled during 1999 when the first DES was implanted in Brazil by
Sousa. However, the possibility of late stent thrombosis (ST) is enhanced by
impaired endothelial regeneration and vasomotion.

####  7.2.1 First-generation Drug-eluting Stents

First-generation DESs originally used two antiproliferative medications were
sirolimus and paclitaxel. When Cementin published a meta-analysis in 2006, It
was discovered that myocardial infarction (MI) and mortality risk were both
increased by stent thrombosis (ST) that developed extremely late or tardy
[32]. Several randomized controlled trials (RCTs) have been conducted to
evaluate both. and demonstrated considerable reductions in ISR, target lesion
and late lumen loss /vascular revascularization rates evaluated to BMS. Each
was constructed from stainless steel and was thick in the struts of greater
than 130m [6, 33, 34]. A very late ST, while currently familiar as a potential
first-generation Drug-eluting Stents (DES) complication, occurs rarely and a
lot of data registries and meta-analyses consume offered comfort regarding
using such methods in practice [35]

####  7.2.2 Second-generation Drug-eluting Stents

With the switch to metal alloys for the platform in the second-generation DES,
the struts could be thinner and more flexible. The Lemus family of drugs, such
as zotarolimus, enviroximes and novelist, which exhibit faster drug release
and consequently earlier endothelial coverage, have been used to create new,
more biocompatible polymer.

###  7.3 BRS stands

for the bioresorbable stent. Resorbing over a period of 6 months to 2 years,
these stents reduce chronic inflammation over a longer period of time and
promote endothelial regeneration. Reproduced with permission [28].

##  8 Complications of stenting

Stenting complications are comparatively rare. Stents have not been associated
with many complications, but there is a small chance that the body will reject
the stent. Discomfort and bleeding at the puncture site where the catheter was
inserted are the most common side effects.

Some people have metal allergies or sensitivities, and stents contain metal
components. Stent manufacturers do not recommend using them on people
sensitive to metal. Dissection of an inner layer in the coronary artery is a
tiny tear that occasionally results from the procedure. The tear usually heals
on its own and is not too large.

In certain circumstances, a stent is used to repair the tear. Immediate
treatment is given if the tear is severe and causes blockage of arterial blood
flow or bleeding out around the heart.

##  9 Limitations of stenting

In over 90 percent of patients, stenting improves blood flow and relieves
symptoms, but there is a chance that symptoms will return within six months.
Symptomatic restenosis can occur in the following cases: 1. About 30 percent
of patients who have surgery to open a blocked artery do not receive stents.
2. Approximately 15 percent of patients with bare metal stents. 3. Less than
10 percent of patients using drug eluting stents, also known as drug eluting
stents. In addition, some medical conditions, such as diabetes and continued
smoking, can increase the risk of narrowing. diffuse narrowed arteries, Low-
density lipoprotein (LDL) cholesterol and high blood pressure. A large blood
vessel located at or near the beginning of a side branch narrows a blood
vessel with numerous implanted stents.

##  10 Regulatory Approval Pathways

The 1994 FDA approval of Palmaz-Schatz stents for coronary use marked an
important milestone [36]. This first bare metal stent underwent prospective
clinical trials to meet regulatory requirements and demonstrate safety and
efficacy. Subsequent stent approvals built on this pathway, with new
generations of stents requiring comparably designed trials. First-generation
drug-eluting stents gained FDA approval in 2003 to 2004 based on trials like
SIRIUS [6]. Approval of these novel, higher-risk devices spurred more rigorous
post-marketing surveillance mandates. Costly and lengthy regulatory processes
posed challenges, limiting the pace of incremental innovation. Efforts to
balance safety with faster access continue to evolve.

##  11 Care after the procedure

In over 90 percent of patients, stenting improves blood flow and relieves
symptoms, but there is a chance that symptoms will return within six months.

Symptomatic restenosis can occur in the following cases:

1\. About 30 percent of patients who undergo surgery to open a blocked artery
do not receive stents.

2\. Approximately 15 percent of patients with bare metal stents.

3\. Less than 10 percent of patients who use drug-eluting stents, also known
as drug-coated stents.

Compared to other coronary artery sites, some are more likely to re-narrow.
Additionally, some medical conditions, for instance diabetes and continued
smoking, might enhance the possibility of narrowing. diffusely narrowed
arteries, high blood pressure, terrible cholesterol (LDL) levels that are too
high.

A major blood vessel that is at the start of a side branch or close to it
narrows, a blood vessel with numerous stents implanted.

##  12 Preventing blood clots

The formation of a blood clot (thrombosis) inside the stent, also known as
stent thrombosis, is one of the most serious complications that can occur
after the insertion of a stent. Fortunately, stent thrombosis due to
administration of aspirin and other anticoagulant drugs both before and after
stenting is rare. It is believed that clotting occurs when the metal of the
stent comes into contact with blood components. Stent thrombosis, which cuts
off the blood supply to the heart, can lead to a heart attack or even death.
Even though greatest occurrences happen in the interior thirty days of stent
placement, Stent thrombosis may happen as early as twenty-four hours, thirty
days, or even up to a year later.

##  13 When to seek help

If any of the following events occur after stenting, seek medical help
immediately:

• Fever greater than 38°C (100.4 F).

• You faint or feel dizzy.

• Your pulse is not normal Start of chest pain.

• The puncture site becomes extremely painful, swollen, warm, bleeds more than
a few drops, or discharges pus.

##  14 Conclusion

The record and development stenting of coronary arteries is amongst the
greatest astonishing characteristics of innovative practice of medicine.
Patients requiring coronary angioplasty typically receive treatment with
coronary artery stenting. Stents have eliminated the mechanical component of
restenosis and acute recoil, eliminating the need for urgent bypass surgery.
Even with BMS, ISR has declined due to significant advances in stent platform
design. As a result of numerous efficacy and safety studies, Stents for
coronary arteries are currently the preferred medication for CAD.
Nevertheless, there is still thrombosis and stent restenosis a major a
significant competition for modern coronary artery stents. Future
interventional cardiologists may be able to use individualized, evidence-based
medicine, in which the selection of a Based on stent on a patient’s inherent
factors, Lesion characteristics and risk profile (for thrombosis, restenosis,
and bleeding).

## References

  * [1] Mohsen Naghavi, Amanuel Alemu Abajobir, Cristiana Abbafati, Kaja M Abbas, Foad Abd-Allah, Semaw Ferede Abera, Victor Aboyans, Olatunji Adetokunboh, Ashkan Afshin, Anurag Agrawal, et al.  Global, regional, and national age-sex specific mortality for 264 causes of death, 1980–2016: a systematic analysis for the global burden of disease study 2016.  The lancet, 390(10100):1151–1210, 2017.
  * [2] Gregory J Dehmer, Douglas Weaver, Matthew T Roe, Sarah Milford-Beland, Susan Fitzgerald, Anthony Hermann, John Messenger, Issam Moussa, Kirk Garratt, John Rumsfeld, et al.  A contemporary view of diagnostic cardiac catheterization and percutaneous coronary intervention in the united states: a report from the cathpci registry of the national cardiovascular data registry, 2010 through june 2011.  Journal of the American College of Cardiology, 60(20):2017–2031, 2012.
  * [3] Andreas R Grüntzig, Åke Senning, and Walter E Siegenthaler.  Nonoperative dilatation of coronary-artery stenosis: percutaneous transluminal coronary angioplasty.  New England Journal of Medicine, 301(2):61–68, 1979.
  * [4] Patrick W Serruys, Peter De Jaegere, Ferdinand Kiemeneij, Carlos Macaya, Wolfgang Rutsch, Guy Heyndrickx, Hakan Emanuelsson, Jean Marco, Victor Legrand, Pierre Materne, et al.  A comparison of balloon-expandable-stent implantation with balloon angioplasty in patients with coronary artery disease.  New England Journal of Medicine, 331(8):489–495, 1994.
  * [5] David L Fischman, Martin B Leon, Donald S Baim, Richard A Schatz, Michael P Savage, Ian Penn, Katherine Detre, Lisa Veltri, Donald Ricci, Masakiyo Nobuyoshi, et al.  A randomized comparison of coronary-stent placement and balloon angioplasty in the treatment of coronary artery disease.  New England Journal of Medicine, 331(8):496–501, 1994.
  * [6] Jeffrey W Moses, Martin B Leon, Jeffrey J Popma, Peter J Fitzgerald, David R Holmes, Charles O’Shaughnessy, Ronald P Caputo, Dean J Kereiakes, David O Williams, Paul S Teirstein, et al.  Sirolimus-eluting stents versus standard stents in patients with stenosis in a native coronary artery.  New England Journal of Medicine, 349(14):1315–1323, 2003.
  * [7] Gregg W Stone, Stephen G Ellis, David A Cox, James Hermiller, Charles O’Shaughnessy, James Tift Mann, Mark Turco, Ronald Caputo, Patrick Bergin, Joel Greenberg, et al.  A polymer-based, paclitaxel-eluting stent in patients with coronary artery disease.  New England Journal of Medicine, 350(3):221–231, 2004.
  * [8] Edoardo Camenzind, P Gabriel Steg, and William Wijns.  Response to letter regarding article,“stent thrombosis late after implantation of first-generation drug-eluting stents: A cause for concern”.  Circulation, 116(16):e390–e390, 2007.
  * [9] Giulio G Stefanini, Bindu Kalesan, Patrick W Serruys, Dik Heg, Pawel Buszman, Axel Linke, Thomas Ischinger, Volker Klauss, Franz Eberli, William Wijns, et al.  Long-term clinical outcomes of biodegradable polymer biolimus-eluting stents versus durable polymer sirolimus-eluting stents in patients with coronary artery disease (leaders): 4 year follow-up of a randomised non-inferiority trial.  The Lancet, 378(9807):1940–1948, 2011.
  * [10] John A Ormiston, Patrick W Serruys, Evelyn Regar, Dariusz Dudek, Leif Thuesen, Mark WI Webster, Yoshinobu Onuma, Hector M Garcia-Garcia, Robert McGreevy, and Susan Veldhof.  A bioabsorbable everolimus-eluting coronary stent system for patients with single de-novo coronary artery lesions (absorb): a prospective open-label trial.  The Lancet, 371(9616):899–907, 2008.
  * [11] David Monagan.  Journey into the heart: a tale of pioneering doctors and their race to transform cardiovascular medicine.  Penguin, 2007.
  * [12] Renate Forssmann-Falck.  Werner forssmann: a pioneer of cardiology.  The American journal of cardiology, 79(5):651–660, 1997.
  * [13] Spencer B King.  The development of interventional cardiology.  Journal of the American College of Cardiology, 31(4S2):64B–88B, 1998.
  * [14] Nicholls M Werner Forssmann Nobel Prize.  for physiology or medicine 1956.  Eur Heart J [Internet], 41(9):980–982, 2020.
  * [15] Michael J Cowley.  Tribute to a legend in invasive/interventional cardiology melvin p. judkins, md (1922–85).  Catheterization and Cardiovascular Interventions, 64(2):259–261, 2005.
  * [16] Thomas B Kinney.  Radiologic history exhibit. charles t. dotter: a pioneering interventional radiologist.  Radiographics, 16(3):697–707, 1996.
  * [17] T Doby.  A tribute to sven-ivar seldinger.  American Journal of Roentgenology, 142(1):1–4, 1984.
  * [18] Misty M Payne.  Charles theodore dotter: the father of intervention.  Texas Heart Institute Journal, 28(1):28, 2001.
  * [19] Radiopaedia.org.  Charles t. dotter, 2023.  Available from: https://radiopaedia.org/articles/charles-t-dotter?lang=us.
  * [20] Joel K Kahn.  Geoffrey o. hartzler.  Clinical Cardiology, 27(1):58–59, 2004.
  * [21] CABRI Trial Participants.  First-year results of cabri (coronary angioplasty versus bypass revascularisation investigation).  The Lancet, 346(8984):1179–1184, 1995.
  * [22] Bypass Angioplasty Revascularization Investigation (BARI) Investigators.  Comparison of coronary bypass surgery with angioplasty in patients with multivessel disease.  New England Journal of Medicine, 335(4):217–225, 1996.
  * [23] RR Saxon, A Chervu, PA Jones, TK Bajwa, DL Gable, ND Souza, WM Bogey, JD Babb, M Echols, and SK Williams.  Radiographic, angiographic, and intravascular ultrasound outcomes with the tacrolimus-eluting stent.  J Interv Cardiol, 16(1):45–52, Feb 2003.
  * [24] Ali Moustapha, Joseph Salloum, Sangeeta Saikia, Hany Awadallah, Mohammed Ghani, Stefano Sdringola, Georges Schroth, Abid Assali, Richard W Smalling, HV Anderson, et al.  Combined cutting balloon angioplasty and intracoronary beta radiation for treatment of in-stent restenosis: Clinical outcomes and effect of pullback radiation for long lesions.  Catheterization and cardiovascular interventions, 57(3):325–329, 2002.
  * [25] Giuseppe De Luca, Arnoud WJ Van’t Hof, Menko-Jan de Boer, Jan Paul Ottervanger, Jan CA Hoorntje, AT Marcel Gosselink, Jan-Henk E Dambrink, Felix Zijlstra, and Harry Suryapranata.  Time-to-treatment significantly affects the extent of st-segment resolution and myocardial blush in patients with acute myocardial infarction treated by primary angioplasty.  European Heart Journal, 25(12):1009–1013, 2004.
  * [26] Ulrich Sigwart, Jacques Puel, Velimir Mirkovitch, Francis Joffre, and Lukas Kappenberger.  Intravascular stents to prevent occlusion and re-stenosis after transluminal angioplasty.  New England Journal of Medicine, 316(12):701–706, 1987.
  * [27] Schächinger Volker and Andreas M Zeiher.  Covered stent grafts: role in intervention of coronary arteries and degenerated vein grafts.  Clinical Research in Cardiology, 91:58–63, 2002.
  * [28] Jiabin Zong, Quanwei He, Yuxiao Liu, Min Qiu, Jiehong Wu, and Bo Hu.  Advances in the development of biodegradable coronary stents: A translational perspective.  Materials Today Bio, page 100368, 2022.
  * [29] Amgad Albohy.  https://www. sciencedirect. com/science/article/abs/pii/s0378874123004993? via% 3dihub.  2023\.
  * [30] PWea Serruys, HE Luijten, KJ Beatt, R Geuskens, PJ De Feyter, Marcel van den Brand, JH Reiber, HJ Ten Katen, Gerrit Anne van Es, and PG Hugenholtz.  Incidence of restenosis after successful coronary angioplasty: a time-related phenomenon. a quantitative angiographic study in 342 consecutive patients at 1, 2, 3, and 4 months.  Circulation, 77(2):361–371, 1988.
  * [31] Rainer Hoffmann, Gary S Mintz, Gaston R Dussaillant, Jeffrey J Popma, Augusto D Pichard, Lowell F Satler, Kenneth M Kent, Jennifer Griffin, and Martin B Leon.  Patterns and mechanisms of in-stent restenosis: a serial intravascular ultrasound study.  Circulation, 94(6):1247–1254, 1996.
  * [32] Edoardo Camenzind, P Gabriel Steg, and William Wijns.  A cause for concern.  Circulation, 115(11):1440–1455, 2007.
  * [33] Carlo Briguori, Flavio Airoldi, Gabriella Visconti, Amelia Focaccio, Gianluca Caiazzo, Bruno Golia, Giuseppe Biondi-Zoccai, Bruno Ricciardelli, and Gerolama Condorelli.  Novel approaches for preventing or limiting events in diabetic patients (naples-diabetes) trial: a randomized comparison of 3 drug-eluting stents in diabetic patients.  Circulation: Cardiovascular Interventions, 4(2):121–129, 2011.
  * [34] Christoph Stettler, Simon Wandel, Sabin Allemann, Adnan Kastrati, Marie Claude Morice, Albert Schömig, Matthias E Pfisterer, Gregg W Stone, Martin B Leon, José Suarez de Lezo, et al.  Outcomes associated with drug-eluting and bare-metal stents: a collaborative network meta-analysis.  The Lancet, 370(9591):937–948, 2007.
  * [35] Pamela S Douglas, J Matthew Brennan, Kevin J Anstrom, Art Sedrakyan, Eric L Eisenstein, Ghazala Haque, David Dai, David F Kong, Bradley Hammill, Lesley Curtis, et al.  Clinical effectiveness of coronary stents in elderly persons: results from 262,700 medicare patients in the american college of cardiology–national cardiovascular data registry.  Journal of the American College of Cardiology, 53(18):1629–1641, 2009.
  * [36] Food and Drug Administration.  Fda approves first coronary stent.  FDA Talk Paper T94-12, Mar 1994.


2em1.6em

# CCEdit: Creative and Controllable Video Editing via Diffusion Models

Ruoyu Feng1,2 *, Wenming Weng1,2, Yanhui Wang1,2,
Yuhui Yuan2, Jianmin Bao2, Chong Luo2 †, Zhibo Chen1 †, Baining Guo2
1University of Science and Technology of China 2Microsoft Research Asia

###### Abstract

In this work, we present CCEdit, a versatile framework designed to address the
challenges of creative and controllable video editing. CCEdit accommodates a
wide spectrum of user editing requirements and enables enhanced creative
control through an innovative approach that decouples video structure and
appearance. We leverage the foundational ControlNet architecture to preserve
structural integrity, while seamlessly integrating adaptable temporal modules
compatible with state-of-the-art personalization techniques for text-to-image
generation, such as DreamBooth and LoRA. Furthermore, we introduce reference-
conditioned video editing, empowering users to exercise precise creative
control over video editing through the more manageable process of editing key
frames. Our extensive experimental evaluations confirm the exceptional
functionality and editing capabilities of the proposed CCEdit framework. Demo
video is available at https://www.youtube.com/watch?v=UQw4jq-igN4.

Figure 1:  The CCEdit framework accommodates a wide spectrum of user editing
demands: the first four rows showcase global style translation driven by
various intentions, the fifth row illustrates foreground editing, and the
sixth row demonstrates background modification.

††footnotetext: ∗ This work is done when Ruoyu Feng is an intern with MSRA.
† Corresponding author.

††footnotetext: 1 CCEdit is currently a research project, and there are no
immediate intentions to integrate it into a product or extend public
accessibility. Any future research endeavor will adhere to Microsoft’s AI
principles.

##  1 Introduction

In recent years, the domain of visual content creation and editing has
undergone a profound transformation, driven by the emergence of diffusion-
based generative models [13, 34, 8]. A large body of prior research has
demonstrated the exceptional capabilities of diffusion models in generating
diverse and high-quality images [27, 23, 30] and videos [12, 33, 3],
conditioned by text prompts. These advancements have naturally paved the way
for innovations in generative video editing [40, 4, 38, 20, 41, 44, 22].

Generative video editing, although in its nascent stages, faces a series of
significant challenges. These challenges include accommodating diverse editing
requests, achieving fine-grained control over the editing process, and
harnessing the creative potential of generative models. Diverse editing
requirements encompass tasks such as stylistic alterations, foreground
replacements, and background modifications. Generative algorithms, while
powerful, may not always align perfectly with the editor’s intentions or
artistic vision, resulting in a lack of precise control. Balancing creativity
and controllability is crucial for a successful generative video editing
system.

In response to these challenges, this paper introduces CCEdit1, an innovative
and comprehensive framework meticulously designed to strike a harmonious
balance between controllability and creativity while accommodating a wide
range of editing requirements. CCEdit unleashes the potential of video editing
by decoupling the control of video structure and appearance during the editing
process.

Specifically, we leverage both text prompt and personalized text-to-image
(T2I) diffusion models [27] for appearance control. Building upon these pre-
trained weights, we employ ControlNet [42] to inherit structural information
from the source video. Users can choose from a variety of structural
information types, such as line drawings[5], PIDI boundaries[35], and depth
maps. For each type of structure information, we train distinct temporal
layers within the main diffusion network while keeping the pre-trained T2I
model unaltered. These temporal layers seamlessly integrate structural control
into the generation network while maintaining temporal consistency across
edited video frames. This approach allows the temporal modules to function as
adaptable components, accommodating various personalized model versions [29,
14] derived from the same text-to-image model, thereby enabling users to
control appearance by selecting their desired image model.

However, there are occasions when users desire finer-grained control over the
appearance of the target video, considering more nuanced aspects. Furthermore,
we empirically observed compatibility issues between the personalized T2I
models and the newly trained temporal modules, especially with certain stylish
LoRA models. In such cases, the video editing output may not perfectly
maintain the style of the adopted image model. To address this, our framework
introduces reference-aware video editing, an elegant mechanism that enables
users to manually or AI-assistedly edit a keyframe and propagate those changes
to other frames in the same video. Technically, we introduce an additional
model to extract features from the edited reference image which are then
incorporated into the features of the center frame extracted by the main
network. The temporal modules propagate this newly introduced information
throughout the entire video sequence, ensuring the generation of a coherent
video.

To complete the system, we incorporate a frame interpolation method for
editing high-frame-rate videos. Qualitative results demonstrate that CCEdit
strikes a delicate balance between artistic creativity and operational
controllability, empowering editors to tailor the editing process precisely to
their unique requirements. The framework’s compatibility with a diverse
spectrum of generative models and the reference-aware approach underscore its
adaptability and efficiency, heralding a future of enhanced generative video
editing experiences.

##  2 Related Work

###  2.1 Diffusion-based Image and Video Generation

Diffusion models (DM)[13, 34, 8] have demonstrated exceptional capabilities in
the field of image synthesis. These models indeed help by learning to
approximate a data distribution through the iterative denoising of a diffused
input. What makes DMs truly practical is the incorporation of text prompt as
condition to control the output image during the generative process[24, 30,
19, 27]. Apart from the proliferation of advanced techniques in the field of
image synthesis, DMs have also excelled in video generation[12, 3, 33]. This
is accomplished by integrating modulated spatial-temporal modules, enabling
the synthesis of high-quality videos while maintaining temporal consistency.

###  2.2 Video Editing with Diffusion Models

Recent studies leverage the inherent generative priors in DMs for image
editing[17, 7, 2, 36, 21, 11]. The same idea is also applied in the field of
video editing. Unlike image editing, video editing involves not only the
manipulation of appearance-based attributes but also requires the meticulous
preservation of temporal coherence throughout the video sequence. A lapse in
maintaining this temporal coherence can result in visual artifacts, such as
flickering and degradation. Some methods[43, 41, 22, 37] attempt to achieve
temporal consistency without training by transitioning from spatial self-
attention mechanisms within text-to-image (T2I) diffusion models to temporal-
aware cross-frame attention techniques. Furthermore, certain methods[39, 32,
16, 44] optimize the parameters of the pre-trained T2I model based on the
input video to attain temporal coherence of the target video. Nonetheless, the
process of optimization for each input video can be time-consuming. Inadequate
tuning of the temporal modules can result in suboptimal temporal coherence.
Recent studies[40, 10, 15] have introduced supplementary trainable temporal
layers to construct text-to-video (T2V) generative models. These models are
trained on extensive text-video paired datasets, and their versatility is
demonstrated in both video generation and video editing tasks. Different from
previous methods, this paper disentangle the video editing process into
distinct components, namely structure control and appearance control.
Structure control aims to maintain the structure information of the input
video, encompassing a wide range types from coarse to fine-grained, such as
line drawings, PIDI boundaries[35], and depth maps. In terms of appearance
control, with the goal of addressing creators’ preferences comprehensively, we
have introduced a spectrum of tools. These include text prompts, personalized
T2I models, and edited reference images, allowing for meticulous control over
the appearance of the output video. Additionally, we develop dedicated
temporal layers to achieve precise alignment and coherence in the temporal
dimension.

###  2.3 Personalized Text-to-Image models

The Stable Diffusion[27] model is trained on a huge dataset that encompasses a
broad spectrum of domains[31]. Although the Stable Diffusion model is highly
versatile and capable of generating a wide array of images, it occasionally
falls short in specific details, particularly when it comes to generating
human faces and hands, where subtle variations can markedly influence the
overall perception. Additionally, it often struggles to precisely meet users’
expectations for specific content, styles, and attributes. Therefore,
personalized T2I models are designed to address these challenges. Two
respective methods are DreamBooth[29] and LoRA[14]. The former uses a unique
string as an indicator to represent the corresponding domain or concept during
training. Once trained, this indicator can be employed to transfer the
expectations to the fine-tuned T2I model. DreamBooth faces challenges due to
the extensive weight parameters, making communication less convenient. To use
less parameters and inherent the generalization of the base model, LoRA is
proposed to fine-tune the model by freezing all original parameters and
introducing the weight residuals $\Delta W$ to update the weights $W$. This
process is formulated as $W^{\prime}=W+\alpha\Delta W$, where $\alpha$ is the
hyperparameter that controls the significance of the added $\Delta W$.
Typically, the parameters of $\Delta W$ are significantly fewer than those of
$W$. Finally, two additional methods for creating robust personalized T2I base
models are fine-tuning the entire model directly on the self-collected
datasets and blending parameters from various models.

Personalized T2I models play a pivotal role in establishing the ecosystem of
contemporary AI content generation. They empower both novices and experienced
artists, as well as enthusiasts, to rapidly and autonomously create stunning
images and develop new models. One of the most significant objectives of our
framework is to ensure compatibility with personalized T2I models, allowing
creators to freely combine and perform highly creative edits on videos using
models from the community. This is primarily attributed to our inheritance of
the original spatial weights from Stable Diffusion model and the additional
introduction of reference-aware editing operations.

Figure 2:  Illustration of our overall framework. Structure and appearance
information in the target video are modulated independently. Structure control
is conducted via the pre-trained ControlNet[42], encompassing a range from
coarse to fine-grained. Appearance control is achieved by the flexible
combination of text prompts, personalized T2I models, and edited center frame
as reference. We omit the autoencoder and iterative denoising process for
simplicity. Best viewed in color.

##  3 Approach

###  3.1 Preliminary

Diffusion Models [13] are probabilistic generative models that approximate a
data distribution $p(\mathbf{x})$ by gradually denoising a normally
distributed variable. Specifically, DMs aim to learn the reverse dynamics of a
predetermined Markov chain with a fixed length of $T$. The forward Markov
chain can be conceptualized as a procedure of injecting noise into a pristine
image, thereby transforming it into a stochastic representation, typically
conforming to a standard Gaussian distribution. Empirically, DMs can be
interpreted as an equally weighted sequence of denoising autoencoders
${\epsilon}_{\theta}(\mathbf{x}_{t},t)$ where $t={1,...,T}$. These
autoencoders are trained to predict a denoised variant of the input
$\mathbf{x}_{t}$. The corresponding objective can be simplified to:

$\mathbb{E}_{\mathbf{x},\epsilon\sim\mathcal{N}(0,1),t}[{\|\epsilon-{\epsilon}_{\theta}(\mathbf{x}_{t},t)\|}_{2}^{2}].$
(1)

Latent Diffusion Models (LDMs) are trained in the learned latent
representation space $\mathbf{z}_{t}$, instead of the pixel space
$\mathbf{x}_{t}$. The bridge between this latent space and the original pixel-
level domain is established via a perceptual compression model. The perceptual
compression model is composed of an encoder $\mathcal{E}$ and a decoder
$\mathcal{D}$, where $\mathbf{z}=\mathcal{E}(\mathbf{x})$ and
$\mathbf{x}\approx\mathcal{D}(\mathcal{E}(\mathbf{x}))$. Then the optimization
objective in Equation (1) is modified as:

$\mathbb{E}_{\mathcal{E}(\mathbf{x}),\epsilon\sim\mathcal{N}(0,1),t}[{\|\epsilon-{\epsilon}_{\theta}(\mathbf{z}_{t},t)\|}_{2}^{2}].$
(2)

###  3.2 Overall Framework

The architecture of our proposed framework is illustrated in Figure 2. Our
objective is to create a comprehensive and user-friendly framework for video
editing. To this end, we decouple the editing process into two distinct
modules: structure control and appearance control.

Specifically, our framework is built on the Stable Diffusion [27] model.
During the training phase, we freeze all weights of the original models to
preserve both its robust visual synthesis and language comprehension
capabilities. Structure control is achieved by incorporating visual priors
such as line drawings[5], PIDI boundaries[35], and depth maps extracted from
the source video. We use ControlNet to extract features and insert them into
the base T2I model. For appearance control, we employ three different
approaches: text prompts, personalized model weights, and customized center
frame as reference. Most importantly, users can flexibly combine these
elements according to their specific requirements to achieve the purpose of
customizing the target video. Finally, the temporal coherence of the generated
video is preserved through our introduced temporal consistency modules. This
ensures a seamless and high-quality visual experience across the video
sequence. Additionally, we also implement the temporal interpolation method in
our framework for high frame rate.

Section 3.3 provides the implementation of structure control. Section 3.4
introduces the appearance control used in our framework. The detailed
implementation of the newly introduced temporal consistency modules is
presented in Section 3.5. Section 3.6 describes the method of temporal
interpolation.

###  3.3 Structure Control

We utilize the pre-trained ControlNet[42] to achieve the inherence of
structure information of the input video. The ControlNet is initialized by
copying the encoder of the T2I model before its training. The features
extracted by ControlNet are integrated into the decoder of the frozen T2I base
model, thereby controlling the structure of the generated image. Besides, the
projection-in and projection-out layers are initialized with zeros before
training to ensure stable and progressive updating. In our framework, we
employ various types of structure information and corresponding ControlNets,
ranging from coarse to fine, to ensure the flexibility of controlling the
structure from varying degrees. To maintain the strong capability of the pre-
trained ControlNet, we freeze the weights of it during training. Specifically,
as illustrated in Figure 2, each frame of the input video undergoes
preprocessing to obtain a structure representation. The representation is then
input into ControlNet, which extracts features for each frame. These features
are subsequently incorporated into the decoder of the main network.

###  3.4 Appearance Control

We propose different approaches to control the appearance of the target video,
from the perspective of coarse-grained to fine-grained. Section 3.4.1
discusses the coarse-grained methods, which involve text prompts and
personalized models. In Section 3.4.2, we delve into the fine-grained
technique, _i.e_., controlling the appearance of the video at the pixel level
using the edited center frame as a reference.

####  3.4.1 Coarse-Grained Appearance Control

Text Prompts. Stable Diffusion [27] is a text-to-image (T2I) model trained on
the large and diverse image-text paired dataset LAION[31], possessing strong
language understanding capabilities. During the training process, we retain
all weights of the original layers to preserve its robust abilities. As a
result, the simplest method of video editing involves directly editing the
text prompts. For instance, in Figure 2, the original text prompt is “a bear
is walking.” The edited text could be “a bear is walking, Van Gogh style.” The
appearance of the target video is controlled by the text prompt, the structure
consistency is maintained by the pre-trained structure ControlNet, and
temporal coherence is preserved by temporal consistency modules.

Personalized Models. Stable Diffusion excels as a general-purpose generative
model for synthesizing realistic images, but it underperforms when tasked with
generating contents within a specific domain or depicting a particular
identity. DreamBooth[29] and LoRA[14] are two representative techniques for
generating custom styles, concepts, identities, _etc_. Recently, the Stable
Diffusion community has seen rapid advancements, with contributors fine-tuning
the T2I model on specialized datasets and sharing the pre-trained weights
online[6, 9]. Users can readily substitute the original T2I model’s weights
with those personalized ones to generate images tailored to specific domains.
As long as the personalized models and LoRAs are trained from the same base
model (_e.g_., Stable Diffusion-v1.5), they can exhibit good compatibility
with each other. Different LoRAs can function like plug-ins on various base
models. In our framework, as shown in Figure 2, the flexibility to change the
weights of the T2I base model and LoRAs serves as another avenue for
appearance control. It’s also achieved by freezing the weights of the original
Stable Diffusion model during the training phase. This enables the generation
of video content tailored to specific styles and domains.

Figure 3: Illustration of the reference-aware pipeline in our framework. Text
prompts and time embedding are incorporated into all modules, which are
omitted for simplicity.

####  3.4.2 Fine-Grained Appearance Control

Customized Center Frame. At times, professional video editors desire to edit
and produce videos strictly according to their own requirements. Additionally,
we empirically find that the trained temporal modules can sometimes suppress
the performance of the personalized T2I base model and LoRAs. Therefore, we
propose the reference-aware mechanism in our framework for controlling the
appearance of the target video from the fine-grained perspective. In the
workflow, users can initiate editing on the center frame initially[18, 11, 42,
28], allowing for meticulous and comprehensive control over the detailed
appearance. More specifically, as shown in Figure 3, the center frame of the
original video is edited and fed into the Appearance ControlNet. Subsequently,
the features extracted from each layer are added to the corresponding features
of the center frame in the base model’s encoder side. Consequently, the
appearance information within the center frame is propagated to all frames
through the temporal modules, achieving the stable and controllable editing.
During training, parameters of newly introduced Appearance ControlNet and
temporal consistency modules are jointly optimized. It’s worth noting that
different appearance control methods are not necessarily used in isolation.
They can be flexibly selected and combined according to the requirements to
achieve different purposes.

###  3.5 Temporal Consistency Modules

Pseudo-3D Layers Design. Inspired by previous works that turn the pre-trained
Stable Diffusion[27] into video generative models and meanwhile protecting the
strong content synthesising capability and language understanding capability
[3, 10], we freeze weights of all layers of original image generation model
and only update newly introduced temporal consistency modules to achieve the
motion smoothness and content coherence. Specifically, 2D Resblocks and
attention layers are transformed into pseudo-3D ones. As illustrated in Figure
4 (a) and (b), a 1D version of corresponding 2D transform layer is appended,
which is conducted on temporal dimension. Formally, suppose the spatial layer
and appended temporal layer are represented by $l^{i}_{\theta}$ and
$l^{i}_{\phi}$ parameterized by parameters $\theta$ and $\phi$, where $i$
indicates the index of corresponding layer. Given the feature
$\mathbf{z}^{i}\in\mathbb{R}^{B{\times}C{\times}T{\times}H{\times}W}$ of the
video clip, where $B$ indicates batch size, $C$ denotes feature channels, $T$
denotes sequence length, $H$ and $W$ presents the height and width of the
latent feature. The operation can be presented as follows (using einops [26]
notation):

$\displaystyle\mathbf{z}^{i}$
$\displaystyle\leftarrow\texttt{rearrange}(\mathbf{z}^{i},\;\texttt{b c t h
w}\rightarrow\texttt{(b t) c h w})$ $\displaystyle\mathbf{z}^{i}$
$\displaystyle\leftarrow l_{\theta}^{i}(\mathbf{z}^{i},\mathbf{c}^{i})$
$\displaystyle\mathbf{z}^{i}$
$\displaystyle\leftarrow\texttt{rearrange}(\mathbf{z}^{i},\;\texttt{(b t) c h
w}\rightarrow\texttt{(b h w) c t})$ $\displaystyle\mathbf{z}^{i}$
$\displaystyle\leftarrow l_{\phi}^{i}(\mathbf{z}^{i},\mathbf{c}^{i})$
$\displaystyle\mathbf{z}^{i}$
$\displaystyle\leftarrow\texttt{rearrange}(\mathbf{z}^{i},\;\texttt{(b h w) c
t}\rightarrow\texttt{b c t h w}),$

where $\mathbf{c}^{i}$ indicates the condition including time step and text
prompt.

Besides, we also apply skip connection and zero-initialized projection out
layer for each newly introduced module for progressive and stable updating.

Anchor-Aware Attention Module. For the technique of taking customized center
frame as reference for appearance control in Section 3.4.2, we additionally
introduce the anchor-aware attention module to enhance the control by the
reference center frame, which is shown in Figure 4 (c). Specifically, each
token calculates the corresponding key and value features based on the
concatenation of the current frame and the center frame. Formally, suppose
that the latent feature $\mathbf{z}$ (we omit the layer index $i$ for
simplicity), $\mathbf{z}_{j}$ and $\mathbf{z}_{k}$ represent features of the
$j$-th frame and the center frame. The anchor-aware attention is formulated as
follows:

$Q=W^{Q}\mathbf{z}_{j},K=W^{K}[\mathbf{z}_{j};\mathbf{z}_{k}],V=W^{V}[\mathbf{z}_{j};\mathbf{z}_{k}],$
(3)

where $W^{Q}$, $W^{K}$, $W^{V}$ are projection layers, and [$\cdot$] indicates
concatenation operation. Besides, we copy the original spatial self-attention
weights as initialization, with the exception of zero-initializing the
projection-out layer.

Figure 4: Details of proposed temporal modules. (a) The modified temporal-
aware Resblock. (b) The modified temporal-aware attention block. (c) The
modified anchor-aware attention block. We apply skip connection and zero-
initialized projection-out layer for each newly introduced module.

###  3.6 Temporal Interpolation

For video editing, a high frame rate of the edited video plays a crucial role
in enhancing visual continuity. The techniques described in Section 3.2,
Section 3.4, and Section 3.5 are capable of conducting edits on key frames
with customized changes. However, this editing process is constrained by
relatively low frame rates due to memory limitations. To achieve this, we
further introduce the temporal interpolation model to interpolate intermediate
frames between every two key frame. Similarly to the information incorporation
mechanism for the edited center frame as a reference introduced in Section
3.4.2, we use the Appearance Controlnet to extract the features of the two
given key frames and then add them to the features of the first and last
frame, respectively. The structure information of both key frames and
intermediate frames is input into the Structure ControlNet as structural
guidance. Then the temporal modules propagate appearance features from both
ends towards the center.

Figure 5: Results of video style translation. Our method can conduct style
transferring by flexibly combining different kinds of structure information
inherence and appearance control. $\langle\cdot\rangle$ indicate the
personalized T2I model we used.

Figure 6: Video editing results with customized center frame as reference. The
first row corresponds to customizing foreground, the second row corresponds to
customizing background, and the third row is taking given reference image to
affect the entire picture. $\langle\cdot\rangle$ indicate the personalized T2I
model we used.

Figure 7:  Video editing results under different structural guidance. The
types of structure information illustrated in this figure include “Lineart
(line drawings[5])”, “Softedge (PIDI boundaries[35])”, “Scribble (human
sketching)”, “Depth”. We use the personalized T2I model “ReV Animated” and
LoRA “kMechAnimal”. The prompt for all samples is “Mech4nim4lAI, tiger,
robotic.”

Figure 8:  Video editing results under different appearance control.
Dependence on the original Stable Diffusion pre-trained model alone failed to
produce satisfactory results. Even after transitioning to a personalized T2I
base model, the generated outcomes remained suboptimal. Only when we initially
edited the center frame and take it as the appearance reference that we
achieved the expected results.

##  4 Experiments

###  4.1 Implementation Details

Training. Stable Diffusion-v1.5 is used as the base model to train temporal
consisitency modules and Appearance ControlNet. We use the pre-trained
ControlNet[42] for the structure information guidance. The training dataset is
a self-collected private dataset. We trained the temporal consistency modules
and Appearance ControlNet for various types of structural information,
including line drawings[5], PIDI boundaries[35], depth maps detected by
Midas[25], and human scribbles. During the training process, we first resize
the shorter side to 384 pixels, followed by a random crop to obtain video
clips with a size of 384$\times$576\. 17 frames at 4 fps are sampled from each
video. The batch size is 16 and the learning rate is $3e-5$.

Evaluation. We validate the effectiveness and functionality of our approach
through the use of diverse combinations of structure control and appearance
control. Structure control encompasses line drawings, PIDI boundaries[35],
depth maps detected by Midas, and human scribbles, in a fine-to-coarse manner.
For appearance control, we collect several personalized T2I base models and
LoRA weights from CivitAI[6] and explored different combinations, which are
illustrated in Table 1. Similar to previous work[10], we employ the “trigger
words” to activate these personalized models.

Model Name | Type
---|---
Counterfeit | T2I Base Model
ToonYou | T2I Base Model
ReV Animated | T2I Base Model
HelloMecha | T2I Base Model
hellonijicute25d | T2I Base Model
A-Zovya Photoreal | LoRA
kMechAnimal | LoRA
Pixel Art Style | LoRA
fat animal | LoRA
Building Block World | LoRA
MoXin | LoRA
mechanical dog | LoRA
Table 1:  Personalized models utilized in the evaluation, all sourced from
CivitAI[6].

###  4.2 Applications

Video Style Translation. The basic function of our framework lies in video
style translation. By flexibly combining various types of structural and
appearance control, users are empowered to translate videos into styles that
align with their preferences. We show some samples in Figure 5, including
anime style, pixel art style, and mechanical style, _etc_. These results
demonstrate the superior coordination of our method in integrating various
types of structure control and appearance control, affirming the compatibility
and controllability of the framework.

Customized Video Editing. Sometimes, users require stronger control over the
content they want to generate. For example, they may want to change only the
foreground, alter just the background, or edit the texture content of a video
in a specific way. In our framework, this can be achieved by first using image
editing techniques to modify the content of the intermediate frame and then
using it as a reference for editing the entire video. As depicted in Figure 6,
we first edit the center frames of the videos by Stable Diffusion Web UI[1],
followed by utilizing these edited central frames as guides for the video
editing process. Thanks to the joint training of Appearance ControlNet and
temporal consistency modules, our method can effectively propagate the edited
information of the center frame to the whole video in a coherent manner.

###  4.3 Ablation Study

Structure Control. As shown in Figure 7, we also experimented with how
different structure control methods affect the results. Line drawings (2nd
row) and PIDI boundaries (3rd row) provide detailed control over the
structure, making them more suitable for use cases that require a significant
preservation of the structure. On the other hand, scribbles (4th row) and
depth maps (5th row) offer a coarser level of control over the structure,
making them suitable for more imaginative use cases.

Appearance Control. Figure 8 illustrate the importance of taking the
customized center frame as a reference in certain scenarios. Initially,
translating the video scenes into “cyberpunk” style (1st row) solely through
prompt adjustments appears challenging, as this word may be unfamiliar to the
pre-trained Stable Diffusion and the temporal consistency modules. Even after
loading a T2I base model that is more familiar with such terminology, it still
doesn’t seem to achieve the desired outcome effectively. This could
potentially be due to the fact that the temporal consistency modules may not
be highly compatible with just any T2I base model. The solution is illustrated
in the third row in Figure 8, that is, first edit the center frame and then
combine it with other appearance control factors to manage the appearance of
the video.

##  5 Limitation and Future Works

Out-of-Distribution Personalized Models. As shown in Figure 9, our approach
encounters challenges in generating satisfactory results when using
personalized models that are too out of distribution. This issue can probably
be tackled by collecting relevant text-video paired data and fine-tuning the
temporal modules.

Figure 9:  Poor temporal coherence can occur when the target video deviates
significantly from the training data distribution. In this experiment, we
employ the T2I base model of “ToonYou”, and the text prompt provided is: “A
man walking in the wonderland, lush, bioluminescent, vibrant, dense, unique,
towering, floating, colorful, enormous, intricate, expansive, rich
biodiversity, exotic, mystical.”

Heavy ControlNet. In this paper, we directly utilize the pre-trained
ControlNet to perform structure control. However, due to ControlNet’s
evolution from the original Stable Diffusion encoder, it possesses a
substantial parameter count (more than 400M). Furthermore, for reference-aware
scenarios, two ControlNets are employed, with each being responsible for
extracting structure and appearance information, further increasing the
parameters of the overall model. This may be unnecessary and could lead to
issues such as increased GPU memory consumption and longer inference time. In
the future, we plan to explore the adoption of more lightweight structure
control networks[18, 28] to address these concerns.

Flickering Problem. We observed flickering in videos with higher frame rates
or after frame interpolation, especially noticeable in high-frequency fine
texture details. This is primarily attributed to our video editing operations
being performed in the latent domain encoded by the 2D auto-encoder.
Introducing additional temporal layers in the autoencoder[3] is an promising
way to solve this problem.

##  6 Conclusion

In this paper, we propose a unified and practical framework for diffusion-
based video editing. Our key design choice is to decouple the video editing
process into structure control and appearance control. We utilize the pre-
trained ControlNet for structure control, and we achieve appearance control
through three different ways, _i.e_., text prompt, personalized model weights,
and customized center frame as reference. For the appearance control of using
a customized center frame, we innovatively introduced the Appearance
ControlNet. This network is responsible for extracting features from the
edited center frame and seamlessly integrating them into the base model.
Additionally, we introduce temporal consistency modules to maintain temporal
coherence. Last but not least, we also propose the temporal interpolation
model to generate high frame rate videos. Our work offers a comprehensive and
flexible set of tools for creative and controllable video editing. We hope
that our work can benefits practitioners in the context of creative and
controllable video editing.

## References

  * [1] AUTOMATIC1111.  Stable Diffusion Web UI, Aug. 2022.
  * [2] Omri Avrahami, Dani Lischinski, and Ohad Fried.  Blended diffusion for text-driven editing of natural images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18208–18218, 2022.
  * [3] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis.  Align your latents: High-resolution video synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023.
  * [4] Wenhao Chai, Xun Guo, Gaoang Wang, and Yan Lu.  Stablevideo: Text-driven consistency-aware diffusion video editing.  arXiv preprint arXiv:2308.09592, 2023.
  * [5] Caroline Chan, Frédo Durand, and Phillip Isola.  Learning to generate line drawings that convey geometry and semantics.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7915–7925, 2022.
  * [6] Civitai.  Civitai.  https://civitai.com/, 2022.
  * [7] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord.  Diffedit: Diffusion-based semantic image editing with mask guidance.  arXiv preprint arXiv:2210.11427, 2022.
  * [8] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  Advances in neural information processing systems, 34:8780–8794, 2021.
  * [9] Hugging Face.  Hugging face.  https://huggingface.co/, 2022.
  * [10] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai.  Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.  arXiv preprint arXiv:2307.04725, 2023.
  * [11] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  arXiv preprint arXiv:2208.01626, 2022.
  * [12] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al.  Imagen video: High definition video generation with diffusion models.  arXiv preprint arXiv:2210.02303, 2022.
  * [13] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in neural information processing systems, 33:6840–6851, 2020.
  * [14] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685, 2021.
  * [15] Jun Hao Liew, Hanshu Yan, Jianfeng Zhang, Zhongcong Xu, and Jiashi Feng.  Magicedit: High-fidelity and temporally coherent video editing.  arXiv preprint arXiv:2308.14749, 2023.
  * [16] Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia.  Video-p2p: Video editing with cross-attention control.  arXiv preprint arXiv:2303.04761, 2023.
  * [17] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Guided image synthesis and editing with stochastic differential equations.  arXiv preprint arXiv:2108.01073, 2021.
  * [18] Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie.  T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.  arXiv preprint arXiv:2302.08453, 2023.
  * [19] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  arXiv preprint arXiv:2112.10741, 2021.
  * [20] Hao Ouyang, Qiuyu Wang, Yuxi Xiao, Qingyan Bai, Juntao Zhang, Kecheng Zheng, Xiaowei Zhou, Qifeng Chen, and Yujun Shen.  Codef: Content deformation fields for temporally consistent video processing.  arXiv preprint arXiv:2308.07926, 2023.
  * [21] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu.  Zero-shot image-to-image translation.  In ACM SIGGRAPH 2023 Conference Proceedings, pages 1–11, 2023.
  * [22] Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen.  Fatezero: Fusing attentions for zero-shot text-based video editing.  arXiv preprint arXiv:2303.09535, 2023.
  * [23] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
  * [24] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In International Conference on Machine Learning, pages 8821–8831. PMLR, 2021.
  * [25] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  IEEE transactions on pattern analysis and machine intelligence, 44(3):1623–1637, 2020.
  * [26] Alex Rogozhnikov.  Einops: Clear and reliable tensor manipulations with einstein-like notation.  In International Conference on Learning Representations, 2021.
  * [27] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [28] Denis Zavadski Carsten Rother.  Controlnet-xs.  2023\.
  * [29] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500–22510, 2023.
  * [30] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  Advances in Neural Information Processing Systems, 35:36479–36494, 2022.
  * [31] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  Advances in Neural Information Processing Systems, 35:25278–25294, 2022.
  * [32] Chaehun Shin, Heeseung Kim, Che Hyun Lee, Sang-gil Lee, and Sungroh Yoon.  Edit-a-video: Single video editing with object-aware consistency.  arXiv preprint arXiv:2303.07945, 2023.
  * [33] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al.  Make-a-video: Text-to-video generation without text-video data.  arXiv preprint arXiv:2209.14792, 2022.
  * [34] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  arXiv preprint arXiv:2011.13456, 2020.
  * [35] Zhuo Su, Wenzhe Liu, Zitong Yu, Dewen Hu, Qing Liao, Qi Tian, Matti Pietikäinen, and Li Liu.  Pixel difference networks for efficient edge detection.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 5117–5127, 2021.
  * [36] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel.  Plug-and-play diffusion features for text-driven image-to-image translation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1921–1930, 2023.
  * [37] Wen Wang, Kangyang Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang, and Chunhua Shen.  Zero-shot video editing using off-the-shelf image diffusion models.  arXiv preprint arXiv:2303.17599, 2023.
  * [38] Yuanzhi Wang, Yong Li, Xin Liu, Anbo Dai, Antoni Chan, and Zhen Cui.  Edit temporal-consistent videos with image diffusion model.  arXiv preprint arXiv:2308.09091, 2023.
  * [39] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.  arXiv preprint arXiv:2212.11565, 2022.
  * [40] Zhen Xing, Qi Dai, Han Hu, Zuxuan Wu, and Yu-Gang Jiang.  Simda: Simple diffusion adapter for efficient video generation.  arXiv preprint arXiv:2308.09710, 2023.
  * [41] Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy.  Rerender a video: Zero-shot text-guided video-to-video translation.  arXiv preprint arXiv:2306.07954, 2023.
  * [42] Lvmin Zhang and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  arXiv preprint arXiv:2302.05543, 2023.
  * [43] Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian.  Controlvideo: Training-free controllable text-to-video generation.  arXiv preprint arXiv:2305.13077, 2023.
  * [44] Min Zhao, Rongzhen Wang, Fan Bao, Chongxuan Li, and Jun Zhu.  Controlvideo: Adding conditional control for one shot text-to-video editing.  arXiv preprint arXiv:2305.17098, 2023.


Figure S14: Analysis of drift on the timescale of seconds. a. Raw results of
the Kruskal-Wallis test applied to experimental data. Results are shown out to
depth 75, but the test was run on the full dataset out to depth 200. From this
we see that no exceptionally small p-values are reported. b Plot of the
cumulative distribution of the obtained p-values, for both the real
experimental data set and a truly drift-less open system simulation. The test
was completed for 200 depths and 4 different chunk sizes, so there are in
total 800 points in the distribution. We see that the p-values generated using
both the real and simulated datasets form a uniform distribution, which
suggests that the null hypothesis is true and there is not evidence of drift
in the data. The data and simulation distributions are also nearly coincident,
which further supports this claim.

The way these p-values are distributed tells us if there is drift in the
bitstrings or not. If the null is true and a statistical test has been used
correctly (it’s assumptions have been met), by the definition of a p-value one
expects p-values to be uniformly distributed. As a test of this, we plot the
distribution of p-values found by applying the previously described procedure
to data generated by an open system simulation. The results of this are shown
by the blue curve in Fig. S14 b. We see that the CDF of the p-values is nearly
exactly linear, as expected. We then plot the distribution of p-values
obtained from running the procedure on real data, as shown by the orange
curve. we find that this distribution is also uniform and is almost
indistinguishable from the distribution generated using simulated data (up to
shot noise at very small p-values). This very strongly suggests that the null
is true and that there is no statistically significant drift on the timescale
of a single package. Combining this with the excellent fitting results shown
in Fig. S13, it seems that for one reason or another this experiment does not
experience drift on the timescale of several minutes.

####  F.3 Statistical Analysis of Parameter Estimates

In any parameter estimation task, there are always two independent measures of
success; bias and variance. Bias describes the difference in the mean value of
an estimator and the actual parameter. It is desirable for an estimator
$\hat{\theta}$ of the underlying parameter $\theta$ to be unbiased, eg
$E(\hat{\theta}-\theta)=0$. In practice, it is of course impossible to measure
bias. Doing this would require knowledge of the true underlying distribution
of $\theta$, which is generally what we are trying to measure in the first
place. Bias can be avoided by choosing a model that is a good fit for the data
being considered. The fit achieved with the open system model in Fig. S13
provides strong evidence that the parameter estimates achieved using this
technique are unbiased on the $10^{-3}$ scale, as it it hard to imagine how
the bias in parameter estimation could be substantially larger than the
average residual in the fit of a minimal physical model.

The variance of a parameter estimate measures only statistical uncertainty and
has nothing to do with the underlying distribution. As such, it can be
rigorously analyzed and quantified. The variance of an estimator is given by
$\sigma^{2}=E\left((\hat{\theta}-\bar{\hat{\theta}})^{2}\right)$. By the
central limit theorem, the standard deviation $\sigma$ of a parameter estimate
will typically scale with $1/\sqrt{n}$, where $n$ is the number of samples
used in making the estimate. As mentioned in the main body and in previous
work zhang2020, the novelty of floquet calibration is that parameters are
extracted at the Heisenberg limit; the scaling of the standard deviation with
number of samples should exceed the central limit theorem and scale with $1/n$
instead of $1/\sqrt{n}$. This scaling will be carefully established for this
open chain control sequence in the following work.

First, it is important to establish that we actually expect to be able to
extract the quasi-energy differences, and not the parameter values themselves,
with Heisenberg scaling. To see this, it is simplest to consider the
populations that would result from the unitary evolution of some initial state
${\lvert k\rangle}{\langle k\rvert}$, as in equation 152. The populations are
tied to the parameter values through explicit dependence on both the floquet
eigenvectors ${\lvert\psi^{\alpha}\rangle}$ and the quasi-energy differences
$\omega_{\alpha\beta}$. However, only the dependance on the quasi-energy
differences scales with the depth of the circuit $d$. Therefore, the
sensitivity of the populations to perturbations in the parameters only grows
with depth though the quasi-energies. Noting that the entire quasi-energy
difference matrix can be constructed from $N-1$ ”fundamental” quasi-energy
differences $\omega_{\beta 1}$, we should be able to learn $N-1$ linear
combinations of the $2(N-1)$ unitary parameters with Heisenberg scaling from a
single set of populations. This same argument can be made looking at the open
system model presented in equation 153.

To solidify this argument we consider evaluating the variance of the parameter
estimate though the inverse of the information matrix. Given a vector of
parameters $\vec{\theta}$, one may obtain the variance-covariance matrix of a
maximum likelihood estimate of $\vec{\theta}$ through the inverse of the
hessian of the log likelihood function. The log-likelihood estimator for this
problem is given by

$\mathcal{L}(\vec{\theta})=\sum_{d=0}^{D}N_{d}\sum_{k=0}^{n}p^{\text{data}}_{dk}\ln\left(p^{\text{model}}_{dk}(\vec{\theta})\right)\>,$
(164)

with $N_{d}$ being the number of bitstrings collected at depth $d$ and the
matrix $P_{dk}$ as defined in section F. $p^{\text{model}}_{dk}$ in the
secular approximation is given explicitly

The variance-covariance matrix $\mathrm{var}(\vec{\theta})$ is then given by

$\displaystyle\begin{split}\mathrm{var}(\vec{\theta})_{kj}=\left(-E\left(\pdv{\mathcal{L}(\vec{\theta})}{\theta_{j},\theta_{k}}\right)\right)^{-1}\>.\end{split}$
(165)

In practice, it is easiest to simply evaluate this expression directly in
TensorFlow to calculate the variance of the obtained parameter estimates.
However, further analytical analysis is useful for gaining an understanding of
the general behavior of this function. A simple expression can be developed by
realizing that Eq. 158 is only strongly coupled to the quasi-energy
differences via the oscillating term $e^{-i\omega_{\alpha\beta}d}$; coupling
through the transition and decay coeffecients is approximately $10^{-5}$ times
weaker due to the $\Gamma$ prefactors and is therefore irrelevant when
considering second derivatives. Following this logic, one can arrive at the
below expression for the Hessian of the log-likelihood function with respect
to the fundamental quasi-energy differences in the open system case.

$\displaystyle\begin{split}\pdv{\mathcal{L}(\vec{\omega})}{\omega_{\beta
1},\omega_{\alpha
1}}\approxeq&-4\sum_{d=0}^{D}d^{2}N_{d}\sum_{k=1}^{n}\frac{f(\alpha,\beta,k,d)}{p^{\text{model}}_{dk}}e^{-dt_{\mathrm{cycle}}(Y_{\alpha
1}+Y_{\beta 1})}\\\ \end{split}$ (166)

Where $f(\alpha,\beta,k,d)$ is a function whose magnitude does not scale with
d,

$f(\alpha,\beta,k,d)=\mathrm{Im}\left({\langle\psi^{\alpha}\rvert}\rho(0){\lvert\psi^{1}\rangle}{\langle
k\rvert\psi^{\alpha}\rangle}{\langle\psi^{1}\rvert
k\rangle}e^{-i\omega_{\alpha
1}dt_{\mathrm{cycle}}}\right)\mathrm{Im}\left({\langle\psi^{\beta}\rvert}\rho(0){\lvert\psi^{1}\rangle}{\langle
k\rvert\psi^{\beta}\rangle}{\langle\psi^{1}\rvert k\rangle}e^{-i\omega_{\beta
1}dt_{\mathrm{cycle}}}\right)\>.$ (167)

For data collected with sparsely sampled depths, equation 166 scales with
$D^{2}$ when the depth is sufficiently small such that $e^{-dt_{\rm
cycle}(Y_{\alpha 1}+Y_{\beta 1})}\approxeq 1$. In this regime, the variance-
covariance matrix will scale with $\frac{1}{D^{2}}$ and the standard
deviations of the quasi energies will scale with $\frac{1}{D}$. This is the
desired Heisenberg scaling. Beyond this regime, the variance will begin to
increase due to incoherent effects. There is therefore an optimal depth that
will allow for estimation of the parameters at the quantum limit, as observed
in the two qubit case in section A and Ref. zhang2020.

If one repeats the above analysis but takes the hessian with respect to the
$2(N-1)$ unitary parameters directly, they will find that only $N-1$ of it’s
eigenvalues have the desired $D^{2}$ scaling. The eigenvectors associated with
these eigenvalues represent the $N-1$ linear combinations of the $2(N-1)$
unitary parameters that can be learned from a single experiment with
Heisenberg scaling. Alternatively, these linear combinations can be found more
simply as the rows of the Jacobian of the fundamental quasi-energy differences
with respect to the parameters.

####  F.4 Readout Correction

A readout correction procedure is applied to the bitstrings in each package to
estimate the probabilities. As discussed in the main text, this is not
strictly necessary as the parameter information is encoded in the frequency
domain. However, completing this process removes an imperfection from the open
system model, reducing the number of experimental imperfections that need to
be studied if the fit to the data is imperfect. We assume that readout error
is a classical process, and it can therefore be inverted by solving the
constrained quadratic problem given below.

$\displaystyle\mathcal{L}=$
$\displaystyle|A\vec{v}^{prep}-\vec{v}^{meas}|^{2}$ (168) $\displaystyle
v^{prep}_{k}>$ $\displaystyle 0$ $\displaystyle\sum_{k}v^{prep}_{k}=$
$\displaystyle 1$

$\vec{v}^{prep}$ is a length $n+1$ vector containing the probabilities
${\langle m\rvert}\rho{\lvert m\rangle}$ for $m\in[0,n]$. $\vec{v}^{meas}$ is
a sparse vector of computational basis measurement probabilities with at most
$N$ entries given a package of $N$ bitstrings. $A$ is therefore a sparse
matrix which can be measured efficiently by sequentially preparing the ground
state and the single excitation states and measuring in the computational
basis. Unlike most NISQ experiments, because we work in the 0-and-1 excitation
subspace the dimension of $\vec{v}^{prep}$ grows linearly instead of
exponentially and the matrix $A$ can be measured without any assumptions about
spatial correlation in the readout error. Therefore, this process will be
optimal up to statistical error and quanutum effects in the readout, such as
AC stark shifts.

###  Appendix G Band-structure demonstration with 30 qubits

Figure S15: Measuring the single-particle band-structure with 30 qubits. a
Schematic of the 54-qubit Sycamore processor. Qubits are shown as gray crosses
and tunable couplers as blue squares. Thirty of the qubits are isolated to
form a one-dimensional ring. b Schematic showing the control sequence used in
this experiment. Each large vertical gray box indicates a cycle of evolution
which we repeat many times. Each cycle contains two sequential layers of
$\surd{\text{i}}\text{SWAP}$ gates (blue). c Raw data measured in this
experiment. Each qubit is measured in the x-basis and the data is plotted as a
function of the position of the qubit along the ring and the number of cycles.
d A Fourier-transform of the data is taken along both the x-axis and the
y-axis revealing the single-particle band-structure. By taking a Fourier-
transform in position, we arrive at a signal with only a single frequency
(each vertical column). This greatly simplifies the data analysis over the
method in the main text where all of the frequencies needed to be extracted
from a single signal.