11institutetext:  DFKI GmbH, Alt-Moabit 91c, 10559 Berlin, Germany
22institutetext: 3pc GmbH Neue Kommunikation, Prinzessinnenstraße 1, 10969
Berlin, Germany 33institutetext: Ada Health GmbH, Adalbertstraße 20, 10997
Berlin, Germany 44institutetext: ART+COM AG, Kleiststraße 23-26, 10787 Berlin,
Germany 55institutetext: Condat AG, Alt-Moabit 91d, 10559 Berlin, Germany
66institutetext: Fraunhofer FOKUS, Kaiserin-Augusta-Allee 31, 10589 Berlin,
Germany 77institutetext: Semtation GmbH, Geschwister-Scholl-Straße 38, 14471
Potsdam, Germany 88institutetext: Staatsbibliothek zu Berlin (SPK), Potsdamer
Straße 33 10785 Berlin, Germany 99institutetext: Ubermetrics Technologies
GmbH, Kronenstraße 1, 10117 Berlin, Germany 1010institutetext: Wikimedia
Deutschland e. V., Tempelhofer Ufer 23-24, 10963 Berlin, Germany
Corresponding author: Georg Rehm – 1010email<EMAIL_ADDRESS>
# QURATOR: Innovative Technologies for Content and Data Curation

Georg Rehm 11 Peter Bourgonje 11 Stefanie Hegele 11 Florian Kintzel 11
Julián Moreno Schneider 11 Malte Ostendorff 11 Karolina Zaczynska 11
Armin Berger 22 Stefan Grill 22 Sören Räuchle 22 Jens Rauenbusch 22
Lisa Rutenburg 22 Andre Schmidt 22 Mikka Wild 22 Henry Hoffmann 33
Julian Fink 33 Sarah Schulz 33 Jurica Ševa 33 Joachim Quantz 44
Joachim Böttger 44 Josefine Matthey 44 Rolf Fricke 55 Jan Thomsen 55
Adrian Paschke 66 Jamal Al Qundus 66 Thomas Hoppe 66 Naouel Karam 66
Frauke Weichhardt 77 Christian Fillies 77 Clemens Neudecker 88 Mike Gerber 88
Kai Labusch 88 Vahid Rezanezhad 88 Robin Schaefer 88 David Zellhöfer 88
Daniel Siewert 99 Patrick Bunk 99 Julia Katharina Schlichting 99
Lydia Pintscher 1010 Elena Aleynikova 1010 Franziska Heine 1010

###### Abstract

In all domains and sectors, the demand for intelligent systems to support the
processing and generation of digital content is rapidly increasing. The
availability of vast amounts of content and the pressure to publish new
content quickly and in rapid succession requires faster, more efficient and
smarter processing and generation methods. With a consortium of ten partners
from research and industry and a broad range of expertise in AI, Machine
Learning and Language Technologies, the QURATOR project, funded by the German
Federal Ministry of Education and Research, develops a sustainable and
innovative technology platform that provides services to support knowledge
workers in various industries to address the challenges they face when
curating digital content. The project’s vision and ambition is to establish an
ecosystem for content curation technologies that significantly pushes the
current state of the art and transforms its region, the metropolitan area
Berlin-Brandenburg, into a global centre of excellence for curation
technologies.

###### Keywords:

Curation Technologies Language Technologies Semantic Technologies Knowledge
Technologies Artificial Intelligence

##  1 Introduction

Digital content and online media have gained immense importance, especially in
business, but also in politics and many other areas of society. Some of the
many challenges include better support and smarter technologies for digital
content curators who are exposed to an ever increasing stream of heterogeneous
information they need to process further. For example, professionals in a
digital agency create websites or mobile apps for customers who provide
documents, data, pictures, videos etc. that are processed and then deployed as
new websites or mobile apps. Knowledge workers in libraries digitize archives,
add metadata and publish them online. Journalists need to continuously stay up
to date to be able to write a new article on a specific topic. Many more
examples exist in various industries and media sectors (television, radio,
blogs, journalism, etc.). These diverse work environments can benefit
immensely from smart semantic technologies that help content curators, who are
usually under great time pressure, to support their processes. Currently, they
use a wide range of non-integrated, isolated, and fragmented tools such as
search engines, Wikipedia, databases, content management systems, or
enterprise wikis to perform their curation work. Largely manual tasks such as
smart content search and production, summarization, classification as well as
visualization are only partially supported by existing tools [14].

The QURATOR project111https://qurator.ai, funded by the German Federal
Ministry of Education and Research (BMBF), with a project runtime of three
years (11/2018-10/2021), is based in the metropolitan region
Berlin/Brandenburg. The consortium of ten project partners from research and
industry combines vast expertise in areas such as Language Technologies as
well as Knowledge Technologies, Artificial Intelligence and Machine Learning.
The project’s main goal is the development of a sustainable technology
platform that supports knowledge workers in various industries. Non-efficient
process chains increase the manual processing effort for workers even more.
The platform will simplify the curation of digital content and accelerate it
dramatically. AI techniques are integrated into curation technologies and
curation workflows in the form of industry solutions covering the entire life
cycle of content curation. The solutions being developed focus on curation
services for the sectors of culture, media, health and industry.

In Section 2 we describe the emerging QURATOR technology platform. In the main
part of this article, Section 3, we provide brief summaries of the ten partner
projects. Section 4 concludes the article with a short summary.

##  2 The QURATOR Curation Technology Platform

The centerpiece of the project is the development of a platform for digital
curation technologies. The project develops, integrates and evaluates various
services for importing, preprocessing, analyzing and generating content that
covers a wide range of information sources and data formats, spanning use
cases from several industries and domains. A special focus of the project is
on the integration of AI methods to improve the quality, flexibility and
coverage of the services.

Figure 1 outlines the concept of the QURATOR Curation Technology Platform
which can be divided into three main layers. In order to process and transform
incoming data, text and multimedia streams from different sources to device-
adapted, publishable content, various groups of components, services and
technologies are applied. First, the set of basic technologies includes
adapters to data, content and knowledge sources, as well as infrastructural
tools and smart AI methods for the acquisition, analysis and generation of
content. Second, knowledge workers can make use of curation tools and services
which have knowledge sources and intelligent procedures already integrated in
order to process content. Third, there are selected use case and application
areas (culture, media, health, industry), i. e., the respective integration of
curation tools and services. Each of the three layers has already been
populated with technology components which are to be further developed and
also extended in the following two years of the project.

Figure 1: Architectural concept of the QURATOR Curation Technology Platform

##  3 QURATOR – Partner Projects

The project consortium includes ten partners: the research centers DFKI and
Fraunhofer FOKUS, the industry partners 3pc, Ada Health, ART+COM, Condat,
Semtation, Ubermetrics as well as Wikimedia Germany and the Berlin State
Library (Stiftung Preußischer Kulturbesitz). Several of these partners already
contributed to previous BMBF-funded projects including Corporate Smart
Content222https://www.unternehmen-region.de/de/7923.php and Digital Curation
Technologies333http://digitale-kuratierung.de, which focused on semiautomatic
methods for the efficient processing, creation and distribution of high
quality media content and laid the groundwork for the QURATOR project.

In the following, we briefly introduce each partner and provide an overview of
their respective projects focusing upon the current state and the next steps.

###  3.1 DFKI GmbH: A Flexible AI Platform for the Adaptive Analysis and
Creative Generation of Digital Content

DFKI (Deutsches Forschungszentrum für Künstliche Intelligenz GmbH) is
Germany’s leading research center in the field of innovative software
technologies based on AI methods. The Speech and Language Technology Lab
conducts advanced research in language technology and provides novel
computational techniques for processing text, speech and knowledge.

In QURATOR, DFKI focuses on the development of an innovative platform for
digital curation technologies [2, 13, 21, 22] as well as on the population of
this platform with various processing services. This platform plays a crucial
role in the project as it is being designed together with all partners who
also contribute services to the platform.444This platform is developed in
close collaboration with the EU project European Language Grid, which is also
coordinated by DFKI, see https://www.european-language-grid.eu and [11] for
more details. Ultimately, the QURATOR platform will contain services, data
sets and components that are able to handle and to process different types and
classes of content as well as content sources. The DFKI services can be
divided into three classes.

_Preprocessing_ encompasses the services that are responsible for obtaining
and processing information from different content sources so that they can be
used in the platform and integrated into other services [23]. These services
include the provisioning of data sets and content (web pages, RSS feeds etc.),
language and duplicate detection as well as document structure recognition.

_Semantic analysis_ includes services that process a document (or part of it)
and add information in the form of annotations. These services are named
entity recognition and linking, temporal expression analysis, relation
extraction, event detection, fake news as well as discourse analysis [3, 25,
16, 9].

_Content generation_ contains services that make use of annotated information
(semantic analysis) to help create a new piece of information. These services
are summarization, paraphrasing, automatic translation and semantic
storytelling for both text and multimedia content [17, 15, 7, 12, 20, 19].

DFKI will continue the development of the different services as well as the
infrastructure. Since a flexible organization needs to be guaranteed, DFKI is
also responsible for the design and implementation of workflows. These will
ultimately enable the joint use of (almost) all the services available in the
platform.

###  3.2 3pc GmbH: Curation Technologies for Interactive Storytelling

3pc creates solutions for the digital age, combining strategy, design,
technology, and communication in a holistic and user-centered approach. As
experts in the development of novel and unique digital products, 3pc
identifies core challenges and key content within complex subject matters.

As part of the QURATOR project, 3pc develops intelligent tools for interactive
storytelling [1, 15] in order to assist editors, content curators, and
publishers from cultural and scientific institutions, corporate communication
divisions, and media agencies. The providers for storytelling face an
increasing challenge telling engaging stories built from vast amounts of data
for a broad range of devices, including wearables, augmented and virtual
reality systems, voice-based user interfaces – and whatever the future holds.
In this context, interactive storytelling is defined as novel media formats
and implementations that exceed today’s rather static websites by far. 3pc is
currently building an asset management tool that enables users to access media
analysis algorithms in an intuitive and efficient way (Figure 2). Media
analysis processes text, images, videos, and audio files in order to enrich
them with additional information such as content description, sentiment or
topic, which is usually a labor-intensive and, therefore, expensive process
often neglected in busy publishing environments. Enriched media becomes
machine-readable, allowing storytellers to find content faster and for new
connections to be forged in order to create richer, interactive stories.
Ultimately, a semantic storytelling machine becomes possible, generating semi-
automatically unique and tailored stories, based entirely on user preferences.
At 3pc, research is conducted through an iterative process by creating
functional prototypes and testing their usefulness on representative members
of different user groups. 3pc ensures that all novel technology solutions are
adapted to each user’s needs, taking into consideration their tasks, behaviour
and knowledge.

Next up, 3pc will extend traditional forms of interactive storytelling by
exploring space, voice, and generated audio as means of human-computer
interaction. Further research will also be conducted on training algorithms
for domain-specific tasks in order to develop curation tools for different
areas of expertise.

[capbesideposition=right,top,capbesidewidth=0.25]figure[]

Figure 2: Prototype of an asset management tool for analyzing and enriching
multimedia files. The screenshot depicts a video analysis, creating semi-
automated transcriptions, sections, clips, and thumbnails, as well as entity
recognition and sentiment analysis.

###  3.3 Ada Health GmbH: Curation of Biomedical Knowledge

Ada Health GmbH is a global health company founded by doctors, scientists, and
industry pioneers to create new possibilities for personal health. Ada’s core
system connects medical knowledge with intelligent technology to help people
actively manage their health and medical professionals to deliver effective
care.

Within the QURATOR project, Ada focuses on supporting the structured medical
knowledge creation by providing a tool for pre-extracting information from
unstructured text. This tool utilizes methods from biomedical Natural Language
Processing (NLP). Since the quality of the medical database is of utmost
importance to ensure accurate diagnosis support, the “human in the loop”
approach leverages the deep medical knowledge provided by Ada’s doctors and
the efficiency of AI methods.

As a first step, Ada’s researchers focus on the extraction of medical entities
from medical case reports. These descriptions of a patient’s symptoms are
usually semi-structured and can function as test cases for Ada’s quality
control. The extraction of a structured case requires NLP methods such as the
detection of relevant paragraphs, named entity recognition and named entity
normalization. Challenging characteristics of named entities in the biomedical
domain are their discontinuous nature in text as well as their high
heterogeneity in terms of linguistic features. Thus, these domain-specific
characteristics require the adaptation and implementation of domain-tailored
NLP solutions. In order to do so, data is required which Ada acquires through
a combination of manual annotation and active learning from feedback given by
the medical content editors.

###  3.4 ART+COM AG: Curation Tools for Multimedia Content

ART+COM Studios designs and develops new media spaces and installations. New
technology is not only used as an artistic medium of expression but as a
medium for the interactive communication of complex information. In the
process, ART+COM improves existing technologies constantly and explores their
applications both independently and in cooperation with other companies and
academic institutions.

The main focus within QURATOR is to develop basic technologies to
automatically process and assess multimedia content and ultimately create
smart exhibits that can organize and generate content automatically.

[capbesideposition=right,top,capbesidewidth=0.45]figure[]

Figure 3: Prototype of the Wikidata knowledge graph tool displaying a
selection of items and their connections to each other. The layout of the
graphs is achieved using force-directed algorithms in combination with high-
dimensional embeddings and manual curation.

The current objective is to categorize entities and visualize their relations
in huge datasets. ART+COM’s curation tool will import results from machine
learning (ML) methods, including NLP, action detection, image recognition and
processing of interconnected data. Subsequently, it should be used to create
automated content analyses and visualizations that are navigable and support
knowledge workers as well as the creation of interactive museum exhibits. The
interconnected entities in Wikidata, one of the most relevant databases for
researchers, are subject of one of the ongoing sub-projects. The project
focuses on an interactive software to visualize, explore, and curate knowledge
contained in Wikidata. Of particular interest are interaction techniques to
filter and select the objects of interest and their relationships between each
other. The software framework also explores potential data arrangements in
two-dimensional and three-dimensional space, tailored to their relevance to
particular questions and interests. Figure 3 shows the prototype of the
Wikidata knowledge graph tool displaying a selection of items and their
connections to each other.

###  3.5 Condat AG: Smart Newsboard

Condat AG has a strong focus on the media industry, mainly on public
broadcasters. Condat support all parts of the distribution chain for these
broadcasters.

Figure 4: Smart Newsboard to produce content based on news around a particular
topic

Within the QURATOR project, Condat develops a Smart Newsboard (Figure 4) as a
means to produce content based on news around a particular topic. This
involves a chain of different curation services such as 1) finding the
original sources around a topic by searches and subscription to RSS feeds,
Twitter channels etc., 2) categorizing and classifying sources into meaningful
groups, either through topic detection (if the categories are not predefined)
or text classification (if the categories are fixed), or even a combination of
both, 3) applying text analysis, named entity recognition and enrichment by
linking the entities to specific resources such as Wikidata which allows the
building of a knowledge graph to find connections between, e. g., people and
events in different contexts and, thus, enable journalists to pursue a deeper
analysis. Condat also explores the identification of temporal expressions
which enables the possibility to generate story outlines based on the sequence
of events as extracted from multiple documents (timelining). Another curation
service relevant for the Smart Newsboard is the summarization of sources. This
includes not only the summarization of individual texts but more importantly
the summarization of multiple documents.

Curation services for summarization, named entity recognition and topic
detection have already been implemented. The next steps are the integration of
additional services, as they become available, and the design of the user
interface.

###  3.6 Fraunhofer FOKUS: Corporate Smart Insights

The Fraunhofer Institute for Open Communication Systems (FOKUS) develops
solutions for the communication infrastructure of the future. With more than
30 years of experience, FOKUS is one of the most important actors in the ICT
research landscape both nationally and worldwide. Fraunhofer FOKUS develops
innovative processes from the original concept up to the pre-product in
companies and institutions. As a member of important standardization bodies,
the institute contributes to the definition of new ICT standards. It
researches and develops application-orientated solutions for partners in
industry, research and public administration in various ICT fields.

Figure 5: The Corporate Smart Insights (CSI) concept.

Fraunhofer FOKUS has significant experience in semantic data intelligence and
AI, concentrated in the DANA group, which drives the research and the
development in the area of Corporate Semantic Web. Using this experience, the
aspect realized in the QURATOR project is an insight-driven AI approach
(Figure 5) which benefits the technological innovation of an Insight Driven
Organisation (IDO).555https://qurator.ai/partner/fraunhofer-fokus/ An IDO
embeds corporate knowledge, reasoning and smart insights learned from data
analytics into the daily decisions and actions of an organisation, including
their argumentation and interpretation.

The technical CSI framework consists of knowledge repositories for the
distributed management of knowledge artefacts, such as semantic knowledge
graphs and terminologies, a standardized API for knowledge bases [10], a
knowledge extraction and analytics666https://www.cyber-
akademie.de/anlage.jsp?id=959 service, and methods for corporate smart
insights knowledge evolution, as well as services to reuse the learned CSI
knowledge for AI, including inference, explainability and
plausibility/validation.

###  3.7 Semtation GmbH: Intelligent Business Process Modelling

Semtation GmbH provides the platform SemTalk (registered trademark) for
modeling and supporting business processes and knowledge structures, an easy
to use but very powerful modeling and portal tool based on Microsoft Visio and
the Microsoft Cloud. SemTalk technology makes use of various tools provided in
Microsoft 365 in order to offer best-in-class portal experiences when it comes
to supporting business processes.

In QURATOR, Semtation pursues the enhancement of business process model usage.
It consists of several tasks that aim in two directions, namely to 1) present
models on other devices but a monitor and 2) recommend information dynamically
based on the process context of the current user. Integrating various AI
technologies is necessary to obtain suitable results for both scenarios. It
helps to recognize your surroundings in order to recommend adequate
information in mixed reality settings and to understand natural language in a
chat scenario. It also makes it easier to understand the current process
context in order to recommend available documents and team members in various
settings based on text analysis and other machine learning use cases.
Semtation has already integrated knowledge graph information in process
portals in order to use available internal information to recommend suitable
documents and people based on the current process or project instance and the
current task.

The next step will be to define a scenario with one of the customers in order
to check requirements and results on a real world basis.

###  3.8 Stiftung Preußischer Kulturbesitz, Staatsbibliothek zu Berlin:
Automated Curation Technologies for Digitized Cultural Heritage

The Berlin State Library (Staatsbibliothek zu Berlin, SBB) is the largest
research library in Germany with more than 12 million documents in its
holdings and more than 2.5 PB of digital data stored throughout various
repositories (as of Oct. 2019). The collection encompasses texts, media and
cultural works from all fields in all languages, from all time periods and all
countries of the world.

Figure 6: Example image from a digitized collection

Within QURATOR, the Berlin State Library is taking part in the R&D activities
on behalf of the Prussian Heritage Foundation (Stiftung Preußischer
Kulturbesitz, SPK). SBB aims to digitize all its copyright-free historical
collections and to make them available on the
web777https://digital.staatsbibliothek-berlin.de as facsimile images with
high-quality text, logically structured and semantically annotated for use by
researchers. In order to achieve this goal, SBB works in a number of research
areas in the context of QURATOR – from layout and text recognition (OCR) and
unsupervised post-correction to named entity recognition (NER), disambiguation
and linking. Due to the huge volume and variety of the documents published
between 1475 and 1945, solutions are required that are particularly robust and
that can be fine-tuned to the complexities of historical fonts, layout,
language and orthography. While the SBB adopts state-of-the-art convolutional
neural networks (CNN) like ResNet50 [5] and UNet [18] in combination with
attention and adds rule-based domain adaptation for layout recognition and the
classification of structural elements, it follows a more classical RNN-LSTM-
CTC approach for text recognition [26], achieving character error rates below
1% with voting between multiple models trained on sufficiently large amounts
of historical document ground truth data [24]. In a related effort, SBB is
also taking part in the development of an open end-to-end framework for
historical document analysis and recognition based on AI methods [8]. For NER,
the recent transformer architecture BERT [4] is utilized and adapted to
historical German through a combination of unsupervised pre-training and
supervised learning [6]. The final goal is to identify and classify named
entities found in the digitized documents, and to disambiguate and link them
to an online knowledge base, e. g., Wikidata.

Eventually, the digitized historical documents shall be made fully searchable
with semantic markup enabling advanced content retrieval scenarios and rich
contextualization of documents with knowledge from third party sources. In a
further step, image-based classification methods will be added to enhance the
document metadata and to complement the functionalities offered through the
full-text search. In the course of 2020, a demonstrator will be launched in
SBB’s research and innovation lab.888https://lab.sbb.berlin

###  3.9 Ubermetrics Technologies GmbH: Curation Technologies for the
Monitoring of Online Content and Risks

Ubermetrics is a leading provider of cloud-based social media monitoring and
content intelligence software. Ubermetrics analyses public data from online,
print, TV and radio sources with a proprietary technology to identify critical
information in order to help organizations to optimize decision processes and
increase their performance (see Figure 7).

Figure 7: Analysis of mentions of natural disasters worldwide

In QURATOR, Ubermetrics researches how to use social media for the monitoring
of both external and internal risks. The focus areas are an easy setup of
risk-related search queries thanks to automated query suggestions and a
condensation of the results found with the help of summarization and duplicate
detection technology. The project aims at showing the developed capabilities
in demonstrators to get feedback for a later product for risk monitoring.

Automatic connections to risk related sources have already been developed and
a first version of query suggestions is available. The next steps are to
improve the query suggestions especially in the risk context and start with
the research and development of text summarization methods.

###  3.10 Wikimedia Deutschland e. V.: Data quality in Wikidata

Wikidata is Wikimedia’s knowledge base. It is a sister project of Wikipedia
and collects general purpose data about the world. Wikidata currently
describes over 63 million entities such as people, geographic entities, events
and works of art. Wikidata, just like Wikipedia, is built by a community –
currently consisting of more than 20,000 editors from all around the world –
that collects and maintains that data. Wikidata’s data powers a large number
of applications, among them search engine instant answers, digital personal
assistants, educational websites, as well as information boxes on Wikipedia
articles. By its nature, Wikidata is an open project. It relies on
contributions from volunteer editors. To build a large enough community for
building and maintaining a general purpose knowledge base the entry barrier
needs to be low. At the same time the pressure to provide high-quality data is
increasing as more and more people are exposed to its data in their day-to-day
life. It is vital for the long-term sustainability of Wikidata to find ways to
stay open while keeping the quality of its data high. On top of that Wikidata
can only follow its mission of giving more people more access to more
knowledge if the data is easily accessible for everyone. In QURATOR, we work
on improving both the quality and accessibility of the data in Wikidata, which
is supposed to become a viable basic building block of the QURATOR platform by
providing easily accessible high-quality data for all partners.

Figure 8: Schema for describing Kakapos in Wikidata

So far three important components with a focus on quality improvements have
been developed. The first presents a way to define schemas for data in order
to allow editors to quickly find data that does not conform to the specified
schema. It is based on the Shape Expression standard (see Figure 8). The
second entails the ability to automatically judge the quality of a data item
using machine learning. Editors can then find especially high and low quality
data items to showcase and improve them respectively. The third is an improved
way to add references for individual data points to improve the verifiability
of the data.

##  4 Summary and Next Steps

This paper provides a snapshot of the technologies and approaches developed as
part of the QURATOR project. Its vision is to offer a broad portfolio of
integrated solutions to the cross-industry challenges that are associated with
the curation of digital content. A platform strategy has been developed to
transform fragmented market areas for curation technologies into a new stand-
alone market and greatly expand it by displacing existing isolated solutions.
QURATOR aims to establish an ecosystem for curation technologies that improve
the state of the art and transform the Berlin-Brandenburg area into a global
center of excellence for curation technologies and the development of
efficient industrial applications.

## Acknowledgements

The research presented in this article is funded by the German Federal
Ministry of Education and Research (BMBF) through the project QURATOR
(Unternehmen Region, Wachstumskern, grant no. 03WKDA1A). http://qurator.ai

## References

  * [1] Berger, A.: Archive zum Sprechen Bringen – Semantic Storytelling oder der Redaktionsworkflow der Zukunft. In: 23. Berliner Veranstaltung der Internationalen EVA-Serie Electronic Media and Visual Arts. pp. 135–141 (2017)
  * [2] Bourgonje, P., Moreno-Schneider, J., Nehring, J., Rehm, G., Sasaki, F., Srivastava, A.: Towards a Platform for Curation Technologies: Enriching Text Collections with a Semantic-Web Layer. In: Sack, H., Rizzo, G., Steinmetz, N., MladeniÄ‡, D., Auer, S., Lange, C. (eds.) The Semantic Web. pp. 65–68. No. 9989 in Lecture Notes in Computer Science, Springer (June 2016), eSWC 2016 Satellite Events. Heraklion, Crete, Greece, May 29 – June 2, 2016 Revised Selected Papers
  * [3] Bourgonje, P., Schneider, J.M., Rehm, G., Sasaki, F.: Processing Document Collections to Automatically Extract Linked Data: Semantic Storytelling Technologies for Smart Curation Workflows. In: Gangemi, A., Gardent, C. (eds.) Proceedings of the 2nd International Workshop on Natural Language Generation and the Semantic Web (WebNLG 2016). pp. 13–16. The Association for Computational Linguistics, Edinburgh, UK (September 2016)
  * [4] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805 (2018)
  * [5] He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 770–778 (2016)
  * [6] Labusch, K., Neudecker, C., Zellhöfer, D.: BERT for Named Entity Recognition in Contemporary and Historic German. In: Preliminary Proceedings of the 15th Conference on Natural Language Processing (KONVENS 2019): Long Papers. pp. 1–9. German Society for Computational Linguistics & Language Technology, Erlangen, Germany (2019)
  * [7] Moreno-Schneider, J., Srivastava, A., Bourgonje, P., Wabnitz, D., Rehm, G.: Semantic Storytelling, Cross-lingual Event Detection and other Semantic Services for a Newsroom Content Curation Dashboard. In: Popescu, O., Strapparava, C. (eds.) Proc. of the Second Workshop on Natural Language Processing meets Journalism – EMNLP 2017 Workshop (NLPMJ 2017). pp. 68–73. Copenhagen, Denmark (2017)
  * [8] Neudecker, C., Baierer, K., Federbusch, M., Boenig, M., Würzner, K.M., Hartmann, V., Herrmann, E.: OCR-D: An End-to-end Open Source OCR Framework for Historical Printed Documents. In: Proceedings of the 3rd International Conference on Digital Access to Textual Cultural Heritage. pp. 53–58. DATeCH2019, ACM, New York, NY, USA (2019). https://doi.org/10.1145/3322905.3322917, http://doi.acm.org/10.1145/3322905.3322917
  * [9] Ostendorff, M., Bourgonje, P., Berger, M., Moreno-Schneider, J., Rehm, G.: Enriching BERT with Knowledge Graph Embeddings for Document Classification. In: Remus, S., Aly, R., Biemann, C. (eds.) Proceedings of the GermEval Workshop 2019 – Shared Task on the Hierarchical Classification of Blurbs. Erlangen, Germany (10 2019), 8 October 2019
  * [10] Paschke, A., Athan, T., Sottara, D., Kendall, E., Bell, R.: A Representational Analysis of the API4KP Metamodel. In: International Workshop Formal Ontologies Meet Industries. pp. 1–12. Springer (2015)
  * [11] Rehm, G., Berger, M., Elsholz, E., Hegele, S., Kintzel, F., Marheinecke, K., Piperidis, S., Deligiannis, M., Galanis, D., Gkirtzou, K., Labropoulou, P., Bontcheva, K., Jones, D., Roberts, I., Hajic, J., Hamrlova, J., Kacena, L., Choukri, K., Arranz, V., Mapelli, V., Vasiljevs, A., Anvari, O., Lagzdins, A., Melnika, J., Backfried, G., Dikici, E., Janosik, M., Prinz, K., Prinz, C., Stampler, S., Thomas-Aniola, D., Perez, J.M.G., Silva, A.G., Berrio, C., Germann, U., Renals, S., Klejch, O.: European Language Grid: An Overview (2020), submitted to LREC 2020. Marseille, France.
  * [12] Rehm, G., He, J., Schneider, J.M., Nehring, J., Quantz, J.: Designing User Interfaces for Curation Technologies. In: Yamamoto, S. (ed.) Human Interface and the Management of Information: Information, Knowledge and Interaction Design, 19th International Conference, HCI International 2017 (Vancouver, Canada). pp. 388–406. No. 10273 in Lecture Notes in Computer Science (LNCS), Springer, Cham, Switzerland (July 2017), part I
  * [13] Rehm, G., Lee, M., Schneider, J.M., Bourgonje, P.: Curation Technologies for a Cultural Heritage Archive: Analysing and Transforming a Heterogeneous Data Set into an Interactive Curation Workbench. In: Antonacopoulos, A., Bechler, M. (eds.) Proceedings of DATeCH 2019: Digital Access to Textual Cultural Heritage. Brussels, Belgium (May 2019), 8-10 May 2019. In print.
  * [14] Rehm, G., Sasaki, F.: Digitale Kuratierungstechnologien – Verfahren für die Effiziente Verarbeitung, Erstellung und Verteilung Qualitativ Hochwertiger Medieninhalte. In: Proceedings der Frühjahrstagung der Gesellschaft für Sprachtechnologie und Computerlinguistik (GSCL 2015). pp. 138–139. Duisburg (9 2015), 30. September–2. Oktober
  * [15] Rehm, G., Schneider, J.M., Bourgonje, P., Srivastava, A., Fricke, R., Thomsen, J., He, J., Quantz, J., Berger, A., König, L., Räuchle, S., Gerth, J., Wabnitz, D.: Different Types of Automated and Semi-Automated Semantic Storytelling: Curation Technologies for Different Sectors. In: Rehm, G., Declerck, T. (eds.) Language Technologies for the Challenges of the Digital Age: 27th International Conference, GSCL 2017, Berlin, Germany, September 13-14, 2017, Proceedings. pp. 232–247. No. 10713 in Lecture Notes in Artificial Intelligence (LNAI), Gesellschaft für Sprachtechnologie und Computerlinguistik e.V., Springer, Cham, Switzerland (January 2018), 13/14 September 2017.
  * [16] Rehm, G., Schneider, J.M., Bourgonje, P., Srivastava, A., Nehring, J., Berger, A., König, L., Räuchle, S., Gerth, J.: Event Detection and Semantic Storytelling: Generating a Travelogue from a large Collection of Personal Letters. In: Caselli, T., Miller, B., van Erp, M., Vossen, P., Palmer, M., Hovy, E., Mitamura, T. (eds.) Proc. of the Events and Stories in the News Workshop. pp. 42–51. Association for Computational Linguistics, Vancouver, Canada (August 2017)
  * [17] Rehm, G., Zaczynska, K., Schneider, J.M.: Semantic Storytelling: Towards Identifying Storylines in Large Amounts of Text Content. In: Jorge, A., Campos, R., Jatowt, A., Bhatia, S. (eds.) Proc. of Text2Story – Second Workshop on Narrative Extraction From Texts co-located with 41th European Conf. on Information Retrieval (ECIR 2019). pp. 63–70. Cologne, Germany (April 2019), 14 April 2019
  * [18] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional Networks for Biomedical Image Segmentation. In: International Conference on Medical Image Computing and Computer-assisted Intervention. pp. 234–241. Springer (2015)
  * [19] Schneider, J.M., Bourgonje, P., Nehring, J., Rehm, G., Sasaki, F., Srivastava, A.: Towards Semantic Story Telling with Digital Curation Technologies. In: Birnbaum, L., Popescu, O., Strapparava, C. (eds.) Proceedings of Natural Language Processing Meets Journalism – IJCAI-16 Workshop (NLPMJ 2016). New York (July 2016)
  * [20] Schneider, J.M., Bourgonje, P., Rehm, G.: Towards User Interfaces for Semantic Storytelling. In: Yamamoto, S. (ed.) Human Interface and the Management of Information: Information, Knowledge and Interaction Design, 19th Int. Conf., HCI International 2017 (Vancouver, Canada). pp. 403–421. No. 10274 in Lecture Notes in Computer Science (LNCS), Springer, Cham, Switzerland (July 2017), part II
  * [21] Schneider, J.M., Rehm, G.: Curation Technologies for the Construction and Utilisation of Legal Knowledge Graphs. In: Rehm, G., Rodriguez-Doncel, V., Schneider, J.M. (eds.) Proc. of the LREC 2018 Workshop on Language Resources and Technologies for the Legal Knowledge Graph. pp. 23–29. Miyazaki, Japan (May 2018)
  * [22] Schneider, J.M., Rehm, G.: Towards a Workflow Manager for Curation Technologies in the Legal Domain. In: Rehm, G., Rodriguez-Doncel, V., Schneider, J.M. (eds.) Proc. of the LREC 2018 Workshop on Language Resources and Technologies for the Legal Knowledge Graph. pp. 30–35. Miyazaki, Japan (May 2018)
  * [23] Schneider, J.M., Roller, R., Bourgonje, P., Hegele, S., Rehm, G.: Towards the Automatic Classification of Offensive Language and Related Phenomena in German Tweets. In: Ruppenhofer, J., Siegel, M., Wiegand, M. (eds.) Proceedings of the GermEval Workshop 2018 – Shared Task on the Identification of Offensive Language. pp. 95–103. Vienna, Austria (September 2018), 21 September 2018
  * [24] Springmann, U., Reul, C., Dipper, S., Baiter, J.: Ground Truth for Training OCR Engines on Historical Documents in German Fraktur and Early Modern Latin. arXiv preprint arXiv:1809.05501 (2018)
  * [25] Srivastava, A., Sasaki, F., Bourgonje, P., Moreno-Schneider, J., Nehring, J., Rehm, G.: How to Configure Statistical Machine Translation with Linked Open Data Resources. In: Esteves-Ferreira, J., Macan, J., Mitkov, R., Stefanov, O.M. (eds.) Proceedings of Translating and the Computer 38 (TC38). pp. 138–148. Editions Tradulex, London, UK (November 2016), http://www.asling.org/tc38/
  * [26] Wick, C., Reul, C., Puppe, F.: Calamari-A High-Performance Tensorflow-based Deep Learning Package for Optical Character Recognition. arXiv preprint arXiv:1807.02004 (2018)


# Non-Cross Diffusion for Semantic Consistency

Ziyang Zheng∗ Ruiyuan Gao∗ Qiang Xu
The Chinese University of Hong Kong
<EMAIL_ADDRESS>
###### Abstract

00footnotetext: ∗Equal contribution.

In diffusion models, deviations from a straight generative flow are a common
issue, resulting in semantic inconsistencies and suboptimal generations. To
address this challenge, we introduce ‘Non-Cross Diffusion’, an innovative
approach in generative modeling for learning ordinary differential equation
(ODE) models. Our methodology strategically incorporates an ascending
dimension of input to effectively connect points sampled from two
distributions with uncrossed paths. This design is pivotal in ensuring
enhanced semantic consistency throughout the inference process, which is
especially critical for applications reliant on consistent generative flows,
including various distillation methods and deterministic sampling, which are
fundamental in image editing and interpolation tasks.

Our empirical results demonstrate the effectiveness of Non-Cross Diffusion,
showing a substantial reduction in semantic inconsistencies at different
inference steps and a notable enhancement in the overall performance of
diffusion models.

Figure 1:  Illustrating xFlow in Diffusion Models. (a) Demonstrates the
ambiguity in training targets caused by crossing flows, leading to the xFlow
problem. (b) Shows how our method eliminates flow crossing by increasing the
dimensionality of network inputs, thus resolving the xFlow problem. (c)
Depicts how xFlow leads to variable sampling results across different steps,
undermining deterministic sampling. (d) Top: Highlights the discrepancies
between outcomes from reduced steps sampling (in red) versus standard results
(from 1000 steps in blue) due to xFlow. Bottom: Our method ensures consistent
outputs across different sampling steps. (e) Top: Exhibits instances where
xFlow causes Out-Of-Distribution (OOD) outcomes in reduced steps sampling (in
red) compared to standard results (from 1000 steps in blue). Bottom: Our
approach minimizes the occurrence of OOD samples.

##  1 Introduction

Diffusion models, as delineated in recent studies [23, 11, 4, 17, 19, 24, 25],
have exhibited remarkable capabilities in image synthesis, bolstering numerous
applications such as text-to-image generation [16, 21], image editing [1, 16,
2, 26], and image inpainting [1, 18]. A key characteristic of these models is
their multi-step generative process, which not only allows for correction of
the diffusion path [25] but also enhances controllability [5, 7].

Despite these advancements, the inference process in diffusion models
typically involves a specific flow, whereas the training process entails
random step selections from multiple flows. This randomness often results in a
given training step correlating with diverse flows, creating ambiguity in
target identification from the network’s perspective, as depicted in Fig.
1(a). We term this phenomenon ‘xFlow’.

xFlow’s emergence during training can hinder the model’s optimization at
certain steps, leading to a spectrum of generative issues. Notably, it
challenges the model’s ability to generate samples via a straight flow,
compromising deterministic sampling across varying step counts, as shown in
Fig. 1(c). It also complicates predicting later sampling steps from earlier
ones, limiting the effectiveness of reward models [30] and guided models [4].
Moreover, in the context of distillation, which typically adopts a progressive
approach, xFlow can introduce misleading signals, as evidenced in Rectified
Flow [14]. Perhaps most critically, xFlow can lead to the generation of Out-
Of-Distribution (OOD) samples or low-quality samples, especially as sampling
step size increases, as illustrated in Fig. 1(d-e).

In this paper, we propose a novel training strategy aimed at resolving the
xFlow challenge in diffusion models. Our method centers on augmenting the
input dimensionality to these models, a change that effectively prevents flow
crossing. As depicted in Fig. 1(a), the issue at hand arises when two flows
intersect, creating ambiguity; the input to the network (for instance, a noisy
image) remains constant, yet it is associated with multiple potential targets
(such as distinct noises originating from different images). To address this,
our approach entails predicting the flow itself during the training phase, as
shown in Fig. 1(b). Notably, we utilize the noise predicted by the network as
the flow’s endpoint, incorporating this element into the model’s input. This
technique sidesteps the pitfall of using groundtruth noise as input, which
would otherwise result in trivial training solutions devoid of substantive
learning. For practical implementation, we found ControlNet [31] particularly
effective in this context. Additionally, our methodology integrates a
bootstrap approach reminiscent of Analog bits [3], which significantly
enhances our model’s optimization and effectively narrows the gap between
training and inference phases.”

To evaluate our approach, we introduce the Inference Flow Consistency
($\operatorname{IFC}$) metric, reflecting xFlow severity. We also utilize
Inception Score (IS) [22] and Fréchet Inception Distance (FID) [9] for
assessing generation quality. Our models, trained from scratch and compared
against baselines on Cifar-10 [13], demonstrate not only an avoidance of xFlow
but also an enhancement in generation quality. The contributions of this paper
include:

  * •

We identify a widespread phenomenon in diffusion models, termed xFlow, leading
to non-straight flow during inference that may generate OOD or suboptimal
samples.

  * •

We attribute xFlow’s origins to the instability of the target during the
training process. Accordingly, we introduce the Non-Cross Diffusion, a novel
training and inference pipeline to mitigate the xFlow problem by enhancing
input dimensionality.

  * •

Our experiments on both a toy model and the Cifar-10 dataset demonstrate that
our method not only improves the proposed $\operatorname{IFC}$ metric by
addressing xFlow, but also significantly enhances other image evaluation
metrics, such as IS and FID.

##  2 Related work

###  2.1 Diffusion models

Diffusion models, as generative models, learn the reverse denoising process
from Gaussian noise to image distribution, achieved through either Markov [11]
or non-Markov operations [23]. They are favored over other generative models
like GAN [8] and VAE [27] due to their training stability and superior
generation quality. Subsequent enhancements to these models primarily concern
varied network architectures [12], noise schedulers or losses [17], transition
from image space diffusion to latent space [19], and improved sampling
techniques [15], with little attention to the xFlow during training. Rectified
Flow [14] noted the mismatch in sampling across different inference steps, a
significant distillation issue, but did not analyze it further. Instead, they
proposed a workaround using a 2-rectified flow to fit another model to a non-
crossing flow between source and target distributions, which depends on a
well-trained diffusion model and requires additional retraining. Our paper is
the first to examine xFlow in diffusion and offer solutions.

###  2.2 Conditional Image Generation

Conditioning techniques are instrumental in managing generated content [19, 6,
29, 28]. For diffusion models, [25] proposes classifier guidance, which is an
efficient method to balance controllability and fidelity using the gradients
from a classifier, while classifier-free guidance [10], being another
important conditioning technique to diffusion models, trains both conditional
and unconditional diffusion models, and combining their score to achieve
better controllability. ControlNet [31] employs pretrained encoding layers
from billions of images as a backbone to learn diverse conditional controls,
which is an architecture adopted in this paper. Analog Bits [3] introduces a
technique that conditions the model on its own previously generated samples
during iterative sampling, akin to our work. However, Analog Bits mainly aims
to enhance sample quality by reusing the previous target, while our focus is
to introduce a condition in the training flow to prevent crossing issues.

Figure 2: The overview of non-cross diffusion. Training stage: The training
phase involves two cases. In Case1, we utilize $\mathbf{0}$ as the condition
and calculate loss function $L_{simple}$ as defined in Eq. 5. For Case2, we
first compute $\hat{\epsilon}$ using $\mathbf{0}$ as condition. Subsequently,
$\hat{\epsilon}$ is employed as the condition to calculate $L_{simple}$.
Throughout the training process, Case1 is applied with a fixed probability
$p$; otherwise, Case2 is implemented. Inference stage: During the inference
phase, 0 is used as the condition in the initial denoising step. This is
followed by iterative utilization of the estimated noise from the previous
step as the condition for subsequent steps.

##  3 Method

In this section, we start with a brief review of the formulation of DDPM [11].
Next, we show the drawback of baseline flow and then introduce Non-Cross
Diffusion to avoid crossing by ascending dimension of input. At last, we
present our training and inference flow of Non-Cross Diffusion.

###  3.1 Preliminary

Given a data distribution $x_{0}\sim q(x_{0})$, we define a forward noising
process $q$ which produces latent ${x_{1},\ldots,x_{T}}$ by adding Gaussian
noise at time with variance $\beta_{t}\in(0,1)$ as follows:

$\displaystyle q(x_{1},\ldots,x_{T})\coloneqq\prod_{t=1}^{T}q(x_{t}|x_{t-1}),$
(1) $\displaystyle
q(x_{t}|x_{t-1})\coloneqq\mathcal{N}(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}\mathbf{I}).$
(2)

With $\alpha_{t}=1-\beta_{t}$ and
$\bar{\alpha}_{t}=\prod_{s=0}^{t}\alpha_{s}$, we write the marginal
$q(x_{t}|x_{0})$ through Eq. 2 as follows:

$\displaystyle
q(x_{t}|x_{0})\coloneqq\mathcal{N}(x_{t},\sqrt{\bar{\alpha}_{t}}x_{0},(1-\bar{\alpha}_{t})\mathbf{I}),$
(3) $\displaystyle
x_{t}=\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon,$ (4)

where $\epsilon\sim\mathcal{N}(0,\mathbf{I})$. Using Bayes theorem, we can
calculate the posterior $q(x_{t}|x_{t},x_{0})$ in terms of $\tilde{\beta}$ and
$\tilde{\mu}_{t}(x_{t},x_{0})$. There are many different ways to parameterize
$\mu_{\theta}(x_{t},t)$. DDPM [11] found that predicting $\epsilon$ worked
best with a loss function:

$\displaystyle
L_{simple}=E_{t,x_{0},\epsilon}[\|\epsilon-\epsilon_{\theta}(x_{t},t)\|^{2}]$
(5)

###  3.2 Understanding Drawbacks of DDPM Flow

Training stage: Given source distribution $\pi_{0}$ and target distribution
$\pi_{1}$, we sample two data pairs
$(x_{0},x_{T}),(y_{0},y_{T})\sim\pi_{0}\times\pi_{1}$. During the training
stage, assume that these two training flows cross at time step $t$, which
means:

$\displaystyle
x_{t}=\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon_{x},$ (6)
$\displaystyle
y_{t}=\sqrt{\bar{\alpha}_{t}}y_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon_{y},$ (7)
$\displaystyle x_{t}=y_{t}.$ (8)

In the cross point, the two training flow aims to minimize the loss function
as follows:

$\displaystyle\min_{\theta}E_{t,x_{0},\epsilon_{x}}[\|\epsilon_{x}-\epsilon_{\theta}(x_{t},t)\|^{2}],$
(9)
$\displaystyle\min_{\theta}E_{t,y_{0},\epsilon_{y}}[\|\epsilon_{y}-\epsilon_{\theta}(y_{t},t)\|^{2}].$
(10)

Since $x_{t}=y_{t}$, we can rewrite the above target as follows:

$\displaystyle\min_{\theta}E_{t,z_{t},\epsilon_{x},\epsilon_{y}}[\|\frac{(\epsilon_{x}+\epsilon_{y})}{2}-\epsilon_{\theta}(z_{t},t)\|^{2}],$
(11)

where $z_{t}=x_{t}=y_{t}$. This implies that for the cross point, the model is
given an incorrect target and will lead to the generation of an OOD sample.

Inference stage. As illustrated in Fig. 1, the decision to change direction
during the inference process is contingent on the timestep of inference.
Specifically, a larger inference step correlates with a smaller stride,
resulting in only a slight influence from the cross point on the inference
flow, typically insufficient to alter its direction. Conversely, a smaller
inference step is associated with a larger stride, significantly impacting the
inference flow due to the cross point and consequently altering its direction.
This alteration can lead to the generation of inconsistent or OOD samples.
Such a phenomenon underscores the dependence of the generated image on the
inference step, potentially reducing the determinism of the diffusion model.

###  3.3 Selection of Condition

In this section, we delve deeper into our proposed method, which focuses on
avoiding the intersection of training flows in a unique and effective manner.
The core principle behind this method is derived from a basic geometrical
concept: any two distinct lines in a plane can intersect at most once. By
leveraging this idea, we aim to ensure that different training flows do not
cross each other, thereby maintaining the integrity and distinctiveness of
each flow.

To operationalize this concept, our approach involves the use of concatenation
of different points in the training flow. Specifically, by concatenating
$x_{t}$ with $x_{i}$, where $i\neq t$, we ensure that two different training
flows will not cross. Practically, we sample two data pairs
$(x_{0},x_{T}),(y_{0},y_{T})\sim\pi_{0}\times\pi_{1}$, and assume that
$x_{t}\neq y_{t}$. Then we have $[x_{i},x_{t}]\neq[y_{i},y_{t}],\forall
i\in[0,T]$, which denotes that by ascending dimensions with the noised image
in timestep $t$, two training flows will no longer intersect. This strategy
effectively creates a multidimensional space where the likelihood of training
flows intersecting is significantly reduced.

Given the continuity inherent in diffusion models, we observe that the
effectiveness of avoiding crossing points improves when the selected points
are farther from these intersections. Thus, we propose to use either initial
noise $x_{T}$,_i.e_. $\epsilon$, or target image $x_{0}$ for dimension
ascension.

Furthermore, we find the distance between randomly sampled noise is stable
while the distance between image or latent is unstable, which is mainly
decided by background or semantic information. Practically, for two randomly
sampled noise $n_{1},n_{2}\in\mathbb{R}^{H\times W\times C}$, we have
$E[\|n_{1}-n_{2}\|^{2}]=2CHW$. Besides, in the inference stage, we can get the
initial noise, but we cannot get the target image. Therefore, we propose to
use initial noise $x_{T}$ instead of target image $x_{0}$ as the condition for
dimension ascending.

###  3.4 Proposed Solution

In this section, we delineate the training and inference stages of our Non-
Cross Diffusion, emphasizing strategies to enhance model performance and
reliability.

Training Stage. The cornerstone of our training strategy is to circumvent
trivial solutions and avoid training collapse. To achieve this, we replace the
use of initial noise $\epsilon$ with predicted noise $\hat{\epsilon}$. This
substitution is critical in refining our model’s predictive accuracy since it
can effectively avoid trivial solutions. Furthermore, we introduce a bootstrap
strategy in the training stage, which adds robustness to the learning process
and avoids misleading of the predicted noise $\hat{\epsilon}$ in the early
stage.

As illustrated in Fig. 2, our training objective is formulated as follows:

$\displaystyle\min_{\theta}E_{t,x_{t},\epsilon,}[\|\epsilon-\epsilon_{\theta}(x_{t},\hat{\epsilon}_{t},t)\|^{2}],$
(12)

where $x_{t}=\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon$.
During training, to mitigate the risk of being misled by estimated noise, we
set $\hat{\epsilon}_{t}=\mathbf{0}$, _i.e_. Case1 in Fig. 2, with a fixed
probability $p$. At other times,$\hat{\epsilon}_{t}$ is assigned the value of
$\epsilon_{\theta}(x_{t},\mathbf{0},t)$,_i.e_. Case2 in Fig. 2, and we do not
backpropagate through estimated noise $\hat{\epsilon}_{t}$.

Inference Stage. As illustrated in Fig. 2, during the inference stage, to
alleviate the computational costs, we use estimated noise in the previous step
instead of the current step as the condition and iteratively predict
$\hat{\epsilon}$ as follows:

$\displaystyle\hat{\epsilon}_{T}=\epsilon_{\theta}(\hat{x}_{T},\mathbf{0},T),$
(13)
$\displaystyle\hat{\epsilon}_{t}=\epsilon_{\theta}(\hat{x}_{t},\hat{\epsilon}_{t+1},t),t<T.$
(14)

When inference step is large, the discrepancy between $\hat{\epsilon}_{t}$ and
$\hat{\epsilon}_{t+1}$ is small, which ensure the performance of our method.

Network Architecture. To effectively utilize $\hat{\epsilon}_{t}$, inspired by
ControlNet [31], Non-Cross Diffusion leverage an addictive U-net branch that
takes as input $\hat{\epsilon}_{t}$. Besides, we introduce some modifications
for optimization. Specifically, we eliminate all zero convolution layers and
create a trainable copy of the U-net encoder to encode the condition
$\hat{\epsilon}_{t}$. Subsequently, the output from this trainable copy is
integrated into the U-net decoder through an addition operation. During the
training stage, this entire network is trained from scratch instead of
finetuning.

Figure 3: Results of the Toy Model. (a) Comparison of Generated Distributions:
This panel illustrates the distributions generated by the baseline model and
our proposed model. As the number of inference steps decreases, the baseline
model tends to produce a significant number of out-of-distribution (OOD)
samples. In contrast, our model effectively mitigates the generation of OOD
samples. (b) Trajectory Analysis: This panel compares the generated
trajectories of the baseline and our models. The baseline model’s inference
flow often redirects at the intersection point, leading to a target OOD
distribution as the inference steps decrease. Our method, however, maintains a
consistent direction in the inference model, thereby straightening the
trajectory.

Inference Flow Consistency. To better evaluate the consistency of the
inference flow, we propose a metric by computing the similarity between
intermediate generated image $\hat{x}_{0}^{t}$ in timestep $t$ and the final
generated image $\hat{x}_{0}$ based on Peak signal-to-noise ratio(PSNR) as
follows:

$\displaystyle\operatorname{IFC}=\frac{1}{T}\sum_{t=0}^{T}PSNR(\hat{x}_{0}^{t},\hat{x}_{0}).$
(15)

Setting | Method | IS | FID
---|---|---|---
DDIM-1000 | iDDPM | 9.02 | 4.70
Ours | 9.51 | 2.88
Ours† | 9.34 | 3.40
DDIM-100 | iDDPM | 8.99 | 5.65
Ours | 9.22 | 3.93
Ours† | 9.15 | 4.21
DDIM-50 | iDDPM | 8.89 | 6.61
Ours | 9.10 | 5.31
Ours† | 9.05 | 5.09
DDIM-20 | iDDPM | 8.59 | 9.82
Ours | 8.77 | 9.87
Ours† | 8.84 | 7.75
DDIM-10 | iDDPM | 8.20 | 15.91
Ours | 7.97 | 20.63
Ours† | 8.50 | 12.85
DDIM-5 | iDDPM | 7.09 | 31.37
Ours | 6.20 | 50.25
Ours† | 7.45 | 27.83
Table 1: We compare the performance of baseline and our method. We generate
50k samples using DDIM with inference steps in {1000, 100, 50, 20, 10, 5}.
Ours† use inference strategy similar with training stage. For $\hat{x}_{t}$
,we first give $\mathbf{0}$ as condition and get estimated noise
$\hat{\epsilon_{t}}$, then we take $\hat{\epsilon_{t}}$ as condition and
compute denoised image $\hat{x}_{t-1}$.

When the training flow changes direction at a particular timestep, it results
in a marked semantic difference between the images generated before and after
that timestep, leading to a lower PSNR. By quantifying this using our PSNR-
based metric, we can effectively assess and ensure the consistency of the
inference process across different stages.

##  4 Experiment

###  4.1 Toy Examples

In this section, we follow the setting in Rectified Flow [14]. We start with
drawing training dataset $(X_{0},X_{1})\sim\pi_{0}\times\pi_{1}$. Given a
sample $\\{x_{0}^{i},x_{1}^{i}\\}$ drawn from $(X_{0},X_{1})$,for baseline
model, we train a 3-layer MLP $v_{\theta}(z,t)$ to transfer source
distribution $\pi_{0}$ to target distribution $\pi_{1}$ with l2-loss as
follow:

$\displaystyle\min_{\theta}\|v_{\theta}(x_{t}^{i},t)-(x_{1}^{i}-x_{0}^{i})\|_{2},$
$\displaystyle x_{t}^{i}=tx_{1}^{i}+(1-t)x_{0}^{i},t\in[0,1).$

Our method enhances this approach by incorporating an additional dimension
with the estimated target as follows:

$\displaystyle\min_{\theta}\|v_{\theta}([x_{t}^{i},\hat{c}_{t}^{i}],t)-(x_{1}^{i}-x_{0}^{i})\|_{2},$
$\displaystyle\hat{c}_{t}^{i}=\begin{cases}\mathbf{0}&\text{p$\leq$0.5}\\\
v_{\theta}([x_{t}^{i},\mathbf{0}],t)&\text{otherwise}\end{cases}$

with $p$ sampled from a uniform distribution $U(0,1)$. The inference process
is also similar to our proposed method, _i.e_., we use the estimated target in
the previous step as the condition.

###  4.2 Comparison Experiment

(a) Baseline

(b) Ours

Figure 4: Here are the generated images using DDIM with inference steps in {5,
10, 20, 50, 100, 1000}. For baseline method, the semantic information of image
with small inference step and large inference step could be greatly different,
which implies that the inference flow changes its direction at some timesteps.

As depicted in Fig. 3, the inference flow of the baseline model changes
direction at the cross point of the two flows. This redirection results from
an incorrect loss function, as illustrated in Eq. 11, leading to the
generation of OOD samples. Our method, by using the estimated result to
introduce an additional dimension, avoids the crossing of training and
inference flows. This ensures a consistent direction in the inference flow and
effectively prevents the generation of OOD samples.

Implementation Details. We train our models of $32\times 32$ images on
Cifar-10 [13]. We evaluate the fidelity of generated samples using IS [22] and
FID [9] and the consistency of inference flow using $\operatorname{IFC}$. For
model on Cifar-10, we refer to the setting of iDDPM [17] except that we use
$L_{simple}$ only and we set the training step to 250k. For baseline model, we
train iDDPM[17] from scratch following the original setting except that we set
the training step the same as ours, _i.e_. 250k.

Comparison of Sampling Quality. In Table 1, we present a comparison of our
model with a baseline model in generating Cifar-10 images. Our model
demonstrates superior performance, showing significant improvements over the
baseline, particularly at inference steps in {1000, 100, 50}. However, as the
inference steps decrease and stride increases, the difference in estimated
noise between successive steps grows. This discrepancy introduces a bias
during the inference stage, leading to a decline in performance. To mitigate
this bias, we propose using the estimated noise from the current step as a
condition. This approach enables our method to enhance the quality of
generated images even at smaller inference steps in {20, 10, 5}.

(a) Baseline

(b) Ours

Figure 5: Displayed are the generated images of baseline model and our model
using DDIM with inference step of 1000. The results demonstrate our model’s
superior image generation capabilities, significantly reducing the occurrence
of OOD samples.

Comparison of Inference Consistency. Table 2 compares the consistency of our
method with the baseline. Our approach demonstrates improved consistency at
inference steps in {50, 20, 10, 5}, achieved by avoiding crossing in the
inference flow. For larger steps in {1000, 100}, cross points in the inference
flow have only a minor impact due to the larger steps and smaller stride.
Consequently, our method and the baseline perform similarly in terms of
$\operatorname{IFC}$.

Setting | Method | $\operatorname{IFC}$
---|---|---
DDIM-1000 | iDDPM | 28.58
Ours | 28.37
DDIM-100 | iDDPM | 29.72
Ours | 29.96
DDIM-50 | iDDPM | 30.85
Ours | 31.40
DDIM-20 | iDDPM | 33.94
Ours | 35.11
DDIM-10 | iDDPM | 38.51
Ours | 40.21
DDIM-5 | iDDPM | 46.11
Ours | 47.96
Table 2: We compare the consistency of baseline and our method. We generate
1000 samples using DDIM with inference steps in {1000, 100, 50, 20, 10, 5}.

###  4.3 Ablation Study

| Setting | IS | FID
---|---|---|---
Bootstrap | w/o bootstrap | 5.93 | 79.70
exp-schedule | 9.22 | 5.24
fix-prob(Ours) | 9.38 | 4.84
Condition |  $x_{0}$ condition | 9.64 | 20.21
mid condition | 7.15 | 59.09
$\hat{\epsilon}$ condition(Ours) | 9.38 | 4.84
Network | Double Unet | 9.65 | 6.26
ControlNet-based(Ours) | 9.38 | 4.84
Inference strategy | zero condition | 9.06 | 6.43
${\epsilon}$ condition | 7.86 | 27.00
$\hat{\epsilon}$ condition(Ours) | 9.38 | 4.84
Table 3: Ablation study. We generate 10k samples using DDIM with 1000
inference step.

In this section, we verify the effectiveness of each component through
ablation studies, summarized in Table 3.

Ablation of bootstrap. The impact of bootstrap strategy is evident from Table
3. We conducted experiments under three settings: without bootstrap (w/o
bootstrap), exponential schedule (exp-schedule), and fixed probability (fix-
probability). In the w/o bootstrap setting,
$\hat{\epsilon}t=\epsilon\theta(x_{t},\mathbf{0},t)$ is consistently used. For
exp-schedule, the probability of using this setting increases exponentially.
The results indicate that the absence of a bootstrap strategy can
significantly diminish model performance.

Ablation of condition. We then examine how different conditions affect the
model. $x_{0}$ condition denotes that the model uses the predicted image as
condition. Mid condition denotes that the model uses the midpoint of the
training flow, _i.e_. $\frac{\hat{x}_{0}+\hat{\epsilon}}{2}$ as the condition.
During the inference stage, each method utilizes its corresponding condition
as the condition. As discussed in Sec. 3.3, the midpoint shows the poorest
performance, likely because it is closest to the training flows cross point.
Due to the stability of $\epsilon$, using $\hat{\epsilon}$ as a condition
leads to a significant improvement of FID compared with $x_{0}$ condition.

Ablation of architecture. The study in Table 3 also investigates network
effectiveness. We experimented with two different architectures, namely Double
Unet and ControlNet-based. The Double Unet simply expands the input channel of
the Unet model, and concatenates the input and condition along the channel.
The ControlNet-based model implements conditioning similar to ControlNet [31].
Results indicate that applying ControNet can improve FID by 1.42.

(a) unconditional

(b) class condition

(c) canny condition

Figure 6: Here are the generated images with different conditions. (a): We
generate images on Cifar-10 using iDDPM with inference step in {5, 10, 20, 50,
100, 200, 500, 1000}. (b): we generate images on ImageNet [20] using LDM [19]
with inference step in {5, 10, 20, 50, 100, 500}. (c): We generate images with
the canny condition using Stable Diffusion [19] with inference step in {5, 10,
20, 50, 100, 500}. We sample all images with DDIM [23].

Ablation of inference strategy. Finally, we compared different inference
strategies. After training a model following Sec. 3.4, we applied various
conditions during inference. The Zero condition always uses $\mathbf{0}$ as
the condition. The $\epsilon$ condition uses the initial noise $\epsilon$, and
the $\hat{\epsilon}$ condition utilizes the estimated noise $\hat{\epsilon}$.
Due to the discrepancy between the initial noise $\epsilon$ and the estimated
noise $\hat{\epsilon}$, the $\epsilon$ condition significantly reduced
performance. While the Zero condition avoids ambiguity in the training stage,
it leads to a slight performance decline during inference as the inference
flow cannot bypass crossing and may redirect at the cross point.

###  4.4 Discussion

This section offers an alternative perspective to understand our Non-Cross
Diffusion. The inclusion of $\hat{\epsilon}_{t}$ in the model input is seen as
a strong conditional constraint, which we propose can reduce the variability
of semantic information, thus lessening the severity of xFlow during
inference.

Specifically, the $\hat{\epsilon}_{t}$ condition in Non-Cross Diffusion is
highly specific at the pixel level, making it an exceptionally stringent
constraint. Consequently, Non-Cross Diffusion effectively mitigates the xFlow
issue. An interesting question arises: how would xFlow be influenced by other
forms of conditions with varying degrees of strength? To investigate this, we
employed the pre-trained ControlNet model for empirical analysis and results.

Fig. 6 shows generated unconditional images from iDDPM, class conditional
images from ImageNet LDM [19], and Canny conditional images with ControlNet
[31]. In Fig. 6(a), unconditional generation on Cifar-10 at steps {5, 10, 20,
50, 100, 200, 500, 1000} presents a significant shift in the semantic content
of the images at each step: from airplane to automobile class. In contrast,
Fig. 6(b) and 6(c) under the tench class and Canny edge conditions,
respectively, demonstrate more consistent semantic content across steps,
despite some variability in details such as the background and tail.
Therefore, we hypothesize that stronger conditions ease the severity of xFlow.

##  5 Conclusion

In this study, we have addressed the xFlow phenomenon in diffusion models,
characterized by deviations in generative flow that result in semantic
inconsistencies and suboptimal image generation. Our novel approach, ‘Non-
Cross Diffusion’, innovates in the realm of generative modeling by adopting
ordinary differential equation models. Our empirical investigations, including
both a toy example and the Cifar-10 image dataset, demonstrate the substantial
efficacy of the Non-Cross Diffusion approach. The results show a marked
reduction in semantic inconsistencies at various inference stages and
significant improvements in the overall performance of diffusion models.

Looking Ahead. The identification of xFlow as a critical issue during
inference opens new avenues for research and application optimization. Despite
the effectiveness of the proposed Non-Cross Diffusion approach on mitigating
xFlow, we acknowledge the challenges associated with retraining large-scale
diffusion models such as Stable Diffusion. However, we are optimistic that
future research will find ways to integrate these improvements into existing
models, potentially circumventing the need for extensive retraining. This
paper lays the groundwork for such advancements, aiming to enhance the
reliability and quality of diffusion model outputs.

## References

  * Avrahami et al. [2022] Omri Avrahami, Dani Lischinski, and Ohad Fried.  Blended diffusion for text-driven editing of natural images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 18208–18218, 2022.
  * Brooks et al. [2023] Tim Brooks, Aleksander Holynski, and Alexei A. Efros.  Instructpix2pix: Learning to follow image editing instructions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 18392–18402, 2023.
  * Chen et al. [2022] Ting Chen, Ruixiang Zhang, and Geoffrey Hinton.  Analog bits: Generating discrete data using diffusion models with self-conditioning.  _arXiv preprint arXiv:2208.04202_ , 2022.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ , 34:8780–8794, 2021.
  * Fan and Lee [2023] Ying Fan and Kangwook Lee.  Optimizing ddpm sampling with shortcut fine-tuning.  _arXiv preprint arXiv:2301.13362_ , 2023.
  * Gao et al. [2023a] Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhenguo Li, Dit-Yan Yeung, and Qiang Xu.  Magicdrive: Street view generation with diverse 3d geometry control.  _arXiv preprint arXiv:2310.02601_ , 2023a.
  * Gao et al. [2023b] Ruiyuan Gao, Chenchen Zhao, Lanqing Hong, and Qiang Xu.  Diffguard: Semantic mismatch-guided out-of-distribution detection using pre-trained diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1579–1589, 2023b.
  * Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  _Advances in neural information processing systems_ , 27, 2014.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30, 2017.
  * Ho and Salimans [2022] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Karras et al. [2022] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  _Advances in Neural Information Processing Systems_ , 35:26565–26577, 2022.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  _Tech Report_ , 2009.
  * Liu et al. [2022] Xingchao Liu, Chengyue Gong, and Qiang Liu.  Flow straight and fast: Learning to generate and transfer data with rectified flow.  _arXiv preprint arXiv:2209.03003_ , 2022.
  * Lu et al. [2022] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.  _Advances in Neural Information Processing Systems_ , 35:5775–5787, 2022.
  * Nichol et al. [2022] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models, 2022.
  * Nichol and Dhariwal [2021] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In _International Conference on Machine Learning_ , pages 8162–8171. PMLR, 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents, 2022.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  _International Journal of Computer Vision (IJCV)_ , 115(3):211–252, 2015.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding.  In _Advances in Neural Information Processing Systems_ , pages 36479–36494. Curran Associates, Inc., 2022.
  * Salimans et al. [2016] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  _Advances in neural information processing systems_ , 29, 2016.
  * Song et al. [2020a] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020a.
  * Song and Ermon [2019] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  _Advances in neural information processing systems_ , 32, 2019.
  * Song et al. [2020b] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020b.
  * Tumanyan et al. [2023] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel.  Plug-and-play diffusion features for text-driven image-to-image translation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1921–1930, 2023.
  * Vahdat and Kautz [2020] Arash Vahdat and Jan Kautz.  Nvae: A deep hierarchical variational autoencoder.  _Advances in neural information processing systems_ , 33:19667–19679, 2020.
  * Yang et al. [2022a] Yijun Yang, Ruiyuan Gao, Yu Li, Qiuxia Lai, and Qiang Xu.  What you see is not what the network infers: detecting adversarial examples based on semantic contradiction.  _arXiv preprint arXiv:2201.09650_ , 2022a.
  * Yang et al. [2022b] Yijun Yang, Ruiyuan Gao, and Qiang Xu.  Out-of-distribution detection with semantic mismatch under masking.  In _European Conference on Computer Vision_ , pages 373–390. Springer, 2022b.
  * Yoon et al. [2023] TaeHo Yoon, Kibeom Myoung, Keon Lee, Jaewoong Cho, Albert No, and Ernest K Ryu.  Censored sampling of diffusion models using 3 minutes of human feedback.  _arXiv preprint arXiv:2307.02770_ , 2023.
  * Zhang et al. [2023] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3836–3847, 2023.

Supplementary Material

##  6 Continuity of Diffusion Models

In Section 3.3, we select the condition based on the inherent continuity in
diffusion models. Continuity, although abstract, is pivotal to understanding
why we select noise as the condition. To make this concept more tangible, we
demonstrate it by introducing variable scale perturbations at different time
steps. Specifically, for a fixed timestep $t$ and diffusion model
$\epsilon_{\theta}$, the $\epsilon_{\theta}(x_{t},t)$ is continuous with
respect to $x_{t}$. This implies that for any $\tilde{x}_{t}$ close to
$x_{t}$, the resulting image remains similar, which shows the continuity
principle in diffusion models.

###  6.1 Perturbation on Noise

Practically, for initial noise $\epsilon$, _i.e_. $t=T$, we add perturbation
as follows:

$\tilde{\epsilon}=\frac{\epsilon+w\cdot\epsilon_{p}}{\sqrt{1+w^{2}}},$ (16)

where $\epsilon_{p}$ is a perturbation sampled from a standard Gaussian
distribution and $w$ denotes the scale of perturbation. For intermediate
$x_{t}$ in timestep $t$, we first estimate the noise through the baseline
unconditional diffusion model:

$\hat{\epsilon}=\epsilon_{\theta}(x_{t},t).$ (17)

Then we add perturbation to the estimated noise $\hat{\epsilon}$:

$\tilde{\epsilon}=\frac{\hat{\epsilon}+w\cdot\epsilon_{p}}{\sqrt{1+w^{2}}},$
(18)

and we get the denoised image as follows:

$x_{t-1}=\sqrt{\bar{\alpha}_{t-1}}\hat{x}_{0}+\sqrt{1-\bar{\alpha}_{t-1}}\tilde{\epsilon},$
(19)

where $\hat{x_{0}}$ is predicted by $\hat{\epsilon}$ and $x_{t}$.

Figure 7: The architecture of our model. We show the shape on each module of
its output.

###  6.2 Result with Perturbation

As depicted in Fig. 8, with a minimal scale, such as 0.01, there is negligible
impact on the semantic content of the images across different steps. The
semantic information of generated images remains intact. As the scale
increases, the disparity between the original and the perturbed images becomes
more pronounced. At a scale of 0.2, the semantic alterations are most notable,
as exemplified by significant changes in elements like the postures of animals
in the images. For instance, the positions of a cat or a deer might shift
noticeably. Despite these changes, the overall semantic context — the
fundamental identity of the subjects — remains unchanged, underscoring the
continuity intrinsic to diffusion models.

However, at a higher scale, such as 0.5, the semantic shift becomes dramatic.
The generated images undergo substantial transformations, to the extent that
the class of the subject in the image can completely change, such as
transitioning from a cat to a frog.

The observation shows that for the cross point, _i.e_. the noised images are
identical, with similar conditions, the generated image will still remain the
same and cannot avoid crossing. With more distinct conditions, the
effectiveness of avoiding crossing increases.

Figure 8: We sample images using iDDPM[17] with 1000 inference steps on
Cifar-10. For the image grid, from left to right, the perturbation is added at
$t=1000,800,600,400,200$. From top to bottom, the perturbation is added with
weight $w=0.01,0.05,0.1,0.2,0.5$. We sample all images with DDIM[23]

##  7 Details of our model

ControlNet-based. Figure 7 presents the architecture of our proposed model.
Notably, we have omitted the zero convolution layer originally found in the
ControlNet [31] design. Both the UNet and Control Blocks are concurrently
trained from scratch. The outputs from Control Block A, B, C, and D are
integrated with the respective outputs of the input blocks. This integration
is achieved by adding the outputs from the Control Blocks to those of the
Input Blocks, followed by concatenation with the corresponding output from the
Output Blocks. The output from the middle block of ControlNet is directly
added to the output of UNet’s middle block, and then this combined output is
fed into UNet’s output block D.

Double UNet in Ablation Study. For Double UNet setting in ablation study, we
modified the input channel of the first layer in UNet. This modification
involved doubling the input channel, _i.e_., changing the input channel of the
first convolution layer from 3 to 6.

##  8 Training and Inference Algorithm

Algorithm 1 shows the training pipeline of our method. As described in Section
3, we replace the use of initial noise $\epsilon$ with predicted noise
$\hat{\epsilon}$ to avoid trivial solutions and training collapse. During
training, we set $\hat{\epsilon}=\mathbf{0}$, with a fixed probability
$p=0.5$, and otherwise, $\hat{\epsilon}$ is assigned the value of
$\epsilon_{\theta}(x_{t},\mathbf{0},t)$.

Algorithm 1 Training Stage

0: diffusion model $\epsilon_{\theta}$, training step $T$

1: repeat

2: sample $x_{0}\sim
q(x_{0}),\epsilon\sim\mathcal{N}(0,\mathbf{I}),t\sim\mathcal{U}(0,T-1)$

3: compute noised image
$x_{t}=\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon$

4: sample $p\sim U(0,1)$

5: $\hat{\epsilon}\leftarrow\mathbf{0}$

6: if $p\geq 0.5$ then

7:
$\hat{\epsilon},\hat{x}_{0}\leftarrow\epsilon_{\theta}(x_{t},\hat{\epsilon},t)$

8: $\hat{\epsilon}\leftarrow stop\\_gradient(\hat{\epsilon})$

9: end if

10:
$\hat{\epsilon},\hat{x}_{0}\leftarrow\epsilon_{\theta}(x_{t},\hat{\epsilon},t)$

11: $Loss=MSE(\epsilon,\hat{\epsilon})$

12: until converged

Algorithm 2 shows the inference pipeline of our method. We use estimated noise
in the previous step instead of the current step as the condition and
iteratively predict $\hat{\epsilon}$, which can effectivly alleviate the
computational costs.

Algorithm 2 Inference Stage

0: diffusion model $\epsilon_{\theta}$, inference step $T$

1: sample $x\sim\mathcal{N}(0,\mathbf{I})$

2: $\hat{\epsilon}\leftarrow\mathbf{0}$

3: for $t$ in $range(T,0)$ do

4: $\hat{\epsilon},\hat{x}_{0}\leftarrow\epsilon_{\theta}(x,\hat{\epsilon},t)$

5:
$x=\sqrt{\bar{\alpha}_{t-1}}\hat{x}_{0}+\sqrt{1-\bar{\alpha}_{t-1}}\hat{\epsilon}$

6: end for

7: return generated image $x$