# Developing, Analyzing, and Evaluating Vehicular Lane Keeping Algorithms
Under Dynamic Lighting and Weather Conditions Using Electric Vehicles

Michael Khalfin1, Jack Volgren2, Matthew Jones3, Luke LeGoullon4, Joshua
Siegel5, and Chan-Jin Chung6  1Department of Computational Applied
Mathematics, Rice University<EMAIL_ADDRESS>2Department of Computer Science
and Engineering, Pennsylvania State University<EMAIL_ADDRESS>3Department
of Mathematics and Computer Science, Willamette University,
<EMAIL_ADDRESS>4Division of Computer Science and Engineering,
Louisiana State University<EMAIL_ADDRESS>5Department of Computer Science
and Engineering, Michigan State University<EMAIL_ADDRESS>6Department of
Mathematics and Computer Science, Lawrence Technological University,
<EMAIL_ADDRESS>
###### Abstract

Self-driving vehicles have the potential to reduce accidents and fatalities on
the road. Many production vehicles already come equipped with basic self-
driving capabilities, but have trouble following lanes in adverse lighting and
weather conditions. Therefore, we develop, analyze, and evaluate two vehicular
lane-keeping algorithms under dynamic weather conditions using a combined deep
learning- and hand-crafted approach and an end-to-end deep learning approach.
We use image segmentation- and linear-regression based deep learning to drive
the vehicle toward the center of the lane, measuring the amount of laps
completed, average speed, and average steering error per lap. Our hybrid model
completes more laps than our end-to-end deep learning model. In the future, we
are interested in combining our algorithms to form one cohesive approach to
lane-following.

###### Index Terms:

self-driving algorithms, machine vision systems, computer-vision based
navigation, lane following algorithms, artificial intelligence for vehicle
control

##  I Introduction

An estimated 9,330 people from the United States died in motor vehicle
accidents in the first quarter of 2023 [1]. Additionally, for seven
consecutive quarters since 2020, fatalities increased, likely due to the
Covid-19 Pandemic.

Self-driving vehicles have the potential to reduce fatalities due to human
error. People believe that automated vehicles should be four to five times
safer than human-driven vehicles, or the traffic risk for automated vehicles
should be two orders of magnitude lower than human-driven vehicles [8]. Self
driving must become more consistent and reliable in order for this standard to
be met.

Figure 1: SAE six levels of vehicle autonomy

According to the Society of Automotive Engineers (SAE) International
association, there are six levels of vehicle autonomy based on the level of
human and computer involvement [5]. Level 2 systems include lane centering and
adaptive cruise control as driver support features but offer drivers the
chance to take control of the vehicle. Although Level 2 systems can be found
in many production vehicles, they often have trouble accounting for volatility
due to changing weather conditions [14]. In our research, we attempt to
reconcile these difficulties using machine vision systems.

Previous research on Level 2 systems used traditional algorithms without deep
learning integration [12]. The researchers involved in this study were
concerned with slow processing speeds and overfitting to training data.
However, there have been recent advancements in deep learning for automated
vehicles [2]. In our study, we endeavor to understand how hand-crafted
computer vision-based algorithms compare to deep learning algorithms, and
whether the two approaches could be combined.

Thus, the goal of this research is to develop, analyze, and evaluate two
vehicular lane-keeping algorithms under dynamic weather conditions using real-
street electric vehicles. One of these algorithms uses image segmentation and
Hough lines, whereas the other algorithm employs an end-to-end convolutional
neural network (CNN) with linear regression. We aim to compare the performance
of the algorithms to each other, as well as to an ordinary human driver.

##  II Literature Review

The architecture of self-driving cars is generally split into perception and
decision-making [2]. The perception system estimates how the vehicle relates
to its environment, using data from on-board sensors, whereas the decision-
making system is responsible for navigation. The localizer subsystem connects
these systems, as it estimates the vehicle’s position relative to its
surroundings [2]. Usually, localizers are Global Position System (GPS)-based,
Light Detection and Ranging (LIDAR)-based, or camera-based.

Preprocessing methods of lane detection aim to remove irrelevant image parts
and enhance the feature of interest [6]. They use image smoothing techniques,
like Median and Gaussian filters, to blur noisy details. At this stage, high
dynamic range (HDR) imaging algorithms can remedy the effects of direct
sunlight [10]. The other parts of the preprocessing process are extraction of
the region of interest (ROI) and inverse perspective mapping (IPM) to relocate
the pixels to a different position [6]. Various deep learning methods have
also been developed for lane following, including CNN- and Long-Short Term
Memory (LSTM) [13].

On the other hand, vehicle steering methods include spring method center
approximation and shifted line following [7]. Spring center approximation
begins with Canny edge detection and Hough line detection. The primary goal of
the algorithm is to push the center of the vehicle to the center of the lane
with spring physics [7]. On the other hand, the shifted line following locates
the rightmost line in the camera view with blob detection and steers the
vehicle at a shifted distance from the line [7]. Some deep learning algorithms
have also been proposed, including end-to-end deep learning [4]. Through our
research study, we aim to use Hough lines, too, but offer a different
approach.

Lastly, object detection is of paramount importance, so that a vehicle can
follow road signals and avoid collisions. There are many object detection
methods using deep learning [15]. In our research study, we are solely
concerned with red line detection to simulate vehicle behavior near a stop
sign. With simple cases like these, deep learning methods are often not
necessary.

##  III Methods

We start by explaining our experimental setup, as well as the structure of our
software architecture. Then, we discuss the combined image segmentation- and
hand-crafted algorithm. Next, we elaborate on our end-to-end deep learning
model. Finally, we go over our red detection node.

###  III-A Experimental Setup

####  III-A1 Simulation

We use the Robot Operating System (ROS) and Python to develop our algorithms.
We use OpenCV, a Computer Vision library, to capture and work with images. We
use TensorFlow and Keras for our deep learning. We test the code on simple-
sim, a 2-dimensional simulator, as opposed to Gazebo, a 3-dimensional
simulator with physics capabilities. Our tests were focused on flat
environments, thus the extra computation time of a 3-dimensional sim was not
necessary.

####  III-A2 Environment

(a) Our test course

(b) Our vehicle

Figure 2: Our real-time experimental setup

The test course is located in Lawrence Technological University (LTU) Parking
Lot H in Southfield, Michigan, a course with rough surfaces, potholes, tight
corners, and thin, fading road lines. The brightness and clarity of lines on
the course range over different times of day and inclement weather conditions.
There are two red markers, one on the outer portion of the outer lane and the
other on the inner portion of the inner lane, to mark the start of the course.
Both are located over straight portions of the rounded square. The vehicle
starts behind either marker and proceeds to drive along the lane until either
5 laps pass or an error occurs.

####  III-A3 Vehicle Specifications

We use a modified Polaris Gem e2 vehicle, generously sponsored by Mobis and
Dataspeed. We call this vehicle Autonomous Campus TranspORt (ACTor) 2. Our
vehicle is equipped with a drive-by-wire (DBW) system, vision sensors,
2-dimensional and 3-dimensional Light Detection and Ranging (LIDAR), global
positioning system (GPS), and on-board Ubuntu Linux computers. The Polaris Gem
e2 has a maximum speed of twenty miles per hour and the battery lasts for
twenty miles. We primarily use a forward-facing Mako G-319 Camera with a
resolution of 2064 × 1544 pixels, maximum frame rate of 37 frames per second,
and native ROS support.

###  III-B Software Architecture

Figure 3: ROS rqt graph for the hybrid model

Our lane-following algorithms are embedded in the Follow Lane Hough and Deep
Lane Follow nodes, respectively. They subscribe to the Camera Image Raw topic
for continuous Blue-Green-Red (BGR) camera feedback. Our lane-following nodes
also subscribe to the Vehicle Twist node to keep track of their forward-linear
x and angular z velocities, which we refer to as their speed and yaw rate
respectively. They then publish to Vehicle Command Velocity and Drive Enabled
topics to engage the DBW system. This is how they drive around the track.
Additionally, our Red Detect node subscribes to the Camera Image Raw topic to
know when a lap has been completed.

In our combined approach, we use a class-based architecture with a camera
callback function. As images are continuously received, they are passed to a
methods which handle preprocessing, Hough lines, and driving. However, in our
end-to-end deep learning node we do not implement a class as we solely have a
short camera callback function. It uses the pretrained deep learning model to
steer the vehicle.

###  III-C Combined Approach

####  III-C1 Image Segmentation

Figure 4: Image segmentation in real-time

Image segmentation is an image classification task that Artificial
Intelligence (AI) models are commonly trained to perform. We decided to use
image segmentation to create a simplified image of the road that could be
passed into lane following algorithms without concern for noise or many false
positives. An AI model trained to perform image segmentation tasks takes an
image array as an input. It identifies features it is trained to look for, in
our case lines on the road. It then returns an array with the values of the
identified objects as 1’s, and the rest of the values as 0’s. While our
segmentation model was only trained to identify the lines on either side of
the vehicle, a more robust model could be trained to identify the difference
between white and yellow lines, or dashed and solid lines. The biggest
drawback when it comes to employing image segmentation is training the AI
model. Every image the model is trained on needs to be drawn over by a human.
Data augmentation can speed up this process, but it still significantly slows
down the training of the model.

####  III-C2 Hand-Crafted Algorithm

After receiving the processed image, probabilistic hough lines are calculated.
We used a function out of the cv2 library that takes in an image and outputs a
list of lines on the image in the form of two (x,y) coordinates. We did
experiment with the standard hough lines function from cv2 as well, but found
that the output lines from the probabalistic function were much easier for the
calculations we sought to do and ran faster. Our function is set to filter out
lines shorter than 70 pixels, merge gaps between parallel lines of less than 4
pixels, and only have slopes between 0.25 and 100. Each of these values were
determined by testing each category manually on our track or in simulation for
maximizing true positives while minimizing false positives. The lines each has
their center calculated and then are sorted into two lists based on those
centers: left and right. We calculate the center points of the left hough
lines and right hough lines respectively to project where the lane lines are
on each side, and then average these x-values. This new average value is the
predicted midpoint of the lane. If the average is farther from the midpoint by
10 pixels, we adjust ACTor 2’s yaw rate by a factor proportional to the
difference between the midpoint and center x-coordinate, seen below in
Algorithm 1. Together, a hard coded speed, set to 1, and yaw rate are encoded
as a Twist message, which is received by the drive-by-wire system at a rate of
.34 hertz. Using this Twist message, ACTor 2 drives in the center of the lane.

initialization;

ratio = self.vel_msg.angular.z = (mid - center) / mid;

if  _center $<$ mid - 10_ then

self.vel_msg.angular.z = ratio;

else if  _center $>$ mid + 10_ then

self.vel_msg.angular.z = ratio;

else

self.vel_msg.angular.z = 0;

end if

Algorithm 1 Steering toward center of Hough lines

(a) Image segmentation

(b) Steering

Figure 5: Both parts of our real-time hybrid algorithm

The cyan line denotes where the algorithm identifies the left lane line, with
a similar process for the yellow line and right lane line. Red marks the
center of the image and green marks the center of the right and left projected
lanes. The blue and magenta lines represent identified hough lines,
effectively where the car sees the lane lines. Magenta denote lines longer
than 180 pixels. Shown in Figure 5b, the car would turn slightly left based on
the information given.

###  III-D End-to-End Deep Learning

####  III-D1 Training

Implementing deep learning algorithms requires training the models through a
variety of scenarios to ensure consistent performance under any conditions. To
train our end-to-end deep learning model, we created a program to
automatically pair the images captured from our camera with the steering angle
of the vehicle. This data could quickly be turned into a database that can be
used to train a TensorFlow model.

####  III-D2 Data Augmentation

We made sure to gather data in different light and weather conditions, but we
could not gather enough data just by driving, so we implemented data
augmentation. Data augmentation is commonly used to train deep learning
models; it involves slightly altering the inputs and pairing them up with the
same outputs. This teaches the model to ignore slight differences in data,
making the model more consistent. In our case, we augmented the data by
adjusting the brightness, contrast, orientation, and color map of the input
images. For every 1 input image, we augmented 9 more inputs, augmenting our
training data by a factor of 10.

Figure 6: Example of data augmentation

####  III-D3 Implementation

When implementing the model, we used common sense measures to ensure an even
more consistent result. These measures included limiting how far the model was
allowed to turn the wheel every time the script ran. Because we trained the
model on modified images, we concluded that the model would perform better if
it was allowed “multiple guesses” on modified images. For every frame captured
by the camera, we created 2 slightly altered images, adjusting the brightness
and contrast of the original. Then we fed the original image and the two
modified images to the model. Next, we took the 3 outputs from the model,
removed outliers that were far off from the previous steering angle, and set
the new steering angle to the average of those outputs. This gave the model
room for error and ensured that the model would work under even more
conditions.

##  IV Discussion

With computational models like ours, it is important to make assumptions. Some
assumptions we made were that vehicles did not have access to a Roadside Unit
and Wireless Fidelity (Wi-Fi) network. Hence, they could not collaborate with
other vehicles using an Intelligent Transportation System [3]. Another
assumption we made was vehicles driving based on input from a sensor,
specifically a camera. We restricted ourselves to a single, forward-facing
camera in order to focus in on lane-following, as if a car can follow a lane
with one camera, other cameras can be focused on other features such as
safety. Hypothetically, all the roadways could be navigated using GPS and path
planning alongside the camera, but that was not the goal of this study.

###  IV-A Failed Approaches

Although our lane-following algorithms were mostly successful, we experimented
with other approaches which worked poorly.

####  IV-A1 Dynamic Threshold

We did not always use image segmentation during our preprocessing step in the
hybrid algorithm. At first, we found the rightmost contour in each BGR image
(the right lane line). Using OpenCV in Python, we median blurred the images
and converted them to grayscale. Next, we chose an arbitrary threshold value
to convert the images to black-white (BW). We selected minimum and maximum
percentages of white pixels allowed per image and checked the percentage in
our current BW image. If this percentage was not within bounds, we adjusted
the threshold using binary search until a suitable threshold was found. Then,
we drove the vehicle at a constant displacement from the right lane line.

However, our dynamic thresholding did not hold up in adverse weather and
volatile lighting conditions. We subsequently pivoted our preprocessing design
to use deep learning based on various images.

####  IV-A2 DeepLSD

We experimented with a line segment detection and refinement method leveraging
deep image gradients, coined DeepLSD[11]. We integrated it into our simple-sim
simulation, where it detected many of the black road markings as lines, while
not marking the white road lines. After early failures, we decided to solely
focus on developing the hybrid and end to end deep learning algorithms for the
sake of time. Although we did not succeed in implementing a working adaptation
of DeepLSD, we would like to further explore its viability in the future.

####  IV-A3 Slope-Based Algorithm

In our hand-crafted steering algorithm, we we were tasked with discovering
creative approaches to make ACTor 2 drive in the center of the lane. Given
images of the course, we originally postulated that the mean slope of the lane
lines on either side of ACTor 2 is related to its yaw rate. However, when we
found no relationship between these values, we tried calculating the mean
slopes differently. We first weighted the positive and negative slopes
equally. Next, we weighted the slopes on the left and right side of the screen
equally. We even tried using the median mean slope weighted by the length of
the lane lines from the past 10 images to fit a logistic-sigmoid (1) curve
with SciPy. Lastly, we tried fitting an arctangent (2) curve and piecewise
function (3). All of our approaches were unsuccessful.

$\displaystyle\frac{L}{1+e^{-k\cdot(x-x_{0})}}$ (1) $\displaystyle
k\cdot\tan^{-1}(w\cdot(x-x_{0}))+y_{0}$ (2)
$\displaystyle\begin{cases}\frac{-1}{x+x_{0}}+k_{1},&\text{if }x<r_{1}\\\
mx+b,&\text{if }r_{1}<x<r_{2}\\\ \frac{-1}{x+x_{1}}+k_{2},&\text{if
}x>r_{2}\end{cases}$ (3)

Figure 7 shows a scatter plot of the mean slope of the lane lines on either
side of ACTor 2 relative to its yaw rate. To produce this data, we manually
drove around the track for 2 laps. The orange line is the baseline model,
which is just the arithmetic average of all the slopes. The red line is a
linear regression using an 80%-20% training-testing dataset split with Scikit-
learn.

We ultimately used a different approach to calculate the vehicle steering, as
described in our methods. We did not rely on the slope or length of the lane
lines, opting instead to direct the vehicle toward the center of the lines.

Figure 7: A scatter plot of mean slope vs yaw rate

###  IV-B Limitations and Future Work

####  IV-B1 Limitations

We did not evaluate night compatibility for our models. But our camera cannot
display images well in the absence of light. To drive the vehicle at night, we
could experiment with higher-definition cameras that have night-vision or
LIDAR sensors.

One major limitation was all our testing taking place on LTU’s Parking Lot H.
We are not sure how our methods would fare in other environments. We chose a
test course with shadows, jagged surfaces, and sharp corners. We explicitly
chose not to drive on other roads near LTU because Level 2 algorithms already
work well on straightaways. On the flip side, roads in local neighborhoods
pose other challenges like dead reckoning curves, where there are no lines,
and parked cars and obstacles.

Another limitation was hardware limitations. We ran our programs every 20
milliseconds, but could not run our programs much faster with the resources
given to us.

Moreover, we did not consider robustness to adverserial attacks. In a future
study, we can look into security considerations.

####  IV-B2 Future Work

In the future, we would like our Follow Lane Hough and Deep Lane Follow nodes
to run simultaneously. Currently, they publish to the Vehicle Command Velocity
and Drive Enabled Topics to deterministically steer the vehicle around the
course. However, we would like both nodes to publish perspective speed and yaw
rate messages to a Control node, as well as their confidence. The Control node
would dynamically weight the outputs of each model to derive new values for
the speed and yaw rate. Then, it would steer the vehicle based on these
values.

One avenue for further exploration is other partial deep learning approaches.
For instance, we can use Recurrent CNN’s to label environments other than our
test course [9]. Scene classification like [9] is a good way to extract
features of representative objects. Then, values for parameters in our models
would be updated accordingly. Furthermore, an alternative approach to image
segmentation is using unsupervised training.

Other ideas we have are end-to-end deep learning approaches that are not based
on linear regression. There is not much literature on reinforcement learning
for self-driving vehicles.

###  IV-C Conclusion

We presented two vehicular lane-following algorithms. The first algorithm uses
a combined traditional- and deep-learning approach, whereas the second
algorithm uses end-to-end linear regression-based deep learning. The hybrid
algorithm completes 5 laps on the inner and outer lane, whereas the second
algorithm fails on the inner lane. In the future, we need to continue working
on Level 2 algorithms in dynamic weather conditions, and ultimately implement
these algorithms in production vehicles. Self-driving algorithms of this type
are becoming increasingly important, as they can reduce vehicular fatalities
on the road. In the robotics, autonomous vehicles, and computer science
fields, it is imperative that we collaborate to increase safety on the
roadways by further designing lane-following algorithms.

## Acknowledgment

Our work was supported by the National Science Foundation under Grants No.
2150292 and 2150096. We thank our mentors, Joe DeRose and Nick Paul, as well
as our teaching assistants, Ryan Kaddis and Justin Dombecki.

## References

  * [1] National Highway Traffic Safety Administration, “Early Estimate of Motor Vehicle Traffic Fatalities For the First Quarter of 2023,” Early Estimate Report, U.S. Department of Transportation, 2023.
  * [2] C. Badue, R. Guidolini, R. V. Carneiro, P. Azevedo, V. B. Cardoso, A. Forechi, L. Jesus, R. Berriel, T. M. Paixão, F. Mutz, L. de P. Veronese, T. Oliveira-Santos, and A. F. De Souza, ”Self-driving cars: A survey,” Expert Systems with Applications, vol. 165, 2020.
  * [3] S. Chen, J. Hu, Y. Shi, Y. Peng, J. Fang, R. Zhao, and L. Zhao, “Vehicle-to-Everything (v2x) services supported by LTE-based systems and 5G,” IEEE Communications Standards Magazine, vol. 1, no. 2, pp. 70-76, July 2017. [Online]. Available: https://doi.org/10.1109/MCOMSTD.2017.1700015
  * [4] H. M. Eraqi, M. N. Moustafa, and J. Honer, “End-to-End Deep Learning for Steering Autonomous Vehicles Considering Temporal Dependencies,” CoRR, vol. abs/1710.03804, 2017. [Online]. Available: https://arxiv.org/abs/1710.03804
  * [5] Society for Automotive Engineering, “SAE Levels of Driving Automation™ Refined for Clarity and International Audience,” 2021. [Online]. Available: https://www.sae.org/blog/sae-j3016-update
  * [6] A. Heidarizadeh, “Preprocessing Methods of Lane Detection and Tracking for Autonomous Driving,” arXiv preprint, 2021. [Online]. Available: https://arxiv.org/abs/2104.04755
  * [7] R. Kaddis, E. Stading, A. Bhuptani, H. Song, C. Chung, and J. Siegel, “Developing, Analyzing, and Evaluating Self-Drive Algorithms Using Electric Vehicles on a Test Course,” in 2022 IEEE 19th International Conference on Mobile Ad Hoc and Smart Systems (MASS), 2022, pp. 687–692. [Online]. Available: https://doi.org/10.1109/MASS56207.2022.00101
  * [8] P. Liu, R. Yang, and Z. Xu, “How Safe is Safe Enough for Self-Driving Vehicles?” Risk Analysis, 2018. [Online]. Available: https://doi.org/10.1111/risa.13116
  * [9] J. Ni, K. Shen, Y. Chen, W. Cao, and S. X. Yang, “An Improved Deep Network-Based Scene Classification Method for Self-Driving Cars,” IEEE Transactions on Instrumentation and Measurement, vol. 71, 2022, pp. 1–14. [Online]. Available: https://doi.org/10.1109/TIM.2022.3146923
  * [10] N. Paul and C. Chung, “Application of HDR algorithms to solve direct sunlight problems when autonomous vehicles using machine vision systems are driving into sun,” Computers in Industry, vol. 98, 2018, pp. 192–196. [Online]. Available: https://doi.org/10.1016/j.compind.2018.03.011
  * [11] R. Pautrat, D. Barath, V. Larsson, M. R. Oswald, and M. Polle-feys, “DeepLSD: Line Segment Detection and Refinement with Deep Image Gradients,” in Computer Vision and Pattern Recognition (CVPR), 2023.
  * [12] S. Rao, A. Quezada, S. Rodriguez, C. Chinolla, C. Chung, and J. Siegel, “Developing, Analyzing, and Evaluating Vehicular Lane Keeping Algorithms Using Electric Vehicles,” Vehicles, vol. 4, no. 4, Oct 2022, pp. 1012–1041. [Online]. Available: https://doi.org/10.3390/vehicles4040055
  * [13] J. Tang, S. Li, and P. Liu, “A review of lane detection methods based on deep learning,” Pattern Recognition, vol. 111, 2020. [Online]. Available: https://doi.org/10.1016/j.patcog.2020.107623
  * [14] S. Zang, M. Ding, D. Smith, P. Tyler, T. Rakotoarivelo, and M. A. Kaafar, “The Impact of Adverse Weather Conditions on Autonomous Vehicles: How Rain, Snow, Fog, and Hail Affect the Performance of a Self-Driving Car,” IEEE Vehicular Technology Magazine, vol. 14, no. 2, 2019, pp. 103–111. [Online]. Available: https://doi.org/10.1109/MVT.2019.2892497
  * [15] Z. Zhao, P. Zheng, S. Xu, and X. Wu, “Object Detection With Deep Learning: A Review,” IEEE Transactions on Neural Networks and Learning Systems, vol. 30, no. 11, 2019, pp. 3212–3232. [Online]. Available: https://doi.org/10.1109/TNNLS.2018.2876865


# Generative Data Augmentation Improves
Scribble-supervised Semantic Segmentation

Jacob Schnell1 Jieke Wang2 Lu Qi2 Vincent Tao Hu3 Meng Tang2
1University of Waterloo 2University of California, Merced 3LMU Munich

###### Abstract

Recent advances in generative models, such as diffusion models, have made
generating high-quality synthetic images widely accessible. Prior works have
shown that training on synthetic images improves many perception tasks, such
as image classification, object detection, and semantic segmentation. We are
the first to explore generative data augmentations for scribble-supervised
semantic segmentation. We propose a generative data augmentation method that
leverages a ControlNet diffusion model conditioned on semantic scribbles to
produce high-quality training data. However, naive implementations of
generative data augmentations may inadvertently harm the performance of the
downstream segmentor rather than improve it. We leverage classifier-free
diffusion guidance to enforce class consistency and introduce encode ratios to
trade off data diversity for data realism. Using the guidance scale and encode
ratio, we are able to generate a spectrum of high-quality training images. We
propose multiple augmentation schemes and find that these schemes
significantly impact model performance, especially in the low-data regime. Our
framework further reduces the gap between the performance of scribble-
supervised segmentation and that of fully-supervised segmentation. We also
show that our framework significantly improves segmentation performance on
small datasets, even surpassing fully-supervised segmentation.

Figure 1:  Segmentation model performance on PascalVOC and its subsets. All
results other than full-mask supervision use scribble-supervision with ResNet-
based RLoss [38] model. Naive data augmentation (i.e., fixed encode ratio
$\lambda=1.0$) harms model performance, especially in the low-data regime,
while our augmentation scheme (Adaptive $\lambda$ Sampling) improves
performance.

##  1 Introduction

With the massive leaps forward in modern deep learning, machine learning model
capacity has never been higher, with some models reaching billions of
parameters [8, 11]. However, for many tasks, the size and complexity of
datasets have not kept up with the explosion in model capacity. Since machine
learning models perform with a large and rich training dataset, the question
of scaling datasets to match model sizes is increasingly pressing. For tasks
like fully-supervised semantic segmentation (FSSS), however, this is
especially expensive due to the need for dense pixel-level annotations. These
annotations must often also be produced by experts with domain-specific
knowledge, exacerbating the costs of data labeling even further.

Weakly-supervised semantic segmentation (WSSS) seeks to reduce the requirement
for dense annotations by using weak annotations. Such methods include
scribble-supervised semantic segmentation, where only a fraction of pixels
along some lines (scribbles) are provided. However, these methods still lag
behind fully-supervised alternatives regarding segmentation quality, with
state-of-the-art methods still achieving 2-4% lower mIoU [41, 25] relative to
fully-supervised models.

Another strategy is to produce synthetic training data using image-generative
models. Prior works have shown that using Generative Adversarial Networks
(GANs) to produce training data improves results in image classification [13]
and semantic segmentation [50, 3], among other tasks. Diffusion models [34,
17, 36], a well-known type of generative models, have demonstrated strong
performance in terms of controllability [31, 48] and fidelity [10, 32].
Several studies have successfully applied diffusion models to synthesize
training data for image classification [1], object detection [49], and fully-
supervised segmentation [42, 46]. This raises the question: Can we also
leverage the power of diffusion models to synthesize training data to further
enhance the performance of scribble-supervised segmentation?

In this work, we use a diffusion model conditioned on semantic scribble labels
to produce high-fidelity synthetic training images. We also note that deep
image-generative models, such as the diffusion models commonly used today,
often require large datasets to produce high-quality images. This leads us to
a paradox where to upscale our training dataset, we need to already have
access to a large training dataset. We address this problem by including a new
parameter in the generative process, the encode ratio, which lets us trade off
image diversity for image photorealism.

Our contributions are summarized as follows:

  * •

We are the first to leverage denoising diffusion models for generative data
augmentation for scribble-supervised semantic segmentation. Our approach
produces a spectrum of synthetic images conditioned on scribbles using
different guidance scales and encode ratios.

  * •

We provide detailed analyses and propose several schemes to combine synthetic
and real data effectively for scribble-supervised semantic segmentation. We
also identify the limitations of naive data augmentation schemes that can harm
segmentation performance relative to not using synthetic training data at all.

  * •

We achieve state-of-the-art results in scribble-supervised semantic
segmentation, closing the gap between weakly-supervised and fully-supervised
models as shown in Fig. 1. In particular, our framework significantly improves
segmentation results in the low-data regime, where only a limited number of
images are available.

##  2 Related work

#### Synthetic training data

Numerous efforts have been dedicated to leveraging synthetic data for training
perception models. IT-GAN [51] shows that GAN-generated samples can help
classification models learn faster and improve performance. DatasetGAN [50]
and BigDatasetGAN [24] employ GANs [19, 4] for jointly generating synthetic
images and their corresponding labels for segmentation tasks.

Recent advances in diffusion models have brought notable stability during
training, robust synthesis capabilities [10], and enhanced controllability
[48]. As a result, there has been a significant shift towards the use of
diffusion models for data synthesis, including for image classification [1,
20], object detection [49], instance segmentation [43], and semantic
segmentation [29, 42, 47]. For example, by fine-tuning an Imagen [32] model on
ImageNet [9], [1] generates synthetic images from text prompts to improve the
performance of image classification. Similarly, D3S [20] introduces a novel
synthetic dataset specially designed to mitigate the foreground and background
biases prevalent in real images. Dataset Diffusion [29] and DatasetDM [42]
jointly generate synthetic images and associated mask annotation, akin to
DatasetGAN, using a StableDiffusion [31] image-generative model. FreeMask [47]
utilizes FreestyleNet [46] to synthesize images conditioned on full mask
annotations.

Our work diverges from these initiatives by focusing on sparse labels (e.g.,
scribbles) from real images as generative conditions, encouraging the creation
of realistic and diverse synthetic images. While FreeMask [47] similarly
conditions synthetic images on real data annotations, our method uses sparse
rather than dense annotations, allowing for broader applications where dense
labeling is expensive.

#### Guidance in Diffusion models

Diffusion models excel in various tasks due to their controllability [48].
They’re used to generate image content [17], image layout [31, 18, 28], audio
content [27], human motion [39], etc. Guidance signals can also be
incorporated to enhance image fidelity [10, 16] relative to unconditional
generation. It has been shown that diffusion models can be guided by
pretraining a noisy-data-based classifier, known as Classifier-guidance [10].
On the other hand, classifier-free guidance [16] removes the need for extra
pretraining by randomly dropping out the guidance signal during training. We
develop a framework that utilizes classifier-free guidance for generative data
augmentation to improve scribble-based segmentation.

#### Weakly-supervised segmentation

Weakly-supervised segmentation methods use weak annotations rather than full
segmentation masks to train segmentation networks for images [38, 21, 6, 25,
41, 44] or point clouds [40]. Forms of weak annotations include points [2, 38,
41], scribbles [38, 25, 41], bounding boxes [22], image-level tags [6], and
text [44]. These methods can be roughly categorized into two groups. The first
group proposes various unsupervised or semi-supervised losses such as entropy
loss [5], CRF loss [38], and contrastive-learning losses [21]. The second
group iteratively refines full-mask pseudo-labels [22, 26] during training to
mimic full supervision. Many weakly-supervised approaches rely on class
activation maps (CAMs) [52, 5] that gives localization cues from
classification networks. Our generative data augmentation approach complements
any existing weakly-supervised segmentation methods, as we show improved
performance of several methods with our synthetic data.

Weak annotations can also be provided as input for segmentation networks at
test time for interactive segmentation [45, 23]. For example, Segment Anything
[23] allows prompts including clicks, bounding boxes, masks, or text. While
Segment Anything [23] provides many masks in a semi-automatic way for training
interactive segmentation, we focus on synthetic image synthesis for training
weakly-supervised segmentation.

##  3 Method

Figure 2: Given a limited number of real scribbles, we pretrain a ControlNet-
based diffusion model for high-fidelity synthesis of images conditioned on
scribbles. We can control the image synthesis with the encode ratio $\lambda$
and the guidance scale $w$. These image-scribble pairs can then be smoothly
integrated into the training of scribble-based semantic segmentation.

In this section, we describe our method of generative data augmentation for
weakly-supervised semantic segmentation outlined in Fig. 2. First, in Sec.
3.1, we provide a background on sampling from diffusion models. Then, in Sec.
3.2, we introduce a variant of ControlNet [48] conditioned on scribble labels
and text prompts. We further discuss how to achieve semantically consistent
images and trade off diversity and photorealism through guided diffusion and
encode ratio in Sec. 3.3 and Sec. 3.4, respectively. Sec. 3.5 proposes several
schemes to effectively combine synthetic and real images for training
segmentation networks.

###  3.1 Background

#### Diffusion models.

Diffusion models [34, 17, 36] learn to reverse a forward process that
gradually adds noise to an image $\mathbf{x}_{\mathrm{ref}}$ until the
original signal is fully diminished. After training, following the reverse
process allows us to sample an image $\mathbf{x}_{0}$ given noise
$\mathbf{x}_{T}\sim\mathcal{N}(0,I)$. Learning this reverse process reduces to
learning a denoiser ${\bm{\epsilon}}_{\theta}$ that recovers the original
image from a noisy image $\mathbf{x}_{t}$ as

$\mathbf{x}_{\mathrm{ref}}\approx
f_{\theta}(\mathbf{x}_{t},t):=(\mathbf{x}_{t}-(1-\overline{\alpha}_{t}){\bm{\epsilon}}_{{\bm{\theta}}}(\mathbf{x}_{t},t))/\sqrt{\overline{\alpha}_{t}}.$
(1)

To get high-quality samples, the standard diffusion model sampling process
[17] requires many (often $T=1,000$) neural function evaluations. Using a non-
Markovian forward process, Denoising Diffusion Implicit Model (DDIM) [35]
samplers forego several intermediate function calls, accelerating sampling.
Let $\tau$ be an increasing subsequence of $[T,\ldots,1]$ and define the DDIM
forward process for some stochasticity parameter $\sigma\in\mathbb{R}_{\geq
0}^{T}$ as

$q_{\sigma}(\mathbf{x}_{\tau_{i-1}}|\mathbf{x}_{\tau_{i}},\mathbf{x}_{0})=\mathcal{N}\Bigg{(}\sqrt{\alpha_{\tau_{i-1}}}\mathbf{x}_{0}\\\
+\sqrt{1-\alpha_{\tau_{i-1}}-\sigma_{\tau_{i}}^{2}}\cdot\frac{\mathbf{x}_{\tau_{i}}-\sqrt{\alpha_{\tau_{i}}}\mathbf{x}_{0}}{\sqrt{1-\alpha_{\tau_{i}}}},\;\sigma_{\tau_{i}}^{2}I\Bigg{)}.$
(2)

We can then sample from the generative process using the abovementioned
forward process. In particular, using $f_{\theta}(x_{t},t)$ as defined in Eq.
(1) we can sample $\mathbf{x}_{\tau_{i-1}}$ from $\mathbf{x}_{\tau_{i}}$ by

$p_{\theta}^{(\tau_{i})}(\mathbf{x}_{\tau_{i-1}}|\mathbf{x}_{\tau_{i}})=\begin{cases}\mathcal{N}(f_{\theta}(\mathbf{x}_{\tau_{i}},\tau_{i}),~{}\sigma_{\tau_{i}}^{2}I)&\text{if
$i=1$}\\\
q_{\sigma}(\mathbf{x}_{\tau_{i-1}}|\mathbf{x}_{\tau_{i}},f_{\theta}(\mathbf{x}_{\tau_{i}},\tau_{i}))&\text{if
$i>1$}\end{cases}$ (3)

We slightly abuse notation here and define $\tau_{0}=0$ so that when $i=1$, we
sample the denoised image $\mathbf{x}_{0}$.

#### Classifier-free guidance.

To trade off mode coverage and sample fidelity in a conditional diffusion
model, [10] proposes to guide the image generation process using the gradients
of a classifier, with the additional cost of training the classifier on noisy
images. To address this drawback, classifier-free guidance [16] does not
require any classifier. They obtain a conditional and unconditional network
combination in a single model by randomly dropping the guidance signal
$\mathbf{c}$ during training. After training, it empowers the model with
progressive control over the degree of alignment between the guidance signal
and the sample by varying the guidance scale $w$ when a larger $w$ leads to
greater alignment with the guidance signal:

$\tilde{{\bm{\epsilon}}}_{\theta}(\mathbf{x}_{t},t;~{}\mathbf{c},w)=(1+w){\bm{\epsilon}}_{\theta}(\mathbf{x}_{t},t;~{}\mathbf{c})-w{\bm{\epsilon}}_{\theta}(\mathbf{x}_{t},t).$
(4)

.

###  3.2 Scribble-conditioned Image Synthesis

We consider a semantic synthesis approach to generating our synthetic training
data. The synthetic training data is generated conditioned on real
segmentation labels from the training dataset. We leverage a typical denoising
diffusion model, ControlNet [48], to achieve image synthesis conditioned on
the segmentation scribbles. Our model is trained using the usual DDPM [17]
object: given a noisy image $\mathbf{x}_{t}$ (in reality $\mathbf{x}_{t}$ is a
latent representation as in [31], but we omit this detail for brevity) and
conditioning input $\mathbf{c}$ it predicts the added noise ${\bm{\epsilon}}$.
Our segmentation scribbles on which the model is conditioned are represented
as RGB images in $\mathbb{R}^{h\times w\times 3}$ with different colors for
every class, though we explore other representations in Sec. 4.3.

Finally, we note that it is difficult for the ControlNet model to produce
semantically consistent images with the given scribble labels. We hypothesize
that this is due to the difficulty of encoding class information in RGB
images, especially in the early stages of training. Therefore, we supplement
our model with text prompts that include all the classes within the image.
Adding these prompts significantly improves image class consistency and leads
to higher-quality images relative to an unchanging default prompt. We explore
the effect of this prompt in Sec. 4.3.

Our ControlNet training objective is thus

$\mathcal{L}_{\mathrm{CN}}({\bm{\theta}})=\mathbb{E}_{(\mathbf{x}_{\mathrm{ref}},\mathbf{c}_{s},\mathbf{c}_{t}),t,{\bm{\epsilon}}}\left[\lVert{\bm{\epsilon}}-{\bm{\epsilon}}_{\theta}(\mathbf{x}_{t},t,\mathbf{c}_{s},\mathbf{c}_{t})\rVert_{2}^{2}\right],$
(5)

where $(\mathbf{x}_{\mathrm{ref}},\mathbf{c}_{s},\mathbf{c}_{t})$ is the
triplet of the original (unnoised) image, the conditioning scribble label, and
the conditioning text prompt and ${\bm{\epsilon}}_{\theta}$ is our ControlNet
diffusion model.

###  3.3 Classifier-free Scribble Guidance

We leverage diffusion guidance to further improve semantic consistency between
the generated synthetic image and conditional input. Following the proposals
from Classifier-free Guided Diffusion [16], we randomly drop out 10% of all
conditioning scribble inputs $\mathbf{c}_{s}$, replacing them with a randomly
initialized, learned embedding $\tilde{\mathbf{c}}$, when training the
ControlNet model. By modifying Eq. 4, we arrive to a new guided noise
prediction function:

$\tilde{{\bm{\epsilon}}}_{\theta}(\mathbf{x}_{t},t;\mathbf{c}_{s},\mathbf{c}_{t},w)=(1+w){\bm{\epsilon}}_{\theta}(\mathbf{x}_{t},t;\mathbf{c}_{s},\mathbf{c}_{t})-w{\bm{\epsilon}}_{\theta}(\mathbf{x}_{t},t;\tilde{\mathbf{c}}).$
(6)

While ControlNet uses a pre-trained Stable-Diffusion model [31], which is
trained conditionally and unconditionally, scribble drop-out during training
can be viewed as finetuning the unconditional diffusion model to our dataset.
We have found that the guidance scale, $w$, can significantly impact the
quality of generated images, especially with respect to the fine-grain details
of the produced image. We further ablate this hyperparameter’s impact in Sec
4.3.

###  3.4 Control Image Diversity via Encode Ratio

The vanilla diffusion model denoises sampled Gaussian noise
$\mathbf{x}_{T}\sim\mathcal{N}(0,I)$ iteratively until $\mathbf{x}_{0}$ at
inference time. In practice, synthetic images generated this way may be
unrealistic, particularly when training data is limited for our scribble-
conditioned diffusion model. To improve photorealism at the cost of diversity,
we propose another forward diffusion process parameter, the encode ratio
$\lambda\in(0,1]$. Specifically, we perform $\lambda\cdot T$ noise-adding
forward diffusion steps to the input images and, during inference, denoise
$\mathbf{x}_{\lambda T}$ iteratively until $\mathbf{x}_{0}$. Thus, for
$\lambda=1$, there is no change, but for small choices of $\lambda$, there is
less noise added to the image $\mathbf{x}_{0}$. As $\lambda\to 0$, the sampled
image will become increasingly similar to the original
$\mathbf{x}_{\mathrm{ref}}$. Therefore, a whole spectrum of synthetic images
with varying levels of similarity to the reference image can be achieved by
varying our choice of $\lambda$. We outline our sampling algorithm, which
combines the accelerated DDIM sampling from Sec. 3.1, the scribble guidance
from Sec. 3.3, and the encode ratio from Sec 3.4 in Algorithm 1. Fig. 3 shows
synthetic images generated with varying guidance scales and encode ratios.

Require : $q_{\sigma}$: forward process

Require : $\mathbf{x}_{\mathrm{ref}}$: a reference image

Require : $w\geq 0$: guidance scale

Require : $\lambda\in[0,1]$: encode ratio

Require : $N\in\\{1,\ldots,T\\}$: number of reverse diffusion process steps

Require : $c_{t}$ and $c_{s}$: text prompt and scribble conditioning

1 ${\bm{\epsilon}}\sim\mathcal{N}(0,I)$

2 $\tau=[\lfloor\frac{\lambda T}{N}n\rfloor:0\leq n\leq N]$

$\triangleright$ Note if $\lambda=1$ then
$\mathbf{x}_{\tau_{N}}\sim\mathcal{N}(0,I)$

3
$\mathbf{x}_{\tau_{N}}=\sqrt{\alpha_{\tau_{N}}}\mathbf{x}_{\mathrm{ref}}+\sqrt{1-\alpha_{\tau_{N}}}{\bm{\epsilon}}$

4 for  _$i=N$ to $1$_ do

$\triangleright$ Predict added noise using diffusion guidance (6)

5
$\tilde{{\bm{\epsilon}}}_{\tau_{i}}=(1+w){\bm{\epsilon}}_{\theta}^{(\tau_{i})}(\mathbf{x}_{\tau_{i}},c_{t},c_{s})-w{\bm{\epsilon}}_{\theta}^{(\tau_{i})}(\mathbf{x}_{\tau_{i}},c_{t},\tilde{c}_{s})$

$\triangleright$ Accelerated DDIM sampling (3)

6
$\hat{\mathbf{x}}_{0}=(\mathbf{x}_{\tau_{i}}-\sqrt{1-\alpha_{\tau_{i}}}\cdot\tilde{{\bm{\epsilon}}}_{\tau_{i}})/\sqrt{\alpha_{\tau_{i}}}$

7 if  _$i=1$_ then

8 $\mathbf{x}_{0}\sim\mathcal{N}(\hat{\mathbf{x}}_{0},\sigma^{2}_{\tau_{1}}I)$

9

10 else

11 $\mathbf{x}_{\tau_{i-1}}\sim
q_{\sigma}(\mathbf{x}_{\tau_{i-1}}|\mathbf{x}_{\tau_{i}},\hat{\mathbf{x}}_{0})$

12

13 end if

14

15 end for

16return $\mathbf{x}_{0}$

Algorithm 1 Conditional DDIM sampling with guidance scale $w$ and encode ratio
$\lambda$

Figure 3:  Left: Our sampled synthetic images conditioned on the ground-truth
scribble. By sampling using different guidance scales and encode ratios we are
able to generate a whole spectrum of realistic synthetic training images.
Right: The ground-truth real image and corresponding scribble label.

###  3.5 Combine synthetic images with real images

Generative data augmentation can, in principle, produce an infinite amount of
synthetic images. However, naively combining real and synthetic images can
harm rather than benefit weakly-supervised segmentation models, as we have
observed. In particular, it is not clear which choices of the guidance scale
$w$ and encode ratio $\lambda$ are optimal. We choose the optimal guidance
scale, as determined in Sec. 4.3. For encode ratio $\lambda$, we propose and
systematically evaluate three strategies for combining synthetic with real
images.

Let $\mathcal{X}=\\{\mathbf{x}_{1},\ldots,\mathbf{x}_{n}\\}$ denote the set of
all real images and $\mathcal{Y}=\\{\mathbf{y}_{1},\ldots,\mathbf{y}_{n}\\}$
denote the set of all (scribble) labels. Then we produce a set of synthetic
images
$\hat{\mathcal{X}}=\\{\hat{\mathbf{x}}_{1},\ldots,\hat{\mathbf{x}}_{n}\\}$
where
$\hat{\mathbf{x}}_{i}=\mathrm{DM}_{\theta}(\mathbf{y}_{i},\mathbf{c}_{i};w,\lambda)$
is the output of our trained diffusion model, $\mathrm{DM}_{\theta}$,
conditioned on the scribble $\mathbf{y}_{i}$ and prompt-condition
$\mathbf{c}_{i}$, given guidance scale $w$ and encode ratio $\lambda$. We may
then produce a new, augmented dataset
$\mathcal{X}^{\prime}=\mathrm{concat}(\mathcal{X},\hat{\mathcal{X}})$ and
$\mathcal{Y}^{\prime}=\mathrm{concat}(\mathcal{Y},\mathcal{Y})$. Note this
means each label, $\mathbf{y}_{i}$, appears twice in our dataset, once for the
real image $\mathbf{x}_{i}$ and once for the synthetic image
$\hat{\mathbf{x}}_{i}$.

  * •

Fixed encode ratio $\lambda=1$: We choose the default encode ratio of $1$
which gives the most diverse synthetic images. The synthetic dataset
$\hat{\mathcal{X}}$ is fixed for this case.

  * •

Uniform $\lambda$-sampling: At each iteration, we sample synthetic images that
are generated based on a range of encode ratios $\lambda$, e.g.,
$\Lambda=\\{0.5,0.6,...,1.0\\}$. In other words, the synthetic data injected
is a mix of images of different levels of similarity relative to real images.
For this case, we form the synthetic dataset $\hat{\mathcal{X}}$ every epoch
by independently and uniformly sampling each image
$\hat{\mathbf{x}}_{i}\in\hat{\mathcal{X}}$ from the set of all synthetic
images $\\{\hat{\mathbf{x}}_{i,\lambda}:\lambda\in\Lambda\\}$ with different
encode ratios $\lambda$.

  * •

Adaptive $\lambda$-sampling: We gradually increase encode ratio $\lambda$
while training downstream segmentation networks, similar to curriculum
learning. Initially, synthetic images used for training are similar to real
images, which are considered an easier curriculum to learn. Synthetic images
diverge increasingly from the real images as training progresses. For this
case, the synthetic dataset is formed at epoch $e$ as
$\hat{\mathcal{X}}=\\{\hat{\mathbf{x}}_{1,\lambda_{e}},\ldots,\hat{\mathbf{x}}_{1,\lambda_{e}}\\}$
where we follow the encode ratio schedule
$[\lambda_{1},\ldots,\lambda_{E}]\in\Lambda^{E}$ where $E$ is the number of
training epochs.

##  4 Experiments

Sec. 4.1 summarize our main results that show improvements on several
scribble-supervised segmentation methods using our generative data
augmentation. In Sec. 4.2, we further explore the challenging scenario with
limited number of real images. We show that naive implementations of
generative data augmentation may harm the performance, whereas our data
augmentation scheme improves. Sec. 4.3 gives an ablation study on guidance
scale and encode ratio, two critical degrees of freedom for our image
synthesis.

#### Dataset and Implementation Details

We report results on the standard PASCAL VOC12 segmentation dataset which
contains 10 582 images for training and 1 449 images for validation. We
utilize scribbles from ScribbleSup dataset [26] with only 3% pixels labeled on
average.

For image synthesis, we use a latent diffusion model [31] with a downsampling
rate of $f=8$, so that an input image of size $512\times 512$ is downsampled
to $64\times 64$. We use Stable Diffusion 1.5 as the backbone for ControlNet
[48] and finetune ControlNet for 200 epochs with a batch size of 16 using two
A100 80GB GPUs. We set $T=1000$ discrete timesteps for ControlNet and use a
linear learning rate scheduler from an initial rate of $10^{-4}$ during
training. For scribble conditioning, we randomly dropout 10% of scribbles,
replacing them with a learned embedding of the same size. Scribble labels are
represented as RBG images in $\\{1,\ldots,255\\}^{512\times 512\times 3}$. We
also provide the text prompt ”a high-quality, detailed, and professional image
of [list of classes]” for ControlNet, as suggested in [48]. We provide
visualizations of our synthetic dataset in the supplementary material.

Evaluation metric. We evaluate both the diversity and fidelity of the
generated images by the Fréchet Inception Distance (FID) [15], as it is the de
facto metric for the evaluation of generative methods, e.g., [10, 19, 4, 32].
It provides a symmetric measure of the distance between two distributions in
the feature space of Inception-V3 [37]. We use FID as our primary metric for
the sampling quality. We realize, however, that FID should not be the only
metric for evaluating the downstream impact of synthetic data for training
segmentation networks. Hence, we also report segmentation results trained with
synthetic data only to evaluate synthetic data, similar to the Classification
Accuracy Score (CAS) proposed by [30] but for semantic segmentation. We report
the standard mean Intersection Over Union (mIOU) metric for segmentation
results.

###  4.1 Generative data augmentation improves scribble-supervised semantic
segmentation

For our experiments, we consider two methods of weakly-supervised semantic
segmentation, including simple regularized losses (RLoss) [38] and the current
state-of-the-art in scribble-supervised segmentation, Adaptive Gaussian
Mixture Models (AGMM) [41]. For both methods, we jointly train them on the
original training set and our augmented training set. Both methods also follow
a polynomial learning rate scheduler. The sampling of synthetic training
images is outlined in Sec. 3.5. Table 1 shows improved results using
generative data augmentation for both RLoss and AGMM. Our method with
synthetic data further reduces the gap between weakly-supervised and fully-
supervised segmentation. We show visualizations of our segmentation results
with and without using our generative data augmentation in Fig. 6. We also
include further visualizations in the supplementary material.

Method | Network | Supervision | Synthetic Data | Augmentation Scheme | mIoU (%)
---|---|---|---|---|---
(1) *DeeplabV3+ [7] | MobileNet [33] | Full mask |  | – | 72.1
(2) *DeeplabV3+ [7] | ResNet101 [14] | Full mask |  | – | 79.3
RLoss [38] | (1) | Scribble |  | – | 68.4
RLoss [38] | (1) | Scribble | ✓ | Uniform $\lambda$ | 69.6
RLoss [38] | (2) | Scribble |  | – | 76.6
RLoss [38] | (2) | Scribble | ✓ | Uniform $\lambda$ | 76.7
AGMM [41] | (2) | Scribble |  | – | 76.4
AGMM [41] | (2) | Scribble |  | – | 78.1
AGMM [41] | (2) | Scribble | ✓ | Fixed $\lambda=1.0$ | 78.0
AGMM [41] | (2) | Scribble | ✓ | Uniform $\lambda$ | 78.6
AGMM [41] | (2) | Scribble | ✓ | Adaptive $\lambda$ | 78.7
AGMM [41] | (2) | Scribble | ✓ | Fixed $\lambda=0.4$ | 78.9
Table 1: Generative data augmentation improves scribble-supervised semantic
segmentation methods including RLoss [38] and AGMM [41] on PascalVOC [12]. The
best results are shown in bold. Note that for AGMM, * is our re-implementation
which gives better results than reported [41].

###  4.2 Low-data Regime Results

For the low-data regime, we only consider the RLoss method due to its
simplicity and speed to train. We consider three different reduced datasets
with 50%, 25%, and 12.5% of all training images used, respectively. For each
of these cases, we train a ControlNet diffusion model on the limited dataset
(following the same experimental setup described at the start of Sec. 4) and
sample synthetic images as usual. The results of training RLoss on each of the
reduced datasets for our different proposed augmentation schemes are reported
in Fig. 1.

We notice that the naive data augmentation fails to help in all of our reduced
datasets and instead reduces model performance in all but the 50% case.
Conversely, our proposed Adaptive $\lambda$ Sampling improves or matches
performance for all four datasets. We hypothesize this is due to the lack of
training images required to ensure high-quality generation from our diffusion
model. This hypothesis is confirmed by the significantly higher FID scores for
synthetic datasets generated with limited training data reported in Fig. 5
middle. We also confirm this hypothesis qualitatively in Fig. 4, where we
observe that fully synthetic images deteriorate in quality as the number of
training images decreases. However, we can stabilize this deterioration by
decreasing the encode ratio $\lambda$ to improve image realism. Using our
Adaptive $\lambda$ sampling, the most synthetic (and thus lowest quality)
images cannot impact model training as significantly due to the reduced
learning from our scheduler.

Figure 4: Synthetic images sampled from diffusion models with different
numbers of training images and encode ratios $\lambda$.

###  4.3 Ablation Studies

Figure 5:  Left: The FID of our full training dataset when generated with different classifier-free guidance scales. Results are reported for ControlNet trained all 10582 images. Middle: The FID of our training dataset when generated with different encode ratios. Results are reported for four ControlNet models trained on a different number of images. Right: The mIoU of a downstream segmentation model when trained on datasets of varying encode ratios. Note $\lambda=0.0$ corresponds to training on real images only. Results are reported for training on both naive data augmentation and only on synthetic images. In both cases, we use all 10582 images for training.  image | AGMM [41] | Fully-supervised | AGMM (+syn. images) | Ground truth
---|---|---|---|---
|  |  |  |
|  |  |  |
Figure 6: Qualitative results on PASCAL dataset. Our generative data
augmentation method improves scribble-supervised semantic segmentation method
such as AGMM [41].

#### Guidance Scale

We report the FID scores of our fully synthetic ($\lambda=1$) datasets as
generated by our model trained on all PascalVOC training images in Fig. 5
left. This ablation study is how we decided to use $w=2$ for all other
experiments, as it yields optimal FID. We include further visualizations of
the impact of the guidance scale on image synthesis in our supplementary
material.

#### Encode Ratio

We report the FID scores of our diffusion models trained with a variable
number of images as a function of the encode ratio $\lambda$ in Fig. 5 middle.
We observe that the FID increases significantly as the number of training
images decreases. However, we can reduce the effect of limited training data
by decreasing the encode ratio to promote image realism. This effect is most
pronounced for the 1323 image-trained diffusion model, where we reduce the FID
score by over 30 points by lowering the encode ratio.

We also evaluate segmentation model performance on synthetic data of varying
encode ratios and report the final mIoU in Fig. 5 right. For these
experiments, we train segmentation models training using the Fixed $\lambda$
data augmentation proposed in Sec. 3.5 and training exclusively on synthetic
training data (i.e., $\mathcal{X}^{\prime}=\hat{\mathcal{X}}$), akin to CAS
[30]. We observe that the impact of varying the encode ratio $\lambda$ is
limited in the data augmentation case but much more significant for the
synthetic-only case. We suppose that for the synthetic-only case, the quality
of the synthetic images is more important, so decreasing the encode ratio to
improve data realism matters more than data diversity. We include further
visualizations of the impact of the encode ratio on image synthesis in our
supplementary material.

#### Conditioning Input

We also ablate modifying the conditioning input to ControlNet. We try
representing scribble labels as one-hot embeddings in $\\{0,1\\}^{h\times
w\times C}$ where there are $C$ total classes. Using these one-hot embeddings,
we obtained a higher FID by 4.4 points relative to RGB embeddings, but we
found no improvement in mIoU results using our Fixed $\lambda$ augmentation
scheme. We also try using text prompts that don’t include the classes in the
image. Using unchanging prompts (i.e., “a high-quality, detailed, and
professional image”) yields lower FID by 3.1 points relative to prompts that
include the classes in the image and 1.9% lower mIoU using our Fixed $\lambda$
augmentation scheme.

##  5 Conclusion and Future Work

We propose leveraging diffusion models conditioned on scribbles to produce
high-quality synthetic training data for scribble-supervised semantic
segmentation. We advocate the use of classifier-free guided diffusion and
introduce the encode ratio to control the generative process, allowing us to
generate a spectrum of images. We also propose two novel data augmentation
methods, Uniform and Adaptive $\lambda$ sampling. We report state-of-the-art
performance on scribble-supervised semantic segmentation with our generative
data augmentation.

In the future, it will be interesting to train generative models for open-
vocabulary image synthesis conditioned on sparse annotations. Our generative
data augmentation has the potential to improve semi-supervised segmentation.
We are also interested in end-to-end training of generative data augmentation
and perception models, as metrics like FID are loosely related to perception
performances.

#### Acknowledgement

The authors would like to thank Prof. Yuri Boykov, Prof. Olga Veksler, and
Prof. Ming-Hsuan Yang for their helpful discussion and comments that improved
the quality of this work.

## References

  * Azizi et al. [2023] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J Fleet.  Synthetic data from diffusion models improves imagenet classification.  _arXiv preprint arXiv:2304.08466_ , 2023.
  * Bearman et al. [2016] Amy Bearman, Olga Russakovsky, Vittorio Ferrari, and Li Fei-Fei.  What’s the point: Semantic segmentation with point supervision.  In _European conference on computer vision_ , pages 549–565. Springer, 2016.
  * Bowles et al. [2018] Christopher Bowles, Liang Chen, Ricardo Guerrero, Paul Bentley, Roger N. Gunn, Alexander Hammers, David Alexander Dickie, Maria del C. Valdés Hernández, Joanna M. Wardlaw, and Daniel Rueckert.  GAN augmentation: Augmenting training data using generative adversarial networks.  _CoRR_ , abs/1810.10863, 2018.
  * Brock et al. [2019] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  In _ICLR_ , 2019.
  * Chang et al. [2020a] Yu-Ting Chang, Qiaosong Wang, Wei-Chih Hung, Robinson Piramuthu, Yi-Hsuan Tsai, and Ming-Hsuan Yang.  Mixup-cam: Weakly-supervised semantic segmentation via uncertainty regularization.  _CoRR_ , abs/2008.01201, 2020a.
  * Chang et al. [2020b] Yu-Ting Chang, Qiaosong Wang, Wei-Chih Hung, Robinson Piramuthu, Yi-Hsuan Tsai, and Ming-Hsuan Yang.  Weakly-supervised semantic segmentation via sub-category exploration.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8991–9000, 2020b.
  * Chen et al. [2018] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  In _PAMI_ , 2018.
  * Dehghani et al. [2023] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Anurag Arnab, Xiao Wang, Carlos Riquelme, Matthias Minderer, Joan Puigcerver, Utku Evci, Manoj Kumar, Sjoerd van Steenkiste, Gamaleldin F. Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Patrick Collier, Alexey Gritsenko, Vighnesh Birodkar, Cristina Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov, Filip Pavetić, Dustin Tran, Thomas Kipf, Mario Lučić, Xiaohua Zhai, Daniel Keysers, Jeremiah Harmsen, and Neil Houlsby.  Scaling vision transformers to 22 billion parameters, 2023.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  ImageNet: A large-scale hierarchical image database.  In _CVPR_ , 2009.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ , 34:8780–8794, 2021.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  _CoRR_ , abs/2010.11929, 2020.
  * Everingham et al. [2010] Mark Everingham, Luc Van Gool, Christopher K. I. Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (VOC) challenge.  In _IJCV_ , 2010.
  * Frid-Adar et al. [2018] Maayan Frid-Adar, Eyal Klang, Michal Amitai, Jacob Goldberger, and Hayit Greenspan.  Synthetic data augmentation using gan for improved liver lesion classification.  In _2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018)_ , pages 289–293, 2018.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _CVPR_ , 2016.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In _NeurIPS_ , 2017.
  * Ho and Salimans [2021] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  In _NeurIPS Workshop_ , 2021.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Hu et al. [2023] Tao Hu, David W Zhang, Yuki M. Asano, Gertjan J. Burghouts, and Cees G.M. Snoek.  Self-guided diffusion models.  In _CVPR_ , 2023.
  * Karras et al. [2019] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In _CVPR_ , 2019.
  * Kattakinda et al. [2022] Priyatham Kattakinda, Alexander Levine, and Soheil Feizi.  Invariant learning via diffusion dreamed distribution shifts, 2022.
  * Ke et al. [2021] Tsung-Wei Ke, Jyh-Jing Hwang, and Stella Yu.  Universal weakly supervised segmentation by pixel-to-segment contrastive learning.  In _International Conference on Learning Representations_ , 2021.
  * Khoreva et al. [2017] Anna Khoreva, Rodrigo Benenson, Jan Hosang, Matthias Hein, and Bernt Schiele.  Simple does it: Weakly supervised instance and semantic segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 876–885, 2017.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  _arXiv preprint arXiv:2304.02643_ , 2023.
  * Li et al. [2022] Daiqing Li, Huan Ling, Seung Wook Kim, Karsten Kreis, Sanja Fidler, and Antonio Torralba.  Bigdatasetgan: Synthesizing imagenet with pixel-wise annotations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 21330–21340, 2022.
  * Liang et al. [2022] Zhiyuan Liang, Tiancai Wang, Xiangyu Zhang, Jian Sun, and Jianbing Shen.  Tree energy loss: Towards sparsely annotated semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16907–16916, 2022.
  * Lin et al. [2016] Di Lin, Jifeng Dai, Jiaya Jia, Kaiming He, and Jian Sun.  Scribblesup: Scribble-supervised convolutional networks for semantic segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3159–3167, 2016.
  * Liu et al. [2023] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley.  Audioldm: Text-to-audio generation with latent diffusion models.  _arXiv preprint arXiv:2301.12503_ , 2023.
  * Meng et al. [2021] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Guided image synthesis and editing with stochastic differential equations.  In _ICLR_ , 2021.
  * Nguyen et al. [2023] Quang Nguyen, Truong Vu, Anh Tran, and Khoi Nguyen.  Dataset diffusion: Diffusion-based synthetic dataset generation for pixel-level semantic segmentation, 2023.
  * Ravuri and Vinyals [2019] Suman V. Ravuri and Oriol Vinyals.  Classification accuracy score for conditional generative models.  _CoRR_ , abs/1905.10887, 2019.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding, 2022.
  * Sandler et al. [2018] Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation.  _CoRR_ , abs/1801.04381, 2018.
  * Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _ICML_ , 2015.
  * Song et al. [2021a] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In _ICLR_ , 2021a.
  * Song et al. [2021b] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In _ICLR_ , 2021b.
  * Szegedy et al. [2016] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In _CVPR_ , 2016.
  * Tang et al. [2018] Meng Tang, Federico Perazzi, Abdelaziz Djelouah, Ismail Ben Ayed, Christopher Schroers, and Yuri Boykov.  On regularized losses for weakly-supervised cnn segmentation.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 507–522, 2018.
  * Tevet et al. [2022] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano.  Human motion diffusion model.  _arXiv preprint arXiv:2209.14916_ , 2022.
  * Unal et al. [2022] Ozan Unal, Dengxin Dai, and Luc Van Gool.  Scribble-supervised lidar semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2697–2707, 2022.
  * Wu et al. [2023a] Linshan Wu, Zhun Zhong, Leyuan Fang, Xingxin He, Qiang Liu, Jiayi Ma, and Hao Chen.  Sparsely annotated semantic segmentation with adaptive gaussian mixtures.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15454–15464, 2023a.
  * Wu et al. [2023b] Weijia Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu, Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, and Chunhua Shen.  Datasetdm: Synthesizing data with perception annotations using diffusion models, 2023b.
  * Xie et al. [2023] Jiahao Xie, Wei Li, Xiangtai Li, Ziwei Liu, Yew Soon Ong, and Chen Change Loy.  Mosaicfusion: Diffusion models as data augmenters for large vocabulary instance segmentation.  _arXiv preprint arXiv:2309.13042_ , 2023.
  * Xu et al. [2022] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang.  Groupvit: Semantic segmentation emerges from text supervision.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18134–18144, 2022.
  * Xu et al. [2017] Ning Xu, Brian Price, Scott Cohen, Jimei Yang, and Thomas Huang.  Deep grabcut for object selection.  _arXiv preprint arXiv:1707.00243_ , 2017.
  * Xue et al. [2023] Han Xue, Zhiwu Huang, Qianru Sun, Li Song, and Wenjun Zhang.  Freestyle layout-to-image synthesis, 2023.
  * Yang et al. [2023] Lihe Yang, Xiaogang Xu, Bingyi Kang, Yinghuan Shi, and Hengshuang Zhao.  Freemask: Synthetic images with dense annotations make stronger segmentation models.  In _NeurIPS_ , 2023.
  * Zhang et al. [2023a] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3836–3847, 2023a.
  * Zhang et al. [2023b] Manlin Zhang, Jie Wu, Yuxi Ren, Ming Li, Jie Qin, Xuefeng Xiao, Wei Liu, Rui Wang, Min Zheng, and Andy J Ma.  Diffusionengine: Diffusion model is scalable data engine for object detection.  _arXiv preprint arXiv:2309.03893_ , 2023b.
  * Zhang et al. [2021] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler.  Datasetgan: Efficient labeled data factory with minimal human effort.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10145–10155, 2021.
  * Zhao and Bilen [2022] Bo Zhao and Hakan Bilen.  Synthesizing informative training samples with gan, 2022.
  * Zhou et al. [2016] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.  Learning deep features for discriminative localization.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2921–2929, 2016.

Supplementary Material

##  6 Validation Set FID Results

In this section, we report the FID of our synthetic images on the validation
set. To achieve this, we provide scribbles from the validation set of
PascalVOC as conditioning input to our trained ControlNet models. Since the
ControlNet models were not trained with data from the validation set, these
are previously unseen scribbles. In Table 2, we report the impact of the
number of training images of the ControlNet model on validation FID. In Fig.
7, we report the impact of the encode ratio on validation FID.

Training Images | FID
---|---
10582 | 43.3
5291 | 53.7
2646 | 57.1
1323 | 58.0
Table 2: FID reported on the validation set. Synthetic images are from our
ControlNet model with a varying number of training images. Synthetic images
are conditioned on scribbles from the validation set, previously unseen to our
model. Figure 7: FID reported on the validation set. Synthetic images are from
our ControlNet model trained on all of PascalVOC. Images are synthesized
conditioned on scribbles from the validation set with varying encode ratios.

##  7 Additional Qualitative Results

In this section, we include additional qualitative results of our method. We
provide additional samples of our training data in Fig. 8 and samples from
previously unseen scribbles from the validation set in Fig. 9. We also provide
visualizations of the effect of the guidance scale on synthesis in Fig. 10 and
the effect of the encode ratio in Fig. 11. The effect of the number of
training images on synthesis is demonstrated in Fig. 12. Finally, we provide
further visualizations of segmentation results in Fig. 13.

Figure 8: Synthetic training images sampled from a ControlNet model trained on all of scribble-supervised PascalVOC. All images are sampled using guidance scaled $w=2.0$ and encode ratio $\lambda=1.0$. Figure 9: Synthetic validation images sampled from a ControlNet model trained on all of scribble-supervised PascalVOC. Images are synthesized conditioned on scribbles from the validation set, which the ControlNet model has not been trained on. All images are sampled using guidance scaled $w=2.0$ and encode ratio $\lambda=1.0$. Figure 10: Synthetic training images sampled from a ControlNet model trained on all of scribble-supervised PascalVOC. We vary the guidance scale but keep the encode ratio $\lambda=1.0$ constant to see the effect of the guidance scale on synthesis. Figure 11: Synthetic training images sampled from a ControlNet model trained on all of scribble-supervised PascalVOC. We vary the encode ratio but keep the guidance scale $w=2.0$ constant to see the effect of the encode ratio on synthesis. Figure 12: Synthetic training images sampled from a ControlNet model. We vary the number of images on which the ControlNet model is trained to see the impact of the number of training images on synthesis. All images are sampled using guidance scaled $w=2.0$ and encode ratio $\lambda=1.0$. image |  AGMM [41] | Fully-supervised | AGMM (+syn. images) | ground truth
---|---|---|---|---
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
Figure 13: Qualitative results on PASCAL dataset. Our generative data
augmentation method improves scribble-supervised semantic segmentation method
such as AGMM [41].