# PowerNet: Transferable Dynamic IR Drop Estimation via Maximum Convolutional
Neural Network

Zhiyao Xie1 Haoxing Ren2 Brucek Khailany2 Ye Sheng2 Santosh Santosh2 Jiang Hu3
Yiran Chen1

###### Abstract

IR drop is a fundamental constraint required by almost all chip designs.
However, its evaluation usually takes a long time that hinders mitigation
techniques for fixing its violations. In this work, we develop a fast dynamic
IR drop estimation technique, named PowerNet, based on a convolutional neural
network (CNN). It can handle both vector-based and vectorless IR analyses.
Moreover, the proposed CNN model is general and transferable to different
designs. This is in contrast to most existing machine learning (ML)
approaches, where a model is applicable only to a specific design.
Experimental results show that PowerNet outperforms the latest ML method by 9%
in accuracy for the challenging case of vectorless IR drop and achieves a
$30\times$ speedup compared to an accurate IR drop commercial tool. Further, a
mitigation tool guided by PowerNet reduces IR drop hotspots by 26% and 31% on
two industrial designs, respectively, with very limited modification on their
power grids.

##  I Introduction

Dynamic IR drop is the deviation of the power supply level from its
specification caused by localized power demand and switching patterns. It must
be restricted in order for a circuit to meet its timing target and function
properly. As such, it is vitally important to verify if IR drop satisfies
design constraints and identify constraint violation regions, a.k.a. hotspots.
As chip complexity continues to grow, IR drop evaluation becomes increasingly
challenging.

In industrial designs, dynamic IR drop estimation is often obtained by running
simulation-based commercial tools, which are known to be accurate but very
time-consuming. Machine learning (ML)-based approaches have been explored in
an effort to achieve faster estimation. Many of these previous works are
summarized in Table I. These works learn to predict dynamic IR drop of each
cell through features such as cell positions, timing windows, path resistance,
etc. with supervised machine learning techniques.

A major weakness shared by most of the previous works is that they are not
“design independent”, i.e., transferable to new designs that are not seen in
its training dataset. In other words, most of these previous works need to
train a new model for each distinct design. Some work [1] even dedicates one
model for every single cell. Training a new model with new labels entails a
long simulation and training time, which defeats the original purpose of fast
estimation. The only exception is [2], which is based on unsupervised learning
and does not learn any previous knowledge.

In addition, most previous ML approaches to IR drop estimation only focus on
vector-based analysis, ignoring vectorless IR drop. For dynamic IR drop, the
peak IR drop in the design can be analyzed either using vectorless analysis or
vector-based analysis using simulation patterns from value change dump (VCD)
files. Vectorless IR drop analysis is highly desirable for IR mitigation
during physical design for two main reasons. Firstly, for a large chiplet,
vector-based IR drop analysis requires a huge number of simulation patterns to
cover most regions and thus can be unbearably slow. Secondly, designers are
unable to obtain accurate power simulation patterns early in the design
process. For large industrial designs, multiple teams work on different RTL
units in parallel and the overall simulation patterns change throughout the
design process. Vectorless IR drop provides a faster and earlier estimation in
this case, however, accurate estimation is more difficult than vector-based
due to the increased diversity in switching activity distribution. We will
demonstrate the accuracy difference between vector-based and vectorless IR
drop analysis in Section V-E.

TABLE I: Comparison Among Different Works

ML Methods | Model | Design Independent
---|---|---
[1] (ITC 12) | Linear Regression | No
[3] (VTS 14) | SVM | No
[2] (ATS 17) | Clustering | Unsupervised
[4] (VTS 18) | ANN | No
[5] (ICCAD 18) | XGBoost | No
PowerNet | Max-CNN | Yes

Our CNN-based method PowerNet provides a transferable ML model for both
vectorless and vector-based IR drop estimations. We put more emphasis on
vectorless estimation in our experiments, considering its higher difficulty
and usability. PowerNet addresses these challenges by its innovative
preprocessed features and CNN architecture. In previous works [5], the design
dependent features such as coordinates and timing information of each cell are
directly fed into the ML model. Since locations and timing do not directly
cause IR drop, directly fitting a model based on these features would likely
introduce the overfitting problem, making the model inaccurate on unseen
designs. Instead, design-dependent information should be preprocessed to
correlate with IR drop before feeding to ML models. It is known that IR drop
directly correlates with cell power consumption. Therefore, PowerNet carefully
incorporates these design-dependent features into power maps during
preprocessing. It also utilizes an innovative CNN architecture to capture
maximum transient IR drop. The main contributions of our work include:

  * •

We propose PowerNet, an innovative CNN method targeting both vectorless and
vector-based IR drop estimation. It is the first method that claims to perform
design-independent fast IR drop estimations.

  * •

For experiments on both vectorless and vector-based IR drop estimations,
PowerNet outperforms all other ML methods on every tested industrial design.
Especially for vectorless prediction, PowerNet gives a 9% higher accuracy.

  * •

PowerNet is 30$\times$ faster than an accurate simulation-based commercial IR
drop analysis tool.

  * •

An IR drop mitigation tool guided by PowerNet reduces IR drop hotspots by 26%
and 31% on two new industrial designs with very limited modifications to the
power grid.

  * •

We provide a detailed analysis on PowerNet’s mechanism by showing two
representative examples.

##  II Problem Formulation

This work aims at detecting locations of IR drop hotspots. Hotspots are
regions where IR drop is greater than a specified threshold. To estimate IR
drop, every design is tessellated into an array of tiles, each of which is an
$l\times l$ square. The tile size $l$ controls the granularity of our
solution. In this way, a design with the size of $W\times H$ is represented as
a $w\times h$ matrix, where $w=W/l$ and $h=H/l$. The IR drop at each tile is
the mean value of IR drop of all cells within it. Then IR drop for the whole
design is $IR\in\mathbf{R}^{w\times h}$. The ground-truth $IR$ is also
referred to as label in this paper. As for input features, different types of
power dissipation values are calculated for each tile. We refer to each
$w\times h$ power matrix as a power map. Essentially, power maps are the
distribution of power density. PowerNet $F$ tries to give the closest
estimation $F^{*}$ on $IR$ based on all $G$ different power maps
$\\{P_{map1}\,...\,P_{mapG}\\}$.

$\displaystyle F:\\{P_{map1}\in\mathbf{R}^{w\times
h}...\,P_{mapG}\in\mathbf{R}^{w\times h}\\}\rightarrow\mathbf{R}^{w\times h}$
$\displaystyle
F^{*}=\operatorname*{arg\,min}_{F}Loss(F(\\{P_{map1}...\,P_{mapG}\\}),IR).$

##  III Algorithm

###  III-A Feature Extraction

According to Ohm’s Law, excessive IR drop can be caused by either high current
or high resistance. As is typical in state-of-the-art VLSI design, we assume a
uniform power grid in the power delivery network (PDN), which means the
resistance distribution across a whole design is also rather uniform. Thus in
PowerNet, we choose not to spend extra time calculating resistance for each
cell. For designs with a non-uniform PDN, each cell’s power value can be
scaled by its resistance. The influence of resistance is further elaborated in
Section V-D. When resistance is considered consistent, current becomes the
only key issue in IR drop estimation. Since local power consumption is
proportional to local current, PowerNet utilizes cell power as its input
features.

For each cell $c$, we do not exhaust all possible features that seem to be
relevant, which make the model too complex and overfit. Instead, we select
features that prove to provide essential information for IR drop estimation.
Hard macros are not included. Below are the details of all features and the
labels extracted from them:

  * •

Power: Three types of power values are extracted.

    * –

Internal power ($p_{i}$)

    * –

Switching power ($p_{s}$)

    * –

Leakage power ($p_{l}$)

  * •

Signal arrival time: The minimum and maximum signal arrival times to the cell
within a clock cycle.

    * –

Min arrival time ($t_{min}$)

    * –

Max arrival time ($t_{max}$)

  * •

Coordinates: The cell location after placement.

    * –

Min and max x axis ($x_{min},x_{max}$)

    * –

Min and max y axis ($y_{min},y_{max}$)

  * •

Toggle rate: Describes how often output changes with regard to a given clock
input.

    * –

Rate ($r_{tog}$)

  * •

IR drop: The difference between the nominal supply voltage and the actual
voltage arrived at each cell. ($ir$)

All of the above features are scalar values. For these power types, internal
power $p_{i}$ means power dissipated by capacitance internal to each cell;
switching power $p_{s}$ is power dissipated by the load capacitance at the
output of the cell; leakage power $p_{l}$, which is relatively small in the
experiment, is consumed by unintended leakage that does not contribute to
function. Based on these basic power types, we can generate more power
information for each cell:

$\displaystyle p_{sca}=(p_{i}+p_{s})*r_{tog}+p_{l}$ $\displaystyle
p_{all}=p_{i}+p_{s}+p_{l}$

Both $p_{sca}$ and $p_{all}$ reflect the overall power dissipated by cells,
but $p_{sca}$ scales the overall power by toggle rate of each cell. PowerNet
learns to combine the total power from these different sources of power
dissipation.

###  III-B Preprocessing by Decomposition

Figure 1: Space decomposition.

After power is extracted, the IR drop seen at each cell is not just simply
proportional to its own cell power but also depends on its neighborhood due to
both spatial and temporal current distributions. Spatially, local current is
proportional to the sum of power demand of all cells in a local region. Hence,
the power of neighboring cells also contributes to IR drop of the analyzed
cell. We amortize cell power into grid tiles by a space decomposition. This
also motivates us to adopt a CNN model in PowerNet, which is inherently
designed for learning scalable two-dimensional patterns. Even when considering
spatial information, a region with high overall power demand may still not be
IR drop hotspot. This case arises when cells in the region do not switch at
the same time. Such asynchronous switching disperses voltage drop into a
larger timing window. As a result, maximum dynamic IR drop, i.e. the highest-
transient voltage drop, can still be low. PowerNet measures such influence by
time decomposition during preprocessing.

Algorithm 1 shows our preprocessing method. It generates power maps based on
cell information. For each design, two types of power maps are generated. The
first type includes $\\{P_{i}$, $P_{s}$, $P_{sca}$, $P_{all}\\}$. They only go
through spatial decomposition and do not carry timing information. The second
type $\\{P_{t}[\,j]\in\mathbb{R}^{w\times h}\mid j\in[1,N]\\}$ goes through
both a space decomposition and a time decomposition.

Algorithm 1 Preprocessing by Decomposition

Input: Features {$p_{i}$, $p_{s}$, $p_{l}$, $t_{min}$, $t_{max}$, $x_{min}$,
$x_{max}$, $y_{min}$, $y_{max}$, $r_{tog}$} for every cell $c$. Design weight
$W$, height $H$, cell number $C$ and clock cycle $T$. Tile size $l$ and time
window $t$.

Preprocess:

1:$w=W/l$, $h=H/l$, $N=T/t$

2:Set $P_{i}$, $P_{s}$, $P_{sca}$, $P_{all}$ $\in\\{0\\}^{w\times h}$

3:Set $\\{P_{t}[\,j]\in\\{0\\}^{w\times h}\mid j\in[1,N]\\}$

4:for each cell $c\in[1,C]$ do

5: $p_{sca}=(p_{i}+p_{s})*r_{tog}+p_{l}$

6: $p_{all}=p_{i}+p_{s}+p_{l}$

7: $x_{n}=\lfloor(x_{min}/l)\rfloor$, $x_{x}=\lceil(x_{max}/l)\rceil$

8: $y_{n}=\lfloor(y_{min}/l)\rfloor$, $y_{x}=\lceil(y_{max}/l)\rceil$

9: $s=(x_{x}-x_{n})*(y_{x}-y_{n})$

10: Set mask $M\in\\{0\\}^{w\times h}$, $M[x_{n}:x_{x}][y_{n}:y_{x}]=1$

11: $P_{i}$ += $M*p_{i}/s$

12: $P_{s}$ += $M*p_{s}/s$

13: $P_{sca}$ += $M*p_{sca}/s$

14: $P_{all}$ += $M*p_{all}/s$

15: for each int $j\in[1,N]$ do

16: if $t_{min}<j*t$ and $t_{max}>j*t$ then

17: $P_{t}[\,j]$ += $M*p_{sca}/s$

Output: Time-decomposed $\\{P_{t}[\,j]\in\mathbb{R}^{w\times h}\mid
j\in[1,N]\\}$,

Power map $P_{i}$, $P_{s}$, $P_{sca}$, $P_{all}$ $\in\mathbb{R}^{w\times h}$

Figure 2: Time decomposition.

As illustrated in Figure 1, space decomposition (Lines 7 to 14) amortizes cell
power into any grid tiles occupied by the cell. Assume the regular squares are
grid tiles and grey rectangles are cells. $P_{1}$ to $P_{5}$ are cell power.
For the leftmost highlighted tile, its power equals
$P_{1}+P_{2}+P_{3}+P_{4}/3+P_{5}/2$. The long cell with power $P_{4}$ only
contributes one-third of its power to that of the highlighted tile, because
altogether it overlaps with three tiles. Similarly, in line 7 to 14, each cell
contributes $p/s$, where $s$ is the number of overlapping tiles.

Lines 15 to 17 perform time decomposition. Every power map $P_{t}[\,j]$
corresponds to one time instant $j*t$. For each cell at $j*t$, it contributes
power to a corresponding power map $P_{t}[\,j]$ only when $j*t$ falls between
its signal arriving time $[t_{min},t_{max}]$. In other words, only cells that
can possibly switch at that instant are considered. Figure 2 demonstrates the
mechanism. Vertical dashed lines are measured instants $1*t$ to $j*t$, and
horizontal bars are the signal arrival time intervals of cells. In this
example, only cells 1 and 3 will be counted for $P_{t}[\,j]$ and no cells are
counted for $P_{t}[1]$.

Figure 3: PowerNet structure.

###  III-C PowerNet Model

Algorithm 2 PowerNet: Algorithm of Maximum CNN

Input: IR drop label $IR\in\mathbb{R}^{w\times h}$, Power $P_{i}$, $P_{s}$,
$P_{sca}$, $P_{all}$ $\in\mathbb{R}^{w\times h}$, Time-decomposed
$\\{P_{t}[\,j]\in\mathbb{R}^{w\times h}\mid j\in[1,N]\\}$, Input window size
$k=2*k_{h}+1$, $k_{h}$ means half size

Training:

1:function getInput($\,j,x,y$)

2: Stack features $I=\\{P_{i}$, $P_{s}$, $P_{sca}$, $P_{all}$,
$P_{t}[j]\\}\in\mathbb{R}^{w\times h\times 5}$

3: $I_{x,y}=I[x-k_{h}:x+k_{h}+1][y-k_{h}:y+k_{h}+1]\in\mathbb{R}^{k\times
k\times 5}$

4: return $I_{x,y}$

5:end function

6:

7:Initiate CNN model $f$: $\mathbb{R}^{k\times k\times
5}\rightarrow\mathbb{R}$, Loss function $J$

8:for $epoch\in[1,N_{epoch}]$ do

9: for $x\in$ shuffle $([1,w])$, $y\in$ shuffle $([1,h])$ do

10: $o_{max}=0$

11: for each $j\in[1,N]$ do

12: $I_{x,y}=$ getInput $(\,j,x,y)$

13: $o_{j}=f(I_{x,y})$

14: if $o_{max}<o_{j}$ then

15: $o_{max}=o_{j}$

16: *Gradient Descent $f$ -= $\nabla J(o_{max},IR[x,y])$

Output: Trained CNN model $f$: $\mathbb{R}^{k\times k\times
5}\rightarrow\mathbb{R}$

Algorithm 2 shows how PowerNet $F$ handles power maps with its CNN model $f$.
For each training epoch, it iterates through every tile $(x,y)$ in training
designs. For every tile, it crops $k\times k$ input windows surrounding it
from all relevant $w\times h$ power maps by the function getInput.

As shown in Lines 11 to 12 and Figure 3, for all $N$ time-decomposed power
maps $\\{P_{t}[\,j]\in\mathbb{R}^{w\times h}\mid j\in[1,N]\\}$, they are
processed separately by the same CNN model, together with all other common
power maps $P_{i}$, $P_{s}$, $P_{sca}$, $P_{all}$. Hence, the input to the CNN
is $\\{P_{i}$, $P_{s}$, $P_{sca}$, $P_{all}$, $P_{t}[\,j]\\}$ in Line 2. It
results in a total of $N$ CNN outputs $\\{o_{j}\mid j\in[1,N]\\}$. Then, the
maximum output $o_{max}=Max(\\{o_{j}\mid j\in[1,N]\\})$ is the prediction
result for the analyzed tile. This maximum structure highlights the only
instant that leads to the peak IR drop. It guides CNN $f$ to learn such a
pattern.

Details of the CNN model $f$ in PowerNet is shown in Figure 4. There are four
convolutional layers, two pooling layers and two fully connected layers. Size
of convolution kernels is given in parentheses. The $C$ under each tensor
gives the number of kernels defined in each convolutional layer. This CNN
structure and hyperparameters like $N$, $k$ are tuned based on the performance
during cross-validation. Choosing larger input $k$, more layers or kernels
turns out to reduce model generalization and slow down the prediction, while a
simpler structure underfits the data. Batch normalization (BN) [6] is applied
to accelerate model convergence. Adam method [7] is used for optimization. We
adopt the mean absolute error between prediction and label (L1 loss) as loss
function.

Figure 4: CNN structure.

##  IV Experimental Results

###  IV-A Experiment Setup

TABLE II: Designs Used in Experiment

Design | D1 | D2 | D3 | D4 | MD1 | MD2
---|---|---|---|---|---|---
# cells (million) | 1.7 | 0.81 | 2.0 | 1.9 | 1.7 | 2.4
Hotspot Portion | 5.6% | 7.7% | 3.1% | 3.1% | 0.65% | 0.50%

In our experiment, we use six industrial designs in a sub-10nm technology node
(Table II) with an IR drop hotspot threshold of $56mV$, 6% of the supply
voltage ($0.94V$). Features and IR drop labels are extracted after clock tree
synthesis (CTS). Though tested at the CTS stage, PowerNet can also be applied
to other stages. We perform vectorless analysis and use results from a
commercial IR drop analysis tool as labels. We train the models and measure
their accuracy on D1 to D4, then mitigate the IR drop of MD1 and MD2 with the
estimation from PowerNet. When testing estimation accuracy on D1 to D4, the ML
model is trained only on data from the other three designs. It ensures that
the tested design is totally unseen to the corresponding model, which
eliminates the possibility of information leakage between the testing and
training datasets.

We implemented CNN and tree-based XGBoost models similar to [5] as a
comparison with PowerNet. Similar to [5], two types of features are extracted,
named cell features and map features. Cell features are one-dimensional and
map features are two-dimensional. For each cell $c$, cell features include its
signal arrival time, coordinates, capacitance, unscaled overall cell power
$p_{all}$, toggle rate $r_{tog}$ and cell type. The current of each cell is
not included because it is not available in our design flow. Since voltages at
different regions are all close to the supply voltage, current can be viewed
as proportional to power. Then, local maps of both unscaled overall power
$p_{all}$ and $r_{tog}$ around it are constructed as its map feature. Notice
that compared with PowerNet, only one type of power $p_{all}$ is used. For the
tree-based XGBoost model, all map features and cell features are directly used
as model input. To fit into XGBoost, the two-dimensional map features are
flattened into one dimension. For the CNN model, map features firstly go
through three convolutional layers, each with 25, 25 and 50 filters. Then, the
output of convolutional layers together with all cell features are fed into
three fully connected layers, each with 512 neurons. A 0.4 dropout rate [8] is
applied. Other hyper-parameters like optimizer or learning rate of baselines
are carefully tuned for their best performance. They are trained and tested on
the same designs as PowerNet for model comparison.

All algorithms are implemented in Python. CNN-related models are built on
PyTorch 1.0 [9]. For PowerNet, we set tile size
$l=1\,$\mathrm{\SIUnitSymbolMicro m}$$, number of measured instants $N=50$,
and an input window size $k=31$ in the experiment. Both training and testing
are implemented on an 8-core CPU machine with 100 GB memory and one NVIDIA
Tesla V100 GPU.

###  IV-B Result Measured in ROC Curve

Figure 5: Comparison of methods by ROC curve. Measured in $1\times 1$ tiles
granularity.

Figure 6: Comparison of methods by ROC curve. Measured in $5\times 5$ tiles
granularity.

Figures 6 and 6 show the performance on different designs when measured in
$1\times 1$ tiles and $5\times 5$ tiles, respectively. Measurement in $5\times
5$ tiles means tessellating both predictions and labels with a larger tile,
whose size is $5l\times 5l$. In this case
$IR\in\mathbf{R}^{\frac{w}{5}\times\frac{h}{5}}$. Sometimes when designers fix
IR drop by performing power grid (PG) enhancements, hotspots shown in
$5l\times 5l$ tile already provide sufficient information. Accuracy is
measured by the area under the ROC curve (AUC $\in[0,1]$). A larger area means
a better accuracy in hotspot identification. PowerNet achieves AUC higher than
0.9 and 0.95 for all designs for the aforementioned two granularity’s,
respectively. On average, for $1\times 1$ tiles, the AUC for CNN, XGBoost and
PowerNet are around 0.83, 0.84 and 0.93. For $5\times 5$ tiles, their AUC are
around 0.86, 0.86 and 0.96. PowerNet’s improvement in accuracy is $9\%$.

Figure 7 shows visualizations of both ground truth and the prediction results
from PowerNet. Only subsets of each design containing IR drop hotspot regions
are displayed. Red color indicates higher values while blue corresponds to
lower values and white means zero values. The white blocks in ground truth are
usually the regions without any cells placed. The comparison shows that
PowerNet can capture most IR drop hotspots.

Figure 7: Visualization of estimation and ground truth.

###  IV-C Results Measured in Error and Ranking

Figure 8: MSE and Kendall ranking coefficient on tiles by IR drop.

Besides ROC curves, which reflect how well models recognize hotspots, we also
measured how models fit and rank tiles according to their IR drop values in
Figure 8. The metrics are mean squared error (MSE) and Kendall rank
coefficient [10] $\tau\in[-1,1]$ between the estimation and ground-truth IR
values for all tiles. A higher value of $\tau$ implies a more accurate ranking
of tiles based on IR drop. The MSE and rank coefficients of PowerNet are
consistently better than those of other ML methods. Note that a high MSE may
be largely contributed by a consistent bias for all inferenced tiles, which
means the model always gives a higher or lower prediction for all tiles in one
design. In this case, it can still identify those higher-IR tiles or most
serious hotspots if it ranks the IR value of tiles accurately.

###  IV-D Inference Time Comparison

TABLE III: Inference Time Comparison.

Method | Commercial Tool | PowerNet | CNN | XGBoost
---|---|---|---|---
Time | 2.5 hour | 5 min | 1.5 min | 1.5 min

The runtimes of the commercial IR drop analysis tool and ML inferences are
measured on a design with around two million cells. Results are shown in Table
III. PowerNet achieves a $30\times$ speedup over the commercial tool. For a
fair comparison, the 2.5 hour for the commercial tool only includes analysis
time. Its overall runtime is more than 4 hours. Other ML methods are even
faster than PowerNet, but are less accurate. PowerNet is slower than the
baseline ML methods because its CNN $f$ generates $N$ outputs $o_{j}$ for each
tile.

###  IV-E IR Drop Mitigation in Design Flow

We also integrated PowerNet into a design flow to mitigate the IR drop of MD1
and MD2. Based on PowerNet’s estimation, we enhanced the local power grid (PG)
in hotspot regions. Notice that the hotspot portions of MD1 and MD2 are much
lower than D1 to D4 in the training set. This is because MD1 and MD2 were
already close to tapeout and most serious IR drop problems were already fixed,
making further IR drop mitigation even more challenging.

Table IV shows the IR drop mitigation result. We only add very thin PG straps
(0.04 $\mathrm{\SIUnitSymbolMicro m}$) at the PowerNet-estimated hotspots.
This is the simplest and most basic fixing method. We choose such conservative
fixing method to prevent occupying too many routing resources. “All IR” and
“Hotspot IR” mean the averaged IR drop values among all tiles and all
hotspots. After PG enhancement, the averaged IR drop for all tiles improves
only 0.4 $\mathrm{mV}$, which indicates that the modification on PG is very
small. In comparison, when measured only on hotspots, IR drop improves 4.3
$\mathrm{mV}$ and 2.6 $\mathrm{mV}$. It shows that PG enhancement is effective
at the right places. With such a limited amount of modification in PG, 23% of
IR drop violation cells or around 30% of hotspots are mitigated.

TABLE IV: Performance on IR Drop Mitigation

Design MD1 | Violated | # | All | Hotspot
---|---|---|---|---
Cell | Hotspots | IR (mV) | IR (mV)
Before Mitigate | 22185 | 5092 | 26.4 | 66.6
After Mitigate | 17052 | 3778 | 26.0 | 62.3
Improvement | 23% | 26% | 0.4 | 4.3
Design MD2 | Violated | # | All | Hotspot
Cell | Hotspots | IR (mV) | IR (mV)
Before Mitigate | 31097 | 3627 | 31.4 | 62.2
After Mitigate | 23941 | 2489 | 31.0 | 59.6
Improvement | 23% | 31% | 0.4 | 2.6

##  V Discussion

###  V-A PowerNet vs. Previous ML Models

Figure 9: Effect of number of instants $N$ on performance.

We highlight four weaknesses of previous CNN and XGBoost baseline models that
prevent them from outperforming PowerNet. First, unnecessary features can
confuse ML models. If cell coordinates and time information are used as
features but do not directly correlate with IR drop, a model can overfit to
designs in the training set. Other features such as cell capacitance can be
redundant when power is already provided. To verify this, we implemented an
XGBoost model without cell coordinates or time information, and its averaged
$1\times 1$ tiles AUC improved from 0.84 to 0.865. When we further removed
cell capacitance from features, the averaged AUC remained at 0.865. Second,
different feature formats make the model inefficient. Notice that cell
features are one dimensional but map features are two dimensional. For
XGBoost, map features must be converted into one dimension, which loses
spatial information. For CNN, cell features must be provided through a fully
connected layer. In such an unusual CNN structure, cell features tend to be
overwhelmed by more than 10,000 outputs from the 50 channels in the last
convolutional layer. In comparison, PowerNet only uses two-dimensional
features. Third, power information may not be fully utilized. When only
overall power $P_{all}$ is chosen as a feature, the rich information from
other power types $P_{i},P_{s},P_{sca}$ is lost. Advanced ML models like CNN
are complex enough to learn patterns from different power types. Fourth, time
information is not well incorporated or captured. Baselines do not have
features like the time-decomposed power maps in PowerNet to measure the worst
transient local IR drop.

Figure 9 isolates the contribution of including both time decomposition and
multiple power types in variations of PowerNet. Average inference AUC accuracy
over D1 to D4 is plotted on the Y-axis and the X-axis shows the number of
sampled time instants. A higher $N$ means sampling more time instants and
generating more corresponding power maps $P_{t}[\,j]$ within a given clock
period $T$. For any region, more power maps better approximate its actual
transient power. The “N=0” indicates no time-decomposed power is adopted at
all. As expected, the time-decomposed power maps improve accuracy by capturing
transient IR drop. When $N>50$, the improvement in accuracy by increasing $N$
diminishes. Baseline models also differ from PowerNet by only using maps of
features ($P_{all}$ and $r_{tog}$) instead of $\\{P_{i}$, $P_{s}$, $P_{sca}$,
$P_{all}\\}$. This variation is indicated as red and green marks in Figure 9,
where time-decomposed power maps $P_{t}[\,j]$ are kept the same for both
variations. In addition to the $2.5\%$ accuracy improvement from time
decomposition, adopting multiple types of power improves accuracy by more than
$2\%$.

###  V-B Time Decomposition Mechanism

Figure 10: IR drop, power maps and maximum instant distribution of two regions
from D1. Instants number $N=50$.

Figure 11: IR drop, power maps and maximum instant distribution of three
regions from D2. Instants number $N=50$.

Figures 11 and 11 show how the combination of space decomposition and time
decomposition helps to explore the potential correlation between power maps
and IR drop. It presents the visualization of $IR$, different power maps and
maximum instant distribution for the local regions from both D1 and D2.
Maximum instant refers to the time instant $j$ selected by maximum structure
($o_{j}=o_{max}$). The areas of interest are highlighted by black squares.
They all contain strong IR drop hotspots. For both D1 and D2, it’s difficult
to observe much correlation in hotspots between their $\\{P_{all}$,
$P_{sca}\\}$ and $IR$. That indicates training models without any time-
decomposed power maps $P_{t}[\,j]$ can be difficult.

However, the correlation becomes much more clear when power maps $P_{t}[\,j]$
are provided. In D2, $P_{t}[40]$ and $IR$ share many common hotspots patterns
in highlighted areas. In this case, $o_{40}=f(\\{P_{i}$, $P_{s}$ $P_{sca}$,
$P_{all}$, $P_{t}[40]\\})$ is likely to accurately predict these common
hotspot regions. However, another power map, $P_{t}[20]$, does not share as
much hotspot patterns with $IR$. Its output $o_{20}$ may be less accurate.
Considering that $P_{t}[20]$ is much weaker than $P_{t}[40]$ for most tiles in
hotspot regions, we can reasonably assume $o_{40}>o_{20}$, or even
$o_{max}=o_{40}$. This is verified by the maximum instant distribution. For
every tile, we checked which instant is selected by the maximum structure. For
almost all tiles at hotspot regions in D2, their
$o_{max}\in\\{o_{35}...o_{40}\\}$. $P_{t}[40]$ indeed contributes more
information than $P_{t}[20]$. It is the contribution of the more accurate
$P_{t}[40]$ instead of $P_{t}[20]$ at these hotspot regions that finally gets
captured by the maximum structure.

Similarly, for D1, the highlighted region on the right correlates well with
$P_{t}[20]$, and region on the left correlates with $P_{t}[27]$. Maximum
instant distribution shows that $o_{max}\in\\{o_{18}...o_{21}\\}$ for most
grids in the right region and $o_{max}\in\\{o_{24}...o_{27}\\}$ for most tiles
in the left region. Then the maximum structure will take $o_{t}[20]$ for grids
on the right and $o_{t}[27]$ for tiles on the left. In this way, the hotspots
caused by transient power at different instants can all be captured.

###  V-C Training Accuracy

Table V shows the training accuracy for three ML methods. XGBoost shows a
higher training accuracy than the CNN baseline, consistent with its better
design-dependent performance in previous work [5]. PowerNet provides the
highest training and inference accuracy among all ML models.

TABLE V: Training Accuracy in ROC AUC (0.01*)

ML Methods |  $1\times 1$ tiles |  $5\times 5$ tiles
---|---|---
D1 | D2 | D3 | D4 | D1 | D2 | D3 | D4
XGBoost | 93 | 94 | 94 | 94 | 97 | 98 | 98 | 97
CNN | 87 | 79 | 80 | 85 | 92 | 85 | 83 | 90
PowerNet | 94 | 98 | 95 | 94 | 98 | 99 | 98 | 98

###  V-D Influence of Resistance

TABLE VI: Inference Accuracy in ROC AUC (0.01*)

ML Methods | D1 | D2 | D3 | D4 | Ave
---|---|---|---|---|---
PowerNet ($1\times 1$ tiles) | 92.1 | 95.4 | 91.4 | 92.6 | 92.9
PRNet ($1\times 1$ tiles) | 92.4 | 95.5 | 90.5 | 93.6 | 93.0
PowerNet ($5\times 5$ tiles) | 95.4 | 96.7 | 94.8 | 97.0 | 96.0
PRNet ($5\times 5$ tiles) | 95.7 | 96.8 | 93.2 | 97.5 | 95.8

We measured the distribution of resistance in our benchmark design. Take D1
for example, the standard deviation in resistance across the whole design is
only 2.8$\mathrm{\SIUnitSymbolOhm}$, $0.6\%$ of its average resistance. For
such a uniform distribution, we chose not to spend extra time calculating per-
cell resistance. However, we did implement another variation of PowerNet where
each cell’s power is scaled with resistance, denoted as “PRNet” in Table VI.
“Ave” means accuracy averaged over all four designs. On average, the
resistance-scaled PowerNet shows similar accuracy to the original one. This
demonstrates that using per-cell resistance as a feature is not necessary for
designs with uniform PDNs. By scaling power with resistance, “PRNet” can be
further applied to designs with non-uniform PDNs.

###  V-E Vector-Based IR Drop Estimation

Figure 12: Power and IR drop of vector-based estimation.

We also measured PowerNet’s performance on vector-based IR drop. The PowerNet
architecture remains exactly the same, but cell power and IR drop are now
collected when the commercial tool simulates IR drop with given simulation
patterns. Figure 12 shows the vector-based power map $P_{all}$ and label $IR$.
Unlike the vectorless case in Figure 11 or 11, the power of a portion of
activated cells is significantly higher than the others. As we mentioned, the
correlation between power and IR drop value turns out to be very strong, which
largely reduced the estimation difficulty.

We perform vector-based estimation on four other industrial designs VD1 to
VD4. All models and procedures are the same as the vectorless case, except
using cell power and IR drop from vector-based simulation. Table VII shows
vector-based estimation accuracy. As expected, all methods provide better
estimation than vectorless scenario. But PowerNet still gives the best
accuracy for every single design. The $1\%$ to $2\%$ improvement should not be
underestimated when accuracy is already very high. To a certain extent,
boosting accuracy from $98\%$ to $99\%$ means reducing half of the errors.

TABLE VII: Vector-Based Inference in ROC AUC (0.01*)

ML Methods |  $1\times 1$ tiles |  $5\times 5$ tiles
---|---|---
VD1 | VD2 | VD3 | VD4 | VD1 | VD2 | VD3 | VD4
XGBoost | 97 | 98 | 98 | 96 | 99 | 97 | 98 | 97
CNN | 96 | 93 | 95 | 95 | 98 | 92 | 97 | 96
PowerNet | 98 | 98 | 99 | 97 | 100 | 98 | 100 | 98

##  VI Conclusion

In this paper, we present a CNN-based dynamic IR drop estimator. Unlike
existing ML works, our model is general and transferable to new designs. We
validate the high accuracy of our approach on multiple industrial designs. It
takes an order of magnitude less estimation time than commercial tools and
significantly outperforms state-of-the-art ML methods in both vector-based and
vectorless IR drop scenarios. The IR drop mitigation tool guided by our model
reduces IR drop by more than 20% with very limited PG modification.

## Acknowledgments

This work is partially supported by Semiconductor Research Corporation Tasks
2810.021 and 2810.022 through UT Dallas’ Texas Analog Center of Excellence
(TxACE).

## References

  * [1] Y. Yamato, T. Yoneda, K. Hatayama, and M. Inoue, “A fast and accurate per-cell dynamic ir-drop estimation method for at-speed scan test pattern validation,” in _2012 IEEE International Test Conference (ITC)_. IEEE, 2012, pp. 1–8.
  * [2] H. Dhotre, S. Eggersglüß, and R. Drechsler, “Identification of efficient clustering techniques for test power activity on the layout,” in _2017 IEEE 26th Asian Test Symposium (ATS)_. IEEE, 2017, pp. 108–113.
  * [3] F. Ye, F. Firouzi, Y. Yang, K. Chakrabarty, and M. B. Tahoori, “On-chip voltage-droop prediction using support-vector machines,” in _2014 IEEE 32nd VLSI Test Symposium (VTS)_. IEEE, 2014, pp. 1–6.
  * [4] S.-Y. Lin, Y.-C. Fang, Y.-C. Li, Y.-C. Liu, T.-S. Yang, S.-C. Lin, C.-M. Li, and E. J.-W. Fang, “IR drop prediction of eco-revised circuits using machine learning,” in _2018 IEEE 36th VLSI Test Symposium (VTS)_. IEEE, 2018, pp. 1–6.
  * [5] Y.-C. Fang, H.-Y. Lin, M.-Y. Sui, C.-M. Li, and E. J.-W. Fang, “Machine-learning-based dynamic IR drop prediction for eco,” in _2018 IEEE/ACM International Conference on Computer-Aided Design (ICCAD)_. IEEE, 2018, pp. 1–7.
  * [6] S. Ioffe and C. Szegedy, “Batch normalization: accelerating deep network training by reducing internal covariate shift,” in _International Conference on Machine Learning (ICML)_ , 2015, pp. 448–456.
  * [7] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _International Conference for Learning Representations (ICLR)_ , 2015.
  * [8] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” _The Journal of Machine Learning Research (JMLR)_ , vol. 15, no. 1, pp. 1929–1958, 2014.
  * [9] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” in _Advances in Neural Information Processing Systems Workshops (NIPS-W)_ , 2017.
  * [10] M. G. Kendall, “Rank correlation methods.” 1948.


# Leveraging Human Salience to Improve Calorie Estimation

Katherine Dearstyne, Alberto Rodriguez

###### Abstract

The following paper investigates the effectiveness of incorporating human
salience into the task of calorie prediction from images of food. We observe a
32.2% relative improvement when incorporating saliency maps on the images of
food highlighting the most calorie regions. We also attempt to further improve
the accuracy by starting the best models using pre-trained weights on similar
tasks of mass estimation and food classification. However, we observe no
improvement. Surprisingly, we also find that our best model was not able to
surpass the original performance published alongside the test dataset,
Nutrition5k [1]. We use ResNet50 and Xception as the base models for our
experiment.

###### Index Terms:

Loss function, pre-training, calorie prediction

##  1 Introduction

As obesity and other food-related illnesses affect a growing proportion of
society, there is an increasing need for individuals to become more aware of
what foods they consume. The science of nutrition is complex and often times
inconclusive, yet the simple formula for managing weight and health can be
boiled down to: calories in = calories out. Since calculating calories for
home cooked meals is often tedious and there is generally little nutrition
information available for food ordered from restaurants, monitoring calorie
consumption can be a difficult task for most people. For these reasons, there
is a need to automatically capture calorie information with as little input
from a user as possible. Therefore, we propose a model that can predict the
number of calories in a dish with just a single image of the plate of food.
However, this is a difficult challenge for many reasons including:

  * •

Calorie calculations are not only dependent on the type of food but also the
volume of the food, which is often hard to detect within a single image.

  * •

Food is taken across many different cameras resulting in differences in
quality, angles, and lighting.

  * •

There are presently very few datasets containing images of food items and
their calorie counts.

In order to overcome these challenges, a model would have to learn to extract
features that are agnostic to specific food types, background scenery, and
quantity. Previously, we investigated using several pre-training steps, namely
food classification and mass estimation, to enhance the model’s ability to
predict calories. We build off this work by investigating whether using human
saliency maps (HSMs) (shown in Figure 1) in our training process can guide the
model towards features human deem most important. This is inspired by the
success of the CYBORG loss function [2] when distinguishing artifically
generated images of faces from real ones. For the goal of calorie prediction,
we adapt this framework to incorporate human judgements of image regions
containing foods with the highest number of calories.

Therefore, we present several experiments that seek to address our four
primary research questions:

  1. RQ1:

Do HSMs improve the performance of calorie estimation?

  2. RQ2:

Can pre-training tasks improve the performance of these models that use HSMs?

  3. RQ3:

Can an ensemble model composed of the best models improve performance?

  4. RQ4:

How does our best model compare against the performance of the model published
on Nutrition5k (test set)?

Figure 1: Example of an image used for calorie estimation alongside the human
saliency map indicating the most caloric regions of the food.

##  2 Experiment Design

To address our research questions, we perform four different experiments:

  1. 1.

To answer RQ1, we evaluate the effectiveness of the HSMs (described in Section
3) by comparing two model architectures (see Section 2.2) on the calorie
estimation task both with and without the HSMs.

  2. 2.

To answer RQ2, we evaluate if the tasks of food categorization and mass
estimation provide better starting weights than ImageNet for training on the
calorie estimation task with integrated HSMs. Each model architecture is first
trained on the food categorization or mass estimation task, and then the
weights of the best-performing models are used as the initial weights for the
calorie prediction task.

  3. 3.

To answer RQ3, we take the best two models and combine them in an ensemble.
Prefacing our results, this turned out to be two Xception models, with one
trained for mass prediction and the other for calorie prediction but
leveraging HSMs.

  4. 4.

To answer RQ4, we take the best model from all of the research questions and
fine-tune it on Nutrition5k. Up to this research question, our experiments
have been using Nutrition5k as a hold-out test set. However, the original
paper [1] reports their performance after training on 80% of the dataset and
testing on 20%. They have published their train and test splits, thus, we take
our best model and attempt to replicate their work to compare the results.
Namely, for this research question we compare two models on the train split,
one starting from ImageNet weights and the other resulting from the best
performance from our prior experiments.

Below, we outline the data, model architectures, and metrics used in these
experiments.

###  2.1 Data

In total, we use four different datasets containing images of food which are
described in Table I alongside their usage within the experiment. The main
calorie prediction task is conducted with MenuMatch [3] as the training and
validation set while Nutrition5k[1] is used as the hold out test set.
FoodImages [4] is used as the training and validation sets for the task of
food classification. Finally, ECUST [5] is used as the training and validation
sets for the mass estimation task. All training sets consist of 80% of the
available training data and the remaining is used for validation. Note, the
food classification and mass tasks don’t contain test sets because they are
used only as starting weights on the calorie prediction task.

Name |  Description |  Size |  Usage
---|---|---|---
Food Image Classification Data [4] |  Images of food spanning over 101 categories. |  101K |  Food classification train and validation set.
ECUST Food Dataset [5] |  Images of food, their calorie counts and volume |  3K |  Mass estimation train and validation set.
MenuMatch [3] |  Images of food and their calorie counts. |  646 |  Calorie estimation train and validation set.
Nutrition5k [1] |  Images of food, their calorie counts, ingredients and weight. |  5k |  Hold-out test set for calorie estimation.
TABLE I: Datasets used during the pre-training and regular training of our
neural models.

###  2.2 Neural Models

#### ResNET

The residual network model, otherwise known as ResNet, is deep convolutional
model based from the VGG architecture but which leverages skip connects to
jump over some layers – mimicking the biology of our brain. Figure 2 displays
a more details view of the underlying architecture of the model.

Figure 2: ResNet model architecture.

#### Xception

Xception is a convolution neural model based on depthwise separable
convolution layers [6]. Figure 3 described in the detail the layers associated
in this model. This model relies on the assumption that the mapping of cross-
channels correlations and spatial correlations in the feature maps of
convolutional neural networks can be entirely decoupled. Xception is an more
robust and efficient version of the Inception model.

Figure 3: Xception model architecture.

We plan adding a final dense layer to the models to match the number of food
categories spanning those in our pre-training task. Then, for the fine-tuning
task we plan on replacing this layer with a single node predicting the number
of calories present in the image. We plan on using the Keras implementation of
the VGG16 and ResNet models respectively [7, 8].

#### Ensemble

For creating an ensemble of two or more models, using the Keras graph API we
take the output of all the models and concatenate them into a tensor. Then,
the tensor is passed into a 100 neuron dense hidden layer before feeding into
the last dense layer composed of a single neuron for regression.

###  2.3 Metrics

The first pre-training task focuses on predicting what categories of food are
present in an image. Our predictions will be represented by an encoded vector
of size $n$ containing 1 wherever a food category is present and 0 otherwise
where $n$ is the number of food classes spanning our pre-training data. We
plan to use the categorical crossentropy loss function along with log loss as
our training metrics based on common uses in the field [9, 10]. Meanwhile, the
second pre-training task (predicting volume) as well as our primary task
(calorie estimation) are both regression problems. Therefore, we are choosing
to use the Root Mean Squared Error as our training metric. We use the mean
squared error loss function to train the mass prediction task as recommended
by other practitioners [11]. However, we compare the results of using the mean
squared error loss function alone with the results of using the CYBORG loss
function [2] for the calorie classification.

Figure 4: Evaluation metrics for the CYBORG loss function (RQ1).

##  3 CYBORG: Integrating Human Saliency Into Calorie Estimation

This section describes the integration of HSMs into the training process for
calorie estimation. Previously, in the CYBORG loss function [2], saliency maps
were compared against a model’s class activation mappings (CAMs) to penalize
the model for focusing on regions deemed less important to the task by humans.
However, this loss function assumes the task to be a classification problem
whereas calorie prediction is a regression task. Therefore, we adapt the
CYBORG loss function for regression tasks as described below.

To first create the HSMs, we instructed workers on Amazon’s Mechanical Turk to
create bounding boxes around the most caloric sections in the MenuMatch
dataset images [3]. These results are converted into heatmaps by translating
the areas captured by the box bounding into high value pixels and setting the
remaining areas to pixel values of 0. Due to budget constraints by the AWS
team, we were prevented from running the experiment multiple times. Instead we
apply blurring to the bounding box, using a large kernel size (250x250),
operating on the assumption that the center region of the bounding boxes
contains the most important information.

To obtain the model’s saliency mappings (MSMs), we extract the feature maps
from the final convolutional layer of the model as well as their associated
weights in the final dense layer which contains a single neuron for the
regression output. We then calculate a weighted average of these feature maps
to summarize the model’s salient regions for each sample. The HSMs are resized
to match the MSMs (7x7), and then both are normalize to the range of 0 and 1,
using min-max scaling.

Finally, the loss between the MSMs and HSMs as well as the loss from the
predicted and true calories are both calculated using the mean squared error
(MSE). In order to combine these two losses ($L_{total}$), we explore two
different methods. The first method, _cyborg-original_ , uses a weighted sum
(see Equation 1), with equal weight on both the calorie and heatmap loss
(alpha=0.5).

$L_{total}=(1-\alpha)*L_{m}-\alpha*L_{c}$ (1)

where $L_{m}$ is the loss between the saliency maps, $L_{c}$ is the calorie
loss and $\alpha$ is the weight factor. Since the calorie loss can easily
reach 100K while the heatmap loss remains under 1, the weighted sum heavily
favored the calorie estimation without the benefit of the feature map. The
second method, _cyborg-multipled_ , addresses this issue by multiplying both
losses together as shown in equation 2.

$L_{total}=L_{m}*L_{c}$ (2)

##  4 Results

The section below describes the results of the experiments answering RQ1-4. We
note that for each of these questions the reported MAE is on the entire
Nutrition5k dataset (RQ1-3) or a section of it (RQ4).

Method | Model | MAE | Improvement
---|---|---|---
Baseline | ResNet50 | 321.60 | 0.0%
Baseline | Xception | 376.0 | -16.9%
CYBORG-Regular | ResNet50 | 240.65 | 25.17%
CYBORG-Regular | Xception | 261.16 | 18.79%
CYBORG-multiplied | ResNet50 | 229.67 | 28.59%
CYBORG-multiplied | Xception | 214.05 | 32.18%
TABLE II: The performance of baseline models alongside those using CYBORG loss function. Task | Model | MAE | Improvement
---|---|---|---
Mass Estimation | ResNet50 | 228.13 | 29.06%
Mass Estimation | Xception | 301.09 | 6.38%
Food Classification | ResNet50 | 228.56 | 28.93%
Food Classification | Xception | 255.43 | 25.91%
TABLE III: The performance of models pre-trained on analogous tasks before fine-tuned using the CYBORG loss function. Mass Model | CYBORG Model | MAE | Improvement
---|---|---|---
Xception | Xception (multiplied) | 376.98 | -14.69%
TABLE IV: The performance of the ensemble model (composed of two models) trained on the MenuMatch dataset and tested on the Nutrition5k dataset. _The ensemble model was composed of the CYBORG-multiplied model trained on calorie prediction and the Xception model trained on mass estimation then calorie estimation._ Model | MAE
---|---
Xception (base) | 217.97
CYBORG-multiplied | 225.29
Published Results | 150.8
TABLE V: The performance of a baseline model, CYBORG-multiplied model,
alongside the original results after training and testing on Nutrition5k using
the published sets.

###  4.1 RQ1: Do human saliency maps improve the results of calorie
estimation?

Figure 4 displays the raw results of the 6 different models with the blue bars
representing the baseline methods, the green showing the models with _cyborg-
original_ loss function, and the red highlighting the models using _cyborg-
multipled_ loss. Table II displays the relative improvement of each method
over the best baseline architecture, Xception. We find that the models using
the CYBORG loss function significantly outperformed the baseline models with
an average relative improvement of 25.3% over the best baseline method on the
unseen test set. Specifically, the greatest improvement came from the models
utilizing the adjusted CYBORG function which combined calorie loss with the
feature map loss by multiplying them together.

###  4.2 RQ2: Can pre-training tasks improve the accuracy of these models that
use human saliency maps?

Table III shows the performance of pre-trained models with the _cyborg-
multipled_ loss function used in the fine-tuning step. Although all the
alternatives surpassed the best baseline model (i.e. no pre-training or CYBORG
loss), there is little to no improvement over any of the models using the
CYBORG loss in Experiment 1, and none of the models from Experiment 2
outperform the best model from 1. This signals that starting with ImageNet
weights is not a bad weight initialization scheme. However, we leave for
future work the exploration of why these weights did not help the model
improve its predictions.

###  4.3 RQ3: Can an ensemble model composed of the best models improve
performance?

Table IV shows the performance of an ensemble model composed of two models.
First, an Xception model pre-trained on mass estimation and then on the
MenuMatch dataset for calorie estimation. Second, an Xception model trained on
calorie estimation on the MenuMatch dataset and leveraging the HSMs and the
CYBORG-multiplied loss function. We notice that the ensemble model did not do
better than the best baseline model, which is surprising considering that one
of its models achieved the best performance (CYBORG-multiplied).

###  4.4 RQ4: How does our best model compare against that the performance
published on Nutrition5k (test set)?

Table V displays the performance of three different models after training and
testing on Nutrition5k, which was previously our hold out test set. The first
row, Xception (base) is an Xception model initialized with ImageNet weights
and trained as described. The second row, CYBORG-multiplied is our Xception
model trained on calorie estimation using the Human Saliency Maps (HSMs) and
performing the best on the entire Nutrition5k dataset (zero shot). The final
row is the results published alongside the original Nutrition5k paper [1].
Surprisingly, not only were we not able to perform better than the published
results, but our CYBORG model was not able to surpass the baseline model
trained on only Nutrition5k train set.

##  5 Conclusion

Our results show that incorporating human judgements into a calorie predictor
model can improve results. When using HSMs during training, we achieved as
much as a 32% decrease in MAE on the test data. This is likely because it
forced the model to focus on features which generalized well across different
datasets. Nonetheless, there is still much to explore regarding how best to
harness the power of these human judgements. As we saw in RQ3-4, using the
starting weights of the best HSM model and then performing further training
resulted in a decrease in performance over even the baseline model. One
possible explanation is that once the model was no longer being penalized for
focusing on other regions of the image, it began ”forgetting” (or not
utilizing) these features. This could be explored further by capturing what
areas have the highest activation after additional training or by freezing
some of the layers in order to retain what was learned during training with
the HSMs.

In most of our experiments we saw a much lower MAE (around 122 calories less)
for validation than we did on the test set. This is not surprising since each
dataset contained very different food categories that presented a challenge
for the model. More interesting, however, is that there was about the same
difference in calories between the validation and test set when we were using
Nutrition5k for both. What is surprising is that the model did not outperform
the previous experiments where it was tested on an entirely unseen dataset
with very distinct food categories. One possible explanation could be that
Nutrition5k is one of the few datasets to contain images at wildly different
angles. Perhaps data augmentation with random rotations on the training
dataset could help improve these results.

In conclusion, human salience shows promise for improving calorie predictions
but more exploration needs to be done to establish the method of training that
can result in the best generalization on real-world data. Nonetheless, with an
average error of around 200 calories, the model can still help humans have a
reasonable calorie estimation of their plate of food, and we are optimistic
these estimations can be improved further through our future explorations.

## References

  * [1] Q. Thames, A. Karpur, W. Norris, F. Xia, L. Panait, T. Weyand, and J. Sim, “Nutrition5k: Towards automatic nutritional understanding of generic food,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 8903–8911.
  * [2] A. Boyd, P. Tinsley, K. Bowyer, and A. Czajka, “Cyborg: Blending human saliency into the loss improves deep learning,” 2021. [Online]. Available: https://arxiv.org/abs/2112.00686
  * [3] “Menu-Match: Restaurant-Specific Food Logging from Images.” [Online]. Available: http://neelj.com/projects/menumatch/data/
  * [4] “Food Images (Food-101).” [Online]. Available: https://kaggle.com/kmader/food41
  * [5] Y. Liang and J. Li, “Computer vision-based food calorie estimation: dataset, method, and experiment,” _arXiv preprint arXiv:1705.07632_ , 2017.
  * [6] F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” _arXiv:1610.02357 [cs]_ , Apr 2017, arXiv: 1610.02357. [Online]. Available: http://arxiv.org/abs/1610.02357
  * [7] K. Team, “Keras documentation: VGG16 and VGG19.” [Online]. Available: https://keras.io/api/applications/vgg/
  * [8] ——, “Keras documentation: ResNet and ResNetV2.” [Online]. Available: https://keras.io/api/applications/resnet/
  * [9] A. Mishra, “Metrics to Evaluate your Machine Learning Algorithm,” May 2020. [Online]. Available: https://towardsdatascience.com/metrics-to-evaluate-your-machine-learning-algorithm-f10ba6e38234
  * [10] B. T, “Comprehensive Guide on Multiclass Classification Metrics,” Oct. 2021. [Online]. Available: https://towardsdatascience.com/comprehensive-guide-on-multiclass-classification-metrics-af94cfb83fbd
  * [11] J. Brownlee, “Regression Metrics for Machine Learning,” Jan. 2021. [Online]. Available: https://machinelearningmastery.com/regression-metrics-for-machine-learning/