# Regression on Deep Visual Features using Artificial Neural Networks (ANNs)
to Predict Hydraulic Blockage at Culverts

Umair Iqbal, Johan Barthelemy, Pascal Perez and Wanqing Li Umair Iqbal is with
the SMART Infrastructure Facility, University of Wollongong, Australia,
<EMAIL_ADDRESS>
###### Abstract

Cross drainage hydraulic structures (i.e., culverts, bridges) in urban
landscapes are prone to getting blocked by transported debris which often
results in causing the flash floods. In context of Australia, Wollongong City
Council (WCC) blockage conduit policy is the only formal guideline to consider
blockage in design process. However, many argue that this policy is based on
the post floods visual inspections and hence can not be considered accurate
representation of hydraulic blockage. As a result of this on-going debate,
visual blockage and hydraulic blockage are considered two distinct terms with
no established quantifiable relation among both. This paper attempts to relate
both terms by proposing the use of deep visual features for prediction of
hydraulic blockage at a given culvert. An end-to-end machine learning pipeline
is propounded which takes an image of culvert as input, extract visual
features using deep learning models (i.e., MobileNet, ResNet50,
EfficientNetB3), pre-process the visual features and feed into regression
model (i.e., Artificial Neural Network (ANN)) to predict the corresponding
hydraulic blockage. Dataset (i.e., Hydrology-Lab Dataset (HD), Visual
Hydrology-Lab Dataset (VHD)) used in this research was collected from in-lab
experiments carried out using scaled physical models of culverts where
multiple blockage scenarios were replicated at scale. Performance of
regression models was assessed using standard evaluation metrics including
Mean Squared Log Error (MSLE), Mean Squared Error (MSE), Mean Absolute Error
(MAE) and $R^{2}$ score. Furthermore, performance of overall machine learning
pipeline was assessed in terms of processing times for relative comparison of
models and hardware requirement analysis. From the results ANN used with
MobileNet extracted visual features achieved the best regression performance
with $R^{2}$ score of 0.7855. Positive value of $R^{2}$ score indicated the
presence of correlation between visual features and hydraulic blockage and
suggested that both can be interrelated with each other.

Keywords— Visual Blockage, Hydraulic Blockage, Machine Learning, Deep
Learning, Convolutional Neural Networks (CNNs), Scaled Physical Models,
Artificial Neural Networks (ANNs)

##  1 Introduction

Blockage of cross drainage hydraulic structures such as culverts and bridges
is a commonly occurring phenomena during floods which often results in
increased damages to property and risk to life [1, 2, 3, 4, 5, 6, 7, 8].
However, consideration of blockage into design guidelines is hindered by
highly variable nature of blockage formulation and unavailability of
historical during floods data to investigate the behaviour of blockage [9, 10,
11]. In context of Australia, many councils and institutions have mentioned
blockage as critical issue (e.g., NSW Floodplain Management Manual [12],
Queensland Urban Drainage Manual [13], Australian Rainfall and Runoff (ARR)
[2, 14, 15, 16]), however, none comprehensively addressed consideration of
blockage into design guidelines. Wollongong City Council (WCC) under the
umbrella of ARR developed a conduit blockage policy in 2002 which was revised
many times over the years [17, 15]. This policy was developed based on the
post flood visual surveys of hydraulic structures and suggested that any
hydraulic structure with diagonal length less than 6m is prone to 100%
blockage during peak floods [18, 4, 6, 17].

However, WCC blockage policy was criticized by the hydrology experts because
of its dependence on the visual assessments rather than the hydraulic
assessments. Furthermore, it is argued that the “degree of blockage” in WCC
policy was assessed from the post flood visual inspections which cannot be
considered as true representation of blockage during peak floods [19]. As a
result of this on-going debate, visual blockage and hydraulic blockage are
introduced as two separate terms. Visual blockage is defined as the percentage
occlusion of hydraulic structure by debris assessed from images and/or
personal observations. However, it is not necessary for a structure with high
visual blockage to be hydraulically blocked as well (e.g., culvert blocked
with porous vegetative debris have high visual blockage but may have
insignificant effect on hydraulic capacity of culvert). Hydraulic blockage, on
the other hand is defined as the reduction in the hydraulic capacity of
structure due to presence of debris material [15]. Quantification of hydraulic
blockage at a given structure is highly variable and almost impossible to
achieve. French et al. [19] suggested that unless the visual blockage is
translated into hydraulic blockage, the WCC policy cannot be considered
reliable for including into design guidelines. Furthermore, hydraulic data
from peak floods is needed to better understand the blockage behaviour and its
impact on the floods. Till date, there is no quantifiable relation reported
between visual blockage and hydraulic blockage.

Research in this paper attempts to relate hydraulic blockage with visual
blockage by proposing an end-to-end machine learning pipeline. Aim of this
approach is to take the image of culvert and predict the corresponding
hydraulic blockage. The proposed machine learning pipeline involves the steps
of extraction of visual features from image using deep learning models (i.e.,
MobileNet, ResNet50, EfficientNetB3), pre-processing of the extracted deep
visual features and predicting the hydraulic blockage by feeding it to
regression model (i.e., Artificial Neural Network (ANN)). The dataset (i.e.,
Hydrology-Lab Dataset (HD), Visual Hydrology-Lab Dataset (VHD)) used in this
research was collected from series of comprehensive in-lab experiments
performed using scaled physical models of culverts to replicate different
flooding and blockage scenarios at scale.

Rest of the paper is organized as follows: Section 2 presents the information
about the visual and hydraulic dataset used in this research. Section 3
presents the information about proposed end-to-end machine learning pipeline
and theoretical background of methods used in the pipeline. Section 4 provides
the information about the model hyperparameters and evaluation measures used
to assess the performance of regression models. Section 5 presents the results
of investigation and highlights the important insights extracted from the
results. Section 6 concludes the study by reporting important results,
highlighted trends and potential future directions.

##  2 Dataset

Two different types of datasets (i.e., hydraulic and visual) were used in this
research collected from comprehensive in-lab experiments using scaled physical
models of culverts. Aim of the experiments was to replicate blockage scenarios
using multiple debris type during different flooding conditions at scale and
record both visual and hydraulic data. Percentage hydraulic blockage was
recorded using mathematical formulation proposed by Kramer et al. [9] as given
in Equation 1.

$\text{Percentage Hydraulic Blockage}=\frac{\text{Upstream
WL}_{\text{blocked}}-\text{Upstream WL}_{\text{unblocked}}}{\text{Upstream
WL}_{\text{blocked}}}\times 100$ (1)

Experiments were performed in a 12m $\times$ 0.2m flume with single and double
circular culvert models. Vegetative and urban debris was used at scale to
simulate different blockage scenarios. Figure 1 shows the Two-Dimensional (2D)
schematic diagram of the experimental setup used to collect the dataset. Point
gauge was used to measure the water levels and was placed at 1m distance from
culvert. In total, 173 unique blockage scenarios were simulated while some
scenarios were repeated. Total of 352 hydraulic data samples were recorded
from the experiments to organize in a dataset called Hydrology-Lab Dataset
(HD).

cameraUpstream TankDownstream Tank1m1mUpstream Point GaugeDownstream Point
GaugeCulvert Model

Figure 1: Two-Dimensional Schematic Diagram of Experimental Setup.

In addition to collection of hydraulic data, a web camera-based setup was
established to record the videos of each simulated blockage scenario. The
visual dataset is referred as Visual Hydrology-Lab Dataset (VHD). For this
investigation, images were extracted from VHD for the time instances when the
hydraulic measurements were taken. In total 352 images were extracted from
video clips each representing the visual of culvert for the time instance at
which corresponding hydraulic measurement was taken.

##  3 Methodology

This section provides the information about the proposed end-to-end machine
learning pipeline and theoretical background of methods used in the pipeline.

The proposed end-to-end machine learning pipeline aimed to relate the visual
blockage with hydraulic blockage. It consisted of three main stages (a) visual
feature extraction (b) data pre-processing (c) ANN regression. The proposed
pipeline was designed to take an image of culvert, extract visual features
using deep learning model, pre-process the extracted features and feed into
the ANN regression model to predict the hydraulic blockage. Figure 2 shows the
functional block diagram of the proposed machine learning pipeline.

CNN Feature
---
Extraction
Data
---
Processing
ANN
---
Regression
% Hydraulic
---
Blockage

Figure 2: Functional Diagram of Proposed Machine Learning Pipeline.

###  3.1 Deep Visual Feature Extraction

As a fist step in the pipeline, image of culvert is processed through a deep
CNN model (e.g., MobileNet, ResNet50, EfficientNetB3) to extract the deep
visual features. For this study, three CNN models are compared to assess the
impact of number of visual features extracted and fundamental principle by
which the visual features are extracted. All the CNN models were used with
ImageNet [20] pre-trained weights and as a feature extractor by removing the
top layers. Brief theoretical background of each CNN model is presented as
follows.

####  3.1.1 MobileNet

MobileNets are proposed by Howard et al. [21] and are compactly designed fast
models for real-world utility. Depthwise separable convolution, a form of
factorize convolution [22] makes them computationally cheaper deep networks.
Accuracy and latency of network is controlled by two hyperparameters (i.e.,
width multiplier, resolution multiplier) to help in building model suitable
for a custom problem. Depthwise convolution can be expressed mathematically as
in Equation 2.

$\displaystyle\hat{\mathbf{G}}_{k,l,m}=\sum_{i,j}\hat{\mathbf{K}}_{i,j,m}\cdot\mathbf{F}_{k+i-1,l+j-1,m}$
(2)

where $\hat{\mathbf{K}}$ denotes the depthwise kernel, $\mathbf{F}$ denotes
the input channel and $\hat{\mathbf{G}}$ denotes the filtered output feature
map.

####  3.1.2 ResNet50

Deep residual network was proposed by He et al. [23] towards improving the
training of extremely deep networks by introducing the idea of reformulating
layers as learning residual functions instead of unreferenced functions. From
empirical results, ResNet proved easier to optimize and improved their
accuracy with increased depth of network. In other words, ResNet allow the
network layers to fit residual mapping instead of fitting for each layer. If
$H(x)$ represents the mapping to be fit by layers of network with input $(x)$,
the residual learning is based on the hypothesis that if certain number of
layers can asymptotically approximate the complicated function, they can also
approximate the residual function $(F(x):=H(x)-x)$.

####  3.1.3 EfficientNetB3

EfficientNet is proposed by Tan and Le [24] as an accurate and efficient
family of ConvNets based on scaled up version of baseline Neural Architecture
Search (NAS) model. Idea of using a simple compound coefficient to uniformly
scale the model in all dimensions (i.e., depth, width, resolution) is
implemented in developing EfficientNets. Scaling up ConvNets by balancing all
dimensions using a constant ratio resulted in better accuracy of models. Based
on this idea, if it is intended to use $2^{n}$ times more computational power,
model can be scaled up in depth by $a^{n}$, in width by $b^{n}$ and in
resolution by $c^{n}$, where $a,b,c$ represent constants.

###  3.2 Data Pre-Processing

At second step of pipeline, extracted visual features were transformed before
feeding to regression model for improved performance. For this study, Standard
Scalar transformation was applied which transforms the data with distribution
having 0 mean and 1 standard deviation. Given a sample $x$, standard scalar
transformation score $z$ can be determined as

$z=\frac{x-\mu}{\sigma}$ (3)

where $\mu$ represents the mean and $\sigma$ represents the standard
deviation. In literature, it has been reported that standard scalar
transformation improves the performance of regression model in comparison to
no transformation applied.

###  3.3 ANN Regression

At the final stage of proposed pipeline, processed visual features were feed
into ANN regression model to predict corresponding hydraulic blockage. ANNs
are machine learning models inspired by the biological functionality of animal
brain and are layer based deep architecture. ANN consists of nodes, layers and
connections. Each node in the network is the representation of a neuron and
apply transformation to input by non-linear activation and transmits it to
other neurons in the network. Layers of ANN consists of number of nodes and
designed to preform specific transformation to input. Furthermore, each layer
is characterized by weights which are updated during the training process to
optimize the desired performance of layer [25, 26, 27, 28]. A neuron $k$ in
layer $L+1$ takes $x_{i}^{L}$ as input and transforms it by applying non-
linear activation into $x_{k}^{L+1}$. Processing of a single neuron in the
network can be mathematically expressed as given in Equation 4.

$x_{k}^{L+1}=f\left(\sum_{i}w_{ik}^{L}x_{i}^{L}+w_{bk}^{L}\right)$ (4)

where $w_{ik}^{L}$ represents the layer $L$ weights, $w_{bk}^{L}$ represents
the bias term of neuron $k$ and $f$ represents the non-linear activation
function.

##  4 Experimental Design

Experiments were performed with three common deep learning models i.e.,
MobileNet, ResNet50, EfficientNetB3 as feature extractor to compare the
performance and decide best among three to be used in final pipeline. All the
CNN models were pre-trained on ImageNet dataset and were used as feature
extractor by removing the top layers. Each CNN model resulted in different
number of visual features (i.e., MobileNet=50176, ResNet50=100352,
EfficientNetB3=153600), therefore, three variants of ANN in terms of number of
hidden layers were used to optimize the training. ANN1 was used with MobileNet
features, ANN2 was used with ResNet50 features and ANN3 was used with
EfficientNetB3 features, respectively. Depth of hidden layers was decided
based on trial and error process with the criteria that increasing number of
hidden layers do not improve the performance anymore. Table 1 presents the
information about the three variants of ANN.

Table 1: ANN Regression Model Variants.

| # of hidden | # of nodes | # of input | # of trainable
---|---|---|---|---
| layers | features | parameters
ANN1 | 5 | [32,16,16,16,16] | 50176 | 1607089
ANN2 | 8 | [32, 16, … , 16] | 100352 | 3213505
ANN3 | 10 | [32, 16, … , 16] | 153600 | 4917985

All the ANN models were trained for 500 epochs with Adam optimizer and
constant learning rate of 0.001. Standard 60:20:20 split of dataset was used
for training, validation and testing, respectively. Furthermore, Mean Absolute
Percentage Error was used as loss metrics during the training process. Models
were trained using Nvidia GeForce RTX 2060 Graphical Processing Unit (GPU)
with 6GB memory and 14 Gbps memory speed.

Performance of regression models was assessed over unseen test data using
standard evaluation metrics including Mean Squared Log Error (MSLE), Mean
Squared Error (MSE), Mean Absolute Error (MAE), and $R^{2}$ score.
Furthermore, overall machine learning pipeline was assessed for processing
times from hardware implementation perspective.

##  5 Results and Discussions

This section presents the results in terms of regression models numerical
results summary, scatter plots, and prediction plots. In addition, overall
machine learning pipeline has been assessed for processing times.

Table 2 presents the summary of recorded quantitative results for the
implemented regression models. From the Table 2, it can be observed that ANN1
model with deep visual features extracted by MobileNet CNN produced the best
results with $R^{2}$ of 0.7855. Interestingly, it has been observed that with
increase in number of deep visual features, performance of ANN regression
degraded. This may be attributed to presence of large number of irrelevant and
uncorrelated features for the ResNet50 and EfficientNetB3 cases.

Table 2: Summary of Empirical Results for Implemented ANN Regression Models | MSLE | MSE | MAE | $\mathbf{R^{2}}$
---|---|---|---|---
ANN1 (with MobileNet features) | 0.10 | 53.54 | 4.67 | 0.7855
ANN2 (with ResNet50 features) | 0.1522 | 86.92 | 5.21 | 0.6517
ANN3 (with EfficientNetB3 features) | 0.4903 | 103.75 | 7.03 | 0.5843

Figure 3 shows the scatter plots for each ANN regression model. From the
plots, it is evident that ANN1 with MobileNet extracted deep visual features
produced the best fit on test data. Figure 4 shows the actual vs predicted
plots for all three ANN models to demonstrate how well each model was able to
track the actual value. ANN1 model was observed to best track the actual
values, however, over-prediction can be observed at majority data instances.
In all three cases, over-prediction was more dominant in comparison to under-
prediction.

Maximum $R^{2}$ score of 0.7855 for MobileNet and positive for all three cases
indicated the presence of correlation between visual features and hydraulic
blockage which suggested that both visual blockage and hydraulic blockage can
be interrelated. However, it is important to mention that dataset used for
this investigation was recorded with same background and lighting conditions
with only variations in culvert type, debris and water levels. This suggests
that, for real-world application, as part of calibration process, camera
should focus only on culvert region avoiding any vegetative background
otherwise performance may significantly degrade given the visual similarity
between vegetative background and vegetative debris material causing blockage.

(a) ANN1 (b) ANN2

(c) ANN3

Figure 3: Scatter Plots for Implemented Regression Models to Predict Hydraulic
Blockage

The proposed machine learning pipeline was assessed in terms of response times
for relative comparison of models towards hardware implementation. Table 3
shows the processing times for each component of proposed pipeline. It can be
observed that ANN1 with MobileNet features was the fastest in predicting the
hydraulic blockage while ANN3 with EfficientNetB3 features was slowest.
Degraded response time may be attributed to increased number of features and
increased depth of ANN model. Reported processing times are not the true
measure of cutting edge hardware performance rather are for relative
comparison purposes. However, technological advancements in the computing
hardware and availability of cutting edge hardware (e.g., Nvidia Jetson Nano,
Nvidia Jetson TX2) has made it possible to implement machine learning and
computer vision algorithms for real-world problems (e.g., tracking of
pedestrian traffic [29], tracking of wildlife [30]).

(a) ANN1 (b) ANN2

(c) ANN3

Figure 4: Actual vs Predicted Plots for Implemented Regression Models. Table
3: Comparison of Machine Learning Pipeline Components for Response Times.

| Image Loading | CNN Model | Feature Extraction | Scalar Transform | ANN Model | Prediction
---|---|---|---|---|---|---
| Loading | Loading
ANN1 | 0.0966 sec | 0.7545 sec | 0.1122 sec | 0.0183 sec | 0.2172 sec | 0.0502 sec
ANN2 | 0.1030 sec | 2.1470 sec | 0.2441 sec | 0.0175 sec | 0.2980 sec | 0.0637 sec
ANN3 | 0.1039 sec | 2.9691 sec | 0.2976 sec | 0.0193 sec | 0.3967 sec | 0.0751 sec

##  6 Conclusion

An end-to-end machine learning pipeline has been successfully implemented to
relate visual blockage with hydraulic blockage by proposing the regression on
deep visual features. Three CNN models (i.e., MobileNet, ResNet50,
EfficientNetB3) were compared for number of extracted visual features and
fundamental principle by which features are extracted. Artificial Neural
Network (ANN) regression model was implemented on deep visual features to
predict the hydraulic blockage. From results, it has been reported that ANN
with MobileNet extracted visual features (i.e., ANN1) achieved the best
regression performance with $R^{2}$ of 0.7855. Regression performance was
observed to be degraded with increase in number of extracted visual features.
The degraded performance may be attributed to presence of increased number of
irrelevant and uncorrelated features in case of ResNet50 and EfficientNetB3.
Positive $R^{2}$ score for all cases with maximum for MobileNet indicated the
presence of correlation between visual features and hydraulic blockage and
suggested that visual blockage and hydraulic blockage can be interrelated.
Performance of regression models is expected to be degraded significantly for
the cases where image contain background with similar visual appearance to the
debris material blocking the culvert. With the availability of hydraulic and
visual data from real-world events, extension of proposed approach for real-
world application is a potential future direction for presented research.

## Acknowledgment

I would like to thank the Wollongong City Council (WCC) for funding this
investigation. This research was funded by the Smart Cities and Suburb Program
(Round Two) of the Australian Government, grant number SCS69244. Further, I
would like to thank the Higher Education Commission (HEC) of Pakistan and the
University of Wollongong (UOW) for funding my PhD studies.

## References

  * [1] R. French and M. Jones, “Culvert blockages in two australian flood events and implications for design,” _Australasian Journal of Water Resources_ , vol. 19, no. 2, pp. 134–142, 2015.
  * [2] ——, “Design for culvert blockage: the arr 2016 guidelines,” _Australasian Journal of Water Resources_ , vol. 22, no. 1, pp. 84–87, 2018\.
  * [3] A. Barthelmess and E. Rigby, “Culvert blockage mechanisms and their impact on flood behaviour,” in _Proceedings of the 34th World Congress of the International Association for Hydro- Environment Research and Engineering_. Barton, ACT: Engineers Australia, 2011, pp. 380–387.
  * [4] E. Rigby and P. Silveri, “The impact of blockages on flood behaviour in the wollongong storm of august 1998,” in _6th Conference on Hydraulics in Civil Engineering: The State of Hydraulics_. Barton, ACT: Engineers Australia, 2001, pp. 107–115.
  * [5] S. Roso, M. Boyd, E. Rigby, and R. VanDrie, “Prediction of increased flooding in urban catchments due to debris blockage and flow diversions,” in _Proceedings of NOVATECH_ , 2004, pp. 8–13.
  * [6] E. Rigby and P. Silveri, “Causes and effects of culvert blockage during large storms,” in _Ninth International Conference on Urban Drainage (9ICUD)_. Lloyd Center Doubletree Hotel, Portland, Oregon, United States: Engineers Australia, September 2002, pp. 1–16.
  * [7] A. Davis, “An analysis of the effects of debris caught at various points of major catchments during wollongong’s august 1998 storm event,” _Bachelor of Engineering Thesis, University of Wollongong_ , 2001.
  * [8] U. Iqbal, P. Perez, W. Li, and J. Barthelemy, “How computer vision can facilitate flood management: A systematic review,” _International Journal of Disaster Risk Reduction_ , vol. 53, p. 102030, 2021.
  * [9] M. Kramer, W. Peirson, R. French, and G. Smith, “A physical model study of culvert blockage by large urban debris,” _Australasian Journal of Water Resources_ , vol. 19, no. 2, pp. 127–133, 2015.
  * [10] J. Blanc, “An analysis of the impact of trash screen design on debris related blockage at culvert inlets,” Ph.D. dissertation, School of the Built Environment, Heriot-Watt University, 2013.
  * [11] J. Blanc, N. P. Wallerstein, S. Arthur, and G. B. Wright, “Analysis of the performance of debris screens at culverts,” in _Proceedings of the Institution of Civil Engineers-Water Management_ , vol. 167, no. 4. Thomas Telford Ltd, 2014, pp. 219–229.
  * [12] NSW, _Floodplain Development Manual_ , New South Wales Government, Sydney, Australia, 2005.
  * [13] N. Jones and C. Lawson, “The queensland urban drainage manual,” _Local Government Engineers Association of Queensland Journal_ , vol. 9, no. 4th quarter, 1991.
  * [14] P. Ollett, B. Syme, and P. Ryan, “Australian rainfall and runoff guidance on blockage of hydraulic structures: numerical implementation and three case studies,” _Journal of Hydrology (New Zealand)_ , vol. 56, no. 2, pp. 109–122, 2017.
  * [15] W. Weeks, G. Witheridge, E. Rigby, A. Barthelmess, and G. O‘Loughlin, “Project 11: Blockage of hydraulic structures,” Engineers Australia, Water Engineering, 11 National Circuit Barton ACT 2600, Tech. Rep. P11/S2/021, February 2013.
  * [16] J. Ball, M. Babister, R. Nathan, P. Weinmann, W. Weeks, M. Retallick, and I. Testoni, “Australian rainfall and runoff-a guide to flood estimation,” 2016\.
  * [17] R. H. Jones, W. Weeks, and M. Babister, _Review of Conduit Blockage Policy Summary Report_. 160 Clarence Street Sydney, NSW, 2000: WMA Water, June 2016.
  * [18] A. Barthelmess and E. Rigby, “Quantification of debris potential and the evolution of a regional culvert blockage model,” in _32nd Hydrology and Water Resources Symposium_. Barton, ACT: Engineers Australia, 2009, pp. 218–229.
  * [19] R. French, E. Rigby, and A. Barthelmess, “The non-impact of debris blockages on the august 1998 wollongong flooding,” _Australasian Journal of Water Resources_ , vol. 15, no. 2, pp. 161–169, 2012.
  * [20] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_. Miami, FL, USA: IEEE, June 2009, pp. 248–255.
  * [21] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” _arXiv preprint arXiv:1704.04861_ , 2017.
  * [22] L. Sifre and S. Mallat, “Rigid-motion scattering for image classification,” Ph.D. dissertation, Ecole Polytechnique, October 2014.
  * [23] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_. Las Vegas, NV, USA: IEEE, June 2016, pp. 770–778.
  * [24] M. Tan and Q. V. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” _arXiv preprint arXiv:1905.11946_ , 2019.
  * [25] A. Abraham, “Artificial neural networks,” _Handbook of measuring system design_ , 2005.
  * [26] K. Mehrotra, C. K. Mohan, and S. Ranka, _Elements of artificial neural networks_. MIT press, 1997.
  * [27] A. Krogh, “What are artificial neural networks?” _Nature biotechnology_ , vol. 26, no. 2, pp. 195–197, 2008.
  * [28] I. A. Basheer and M. Hajmeer, “Artificial neural networks: fundamentals, computing, design, and application,” _Journal of microbiological methods_ , vol. 43, no. 1, pp. 3–31, 2000.
  * [29] J. Barthélemy, N. Verstaevel, H. Forehead, and P. Perez, “Edge-computing video analytics for real-time traffic monitoring in a smart city,” _Sensors_ , vol. 19, no. 9, p. 2048, 2019.
  * [30] B. Arshad, J. Barthelemy, E. Pilton, and P. Perez, “Where is my deer?-wildlife tracking and counting via edge computing and deep learning,” in _2020 IEEE Sensors_. IEEE, 2020, pp. 1–4.


#  Advancing LLM Reasoning Generalists with Preference Trees

Lifan Yuan1,2, Ganqu Cui1∗, Hanbin Wang3,4∗, Ning Ding122footnotemark: 2,
Xingyao Wang2, Jia Deng5,
Boji Shan6, Huimin Chen1, Ruobing Xie7, Yankai Lin5, Zhenghao Liu3, Bowen
Zhou1,
Hao Peng2, Zhiyuan Liu122footnotemark: 2, Maosong Sun1
1Tsinghua University 2University of Illinois Urbana-Champaign 3Northeastern
University
4 ModelBest.Inc 5 Renmin University of China 6 BUPT 7 Tencent
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Equal Contribution.Corresponding Authors.

###### Abstract

We introduce Eurus, a suite of large language models (LLMs) optimized for
reasoning. Finetuned from Mistral-7B and CodeLlama-70B, Eurus models achieve
state-of-the-art results among open-source models on a diverse set of
benchmarks covering mathematics, code generation, and logical reasoning
problems. Notably, Eurus-70B beats GPT-3.5 Turbo in reasoning through a
comprehensive benchmarking across 12 tests covering five tasks, and achieves a
33.3% pass@1 accuracy on LeetCode and 32.6% on TheoremQA, two challenging
benchmarks, substantially outperforming existing open-source models by margins
more than 13.3%. The strong performance of Eurus can be primarily attributed
to UltraInteract, our newly-curated large-scale, high-quality alignment
dataset specifically designed for complex reasoning tasks. UltraInteract can
be used in both supervised fine-tuning and preference learning. For each
instruction, it includes a preference tree consisting of (1) reasoning chains
with diverse planning strategies in a unified format, (2) multi-turn
interaction trajectories with the environment and the critique, and (3)
pairwise data to facilitate preference learning. UltraInteract allows us to
conduct an in-depth exploration of preference learning for reasoning tasks.
Our investigation reveals that some well-established preference learning
algorithms may be less suitable for reasoning tasks compared to their
effectiveness in general conversations. Inspired by this, we derive a novel
reward modeling objective which, together with UltraInteract, leads to a
strong reward model.111Models and data are available at:
https://github.com/OpenBMB/Eurus.

Figure 1: Evaluation results on LeetCode and TheoremQA, two challenging OOD
coding and math benchmarks with only test sets. Our Eurus-7B is comparable
with baselines that are 10x larger and Eurus-70B is the only one on par with
GPT-3.5 Turbo.

##  1 Introduction

Current alignment techniques have significantly advanced the development of
open-source large language models (LLMs) that effectively meet user
expectations and align with human values (Touvron et al., 2023; Tunstall et
al., 2023). On complex reasoning, success has been achieved by specializing
models for specific capabilities, such as coding (Wei et al., 2023; Guo et
al., 2024a; Zheng et al., 2024) and solving math problems (Fu et al., 2023;
Yue et al., 2023; Luo et al., 2023a; Toshniwal et al., 2024). However, these
models still fall short, by large margins, of the most advanced proprietary
models in their all-around capabilities to tackle a diverse range of
challenging problems. We conjecture that this performance gap can be primarily
attributed to (1) the lack of high-quality alignment data and (2) the
underexploration of preference learning techniques for improving models’
complex reasoning capabilities. In this paper, we take strides towards
bridging this gap by addressing both factors and developing Eurus.

Eurus consists of a suite of LLMs finetuned from Mistral-7B (Jiang et al.,
2023a) and CodeLLaMA-70B (Roziere et al., 2023). Across a diverse set of
complex reasoning benchmarks that are mostly out-of-distribution (OOD), Eurus
achieves state-of-the-art overall performance among all open-source models. In
particular, Eurus excels in solving challenging problems that often require
sophisticated planning, reasoning, tool integration, and the ability to
interact with and learn from the environment and users. As shown in Figure 1,
on university-level STEM questions TheoremQA (Chen et al., 2023) and
competition-level coding problems LeetCode Contest (Guo et al., 2024a),
Eurus-70B significantly outperforms all open-source models, achieving
comparable performance to GPT-3.5 Turbo.

Eurus models are trained on UltraInteract, our newly-curated, large-scale, and
high-quality alignment data specifically designed to improve LLMs’ reasoning
capabilities. UltraInteract consists of a diverse set of instructions spanning
math, coding, and logical reasoning problems from 12 established datasets. For
each instruction, UltraInteract collects a preference tree that includes: (1)
Diverse planning strategies in a unified pattern, such as sequential
processing (Wei et al., 2022) and tool creation (Qian et al., 2023), followed
by executing step-by-step actions formatted in either text or code, to provide
divserse reasoning trajectories. (2) Multi-turn interaction trajectories with
the environment and the critique, to improve models’ capabilities to learn
from feedback and correct previous errors (Wang et al., 2023b). (3) Paired
correct and incorrect actions organized in tree structures, to facilitate
preference learning. In total, UltraInteract contains 86K instructions and
220K action pairs, where each pair consists of an instruction, a correct
response, and an incorrect one. Conceptually, UltraInteract’s data resemble
imbalanced binary trees as shown in Figure 2.

UltraInteract can be used in both supervised fine-tuning and preference
learning. Our experiments show that, using UltraInteract along with
established datasets in instruction fine-tuning already achieves strong
performance. UltraInteract further facilitates preference learning for
reasoning tasks, improving the performance even further with KTO (Ethayarajh
et al., 2024) and NCA (Chen et al., 2024a). Surprisingly, applied to an
instruction finetuned Eurus model, DPO (Rafailov et al., 2023) hurts the
performance.

Through careful analysis, we provide evidence that the performance in
reasoning correlates with the value of rewards of chosen data—a higher final
reward often indicates a better reasoning capability. Besides, our
investigation suggests that DPO may be less suitable for reasoning tasks than
KTO and NCA. Inspired by this fresh finding, we devise a new objective for
reward modeling to augment the Bradley-Terry objective (Bradley & Terry,
1952), explicitly encouraging training to increase the absolute rewards of
chosen solution and decrease those of rejected data. Furthermore,
UltraInteract leads to our reward model Eurus-RM-7B, which achieves a better
correlation with human annotators than all existing models on AutoJ (Li et
al., 2023a) and MT-Bench (Zheng et al., 2023), including GPT-4 (OpenAI, 2023).
Eurus-RM-7B demonstrates especially strong preference modeling performance on
reasoning tasks.

Checkpoints of our Eurus models, accompanying UltraInteract alignment data to
reproduce this research, will be publicly available.

##  2 UltraInteract: Tree-structured Alignment Data for Reasoning

Figure 2: Left: CodeActInstruct (Wang et al., 2024) and Code-Feedback (Zheng
et al., 2024); Middle: HH-RLHF (Bai et al., 2022); Right: UltraInteract. Each
instruction in UltraInteract is constructed as a preference tree.

Solving complex problems often requires the model’s capability in planning and
reasoning, integrating with tools, and interacting with and learning from both
the environment and the users. This is reflected in UltraInteract’s design
choices: (1) Its instructions are diverse, challenging, and of a large scale
(§2.1); (2) It provides multi-turn trajectories that solve the input
instruction through multiple turns of interaction with and learning from the
environment and critique. At each turn, it breaks down the problem into
smaller ones (§2.2). (3) UltraInteract includes pairwise data to facilitate
preference learning (§2.3).

Conceptually, UltraInteract collects a preference tree for each instruction,
with the instruction being the root and each action a node (Figure 2). A
trajectory is a root-to-leaf path consisting of a sequence of actions. In each
preference tree, all nodes of correct actions and all trajectories ending with
correct actions can be used for SFT. Paired correct and incorrect nodes or
trajectories can be used for preference learning.

###  2.1 Instruction Selection Emphasizing Complexity, Quality, and Diversity

We target three representative reasoning tasks: math problem-solving, code
generation, and logical reasoning. The complexity, quality, and diversity of
the alignment data are crucial to the model’s performance (Liu et al., 2023).
Following Wang et al. (2023b), we select challenging problems that
GPT-3.5-Turbo fails to solve. We intentionally restrict the selection of the
datasets to those with ground-truth solutions, aiming to ensure high-quality
oversight signals rather than relying on LLM-as-a-judge annotation (Weyssow et
al., 2024). Besides, the gold solutions also serve as references for the
critique model to generate feedback. To promote UltraInteract’s diversity, we
pick datasets of different categories. For each dataset, we include distinct
reasoning patterns based on question categories or formulations necessary to
solve the problems. Table 6 summarizes the datasets selected by UltraInteract.
Except for MATH, none of the training datasets is used in our evaluation.

Figure 3: An illustrative example of an UltraInteract trajectory over two
turns. In each turn, the actor model generates step-by-step reasoning chains,
and the environment and the critique model provide observations and textual
critique respectively.

###  2.2 Decomposition and Interaction at Each Turn

Figure 3 provides an illustrative example. In what follows, we connect the
actor model with a Python interpreter as the “environment”. Unless otherwise
specified, we use GPT-3.5 Turbo as the actor model.

Following Wang et al. (2024), the actor model first decomposes the input
problem into several sub-problems and then solves each by generating Python
code pieces as actions and using the environment to execute them. To promote
solution diversity, the actor model randomly samples one reasoning schema in
the form of either CoT (Wei et al., 2022) or modularization programming (Qian
et al., 2023; Yuan et al., 2023). The actor then generates actions in text or
code to solve each sub-problem, with each step being marked by explicit
notations.

Multi-turn interactions with the environment are often necessary to solve
challenging problems (Wang et al., 2023b). To improve such capabilities of the
models, UltraInteract collects trajectories in which the actor model interacts
with the environment and a critique model (a proxy for user) and refines its
action based on their feedback.

The environment receives an action from the actor model along with the
interaction history, and then the code interpreter returns two kinds of
“Observation”: (1) Python execution results, either program outputs or error
traceback messages; (2) binary feedback, indicating whether the solution is
correct or not. Then, the observations along with the history will be passed
to a critique model, which locates the errors and provides suggestions for
improvements. To avoid potential bias introduced by self-correction (Wang et
al., 2023b; Xu et al., 2024), we adopt a stronger model, GPT-4, as the
critique and ensure critique quality by providing GPT-4 with ground truth
answers as references.

This procedure resembles Wang et al. (2024). However, we adopt more diverse
reasoning patterns to teach LLMs to learn rationales rather than simply
memorizing answers (Mitra et al., 2023), and learn to create and use tools
(Qian et al., 2023; Yuan et al., 2023; Qin et al., 2023). Besides, we believe
that it is important for LLMs to learn from the feedback provided by the
critique rather than solely from observations of the environment.

###  2.3 Preference Trees Facilitates Preference Learning Across Multiple
Turns

Unlike open-ended conversations, where human preference is ambiguous and
challenging to specify, many reasoning tasks have clear and objective
preferences for correct actions. The preference annotation is threfore an
evaluation of the correctness of the solutions conditioning ground truth ones,
which come with the datasets in UltraInteract. This eliminates the need for
human or LLM-based preference annotation and ensures high data quality. To
facilitate preference learning, UltraInteract pairs correct and incorrect
actions.

Sampling Paired Correct and Incorrect Actions at Each Turn. For each
instruction in UltraInteract, we sample, from the actor model, a pair of
correct and incorrect actions following §2.2. We follow Cui et al. (2023) to
sample the pair from different actor models to ensure response diversity. To
prevent models from exploiting shortcuts based on surface features, we exclude
instances that fail to pass the Python syntax check.

Certain challenging problems in UltraInteract pose difficulties in obtaining
correct actions, even using strong actors such as GPT-4, with nearly zero
pass@100 accuracies. To improve the pass rates of the actor models while
keeping the expense under control, we sequentially take the following steps.
(1) Directly sampling 20 actions and randomly keeping a correct one, if any.
(2) If no correct action is obtained, we repeat the above process up to three
times, progressively switching from more cost-effective models to the strong
yet expensive GPT-4 Turbo. (3) For the remaining difficult problems where no
correct action is acquired after the previous two steps, we provide the actor
with ground-truth rationales and answers, and then apply various techniques to
elicit correct actions. The specific information provided and the techniques
applied vary depending on the tasks (Appendix A.2).

Tree-structured Action Pairs Across Multiple Turns. After each turn, the
correct action concludes its trajectory. We expand the incorrect action into
the next turn, and have the actor interact with the environment and the
critique to refine its solution (§2.2). We then repeat the procedures
introduced earlier in this section to collect an additional action pair. By
expanding the incorrect action, UltraInteract can provide data to help models
learn from feedback, and collect multiple action pairs for preference learning
across multiple turns.

Conceptually, for every instruction, UltraInteract constructs a binary
preference tree with each action being a node (Figure 2). We cap the tree at a
maximum of five turns.

Additional Instruction-action Pairs for Challenging Problems. We believe the
challenging instructions that make it to step (3) above can provide valuable
training signals. Therefore, for a subset of these problems with multiple
ground truth solutions, we further sample additional correct actions to cover
all ground truths. Accordingly, we further sample incorrect actions to pair
with these additional correct actions, so that they can be used in both
supervised fine-tuning and preference learning.

With the tree-structured data, UltraInteract enables comparisons at every
turn, in contrast to comparing only at the last turn (Bai et al., 2022), and
thus can improve the models’ interaction ability. Closing this section, Table
1 summarizes some statistics of UltraInteract, and more details are in
Appendix A.4.

Table 1: Some statistics of UltraInteract.

Task | Type | # Instructions | # Turns per Traj. |  # Tokens per Traj. |  Avg. # Traj per Ins. |  Total # Pairs |  # Correct Answers
---|---|---|---|---|---|---|---
w/ Interaction? | w/ Tool? | T1 | T2 | T3 | T4 | T5
Math | ✓ | ✓ | 22,928 | 10,440 | 4,122 | 1,898 | 904 | 5,564 | 1,750.0 | 1.0 | 42,780 | 68,033
✗ | ✓ | 2,757 | 16,154 | - | - | - | - | 439.1 | 5.9 | 13,217 | 16,154
✓ | ✗ | 22,639 | 10,708 | 3,521 | 1,459 | 723 | 6,228 | 1,521.9 | 1.0 | 44,750 | 62,182
✗ | ✗ | 2,083 | 16,348 | - | - | - | - | 538.1 | 7.8 | 12,624 | 16,348
Coding | ✓ | - | 20,463 | 13,265 | 2,584 | 987 | 379 | 3,248 | 1,728.5 | 1.0 | 18,106 | 22,215
✗ | - | 8,495 | 92,618 | - | - | - | - | 1,070.4 | 5.5 | 78,634 | 92,618
Logic | ✓ | ✓ | 2,086 | 1,685 | 298 | 72 | 8 | 23 | 1,299.8 | 1.0 | 1,750 | 2,198
✓ | ✗ | 4,467 | 2,453 | 1,674 | 340 | 0 | 0 | 1,266.7 | 1.0 | 7,958 | 7,231
Total | - | - | 85,918 | 163,671 | 12,199 | 4,756 | 2,014 | 15,063 | 1,201.8 | 2.3 | 219,819 | 286,979

##  3 Eurus: State-of-the-art Open LLMs in Reasoning

UltraInteract helps us develop Eurus, a suite of LLMs and a reward model (RM).

Supervised Fine-Tuning. Eurus-7B-SFT is fine-tuned from Mistral-7B (Jiang et
al., 2023a) and Eurus-70B-SFT from CodeLLaMA-70B (Roziere et al., 2023).
First, we perform SFT using all correct actions (287K) in UltraInteract. We
find it yields better performance to discard interaction history and train
only on correct leaf nodes in each tree. To improve general instruction-
following ability, we include into our SFT data mixture UltraChat (Ding et
al., 2023),
ShareGPT222https://huggingface.co/datasets/openchat/openchat_sharegpt4_dataset,
and OpenOrca (Lian et al., 2023). Please find mixture ratios in Appendix B.

Perference Learning. Based on Eurus-SFT models, we explore three preference
learning algorithms, DPO (Rafailov et al., 2023), KTO (Ethayarajh et al.,
2024), and NCA (Chen et al., 2024a). Differently from SFT, here we include all
multi-turn trajectory pairs in our UltraInteract (220K) and include all
UltraFeedback (Cui et al., 2023) pairs (340K).

Reward Modeling. Similarly to the preference learning, we use all 220K multi-
turn trajectory pairs from UltraInteract; it is further augmented with the
240K single-turn action pairs from UltraInteract. More details are in the
Appendix B. We include all 340K pairs from UltraFeedback and one pair for each
instruction from UltraSafety (Guo et al., 2024b), totaling 3K. Eurus-RM-7B is
initialized from Eurus-7B-SFT with a new linear layer.

Our findings in §6 indicate that the absolute values of rewards make a big
difference in the models’ reasoning performance. We therefore augment the
established Bradley-Terry (BT) objective $\mathcal{L}_{\text{BT}}$ with an
additional term $\mathcal{L}_{\text{DR}}$ to directly increase the reward of
the chosen actions for instances from UltraInteract, and decrease those of the
rejected ones:

$\mathcal{L}_{\text{{UltraInteract}}}=\underbrace{-\log\Bigl{(}\sigma\bigl{(}r_{\theta}(x,y_{c})-r_{\theta}(x,y_{r})\bigr{)}\Bigr{)}}_{\text{$\mathcal{L_{\text{BT}}}$:
optimize relative
rewards}}\underbrace{-\log\Bigl{(}\sigma\bigl{(}r_{\theta}(x,y_{c})\bigr{)}\Bigr{)}-\log\Bigl{(}\sigma\bigl{(}-r_{\theta}(x,y_{r})\bigr{)}\Bigr{)}}_{\text{$\mathcal{L_{\text{DR}}}$:
increase $r_{\theta}(x,y_{c})$ and decrease $r_{\theta}(x,y_{r})$}}$

For instances from other datasets, we train with $\mathcal{L_{\text{BT}}}$.
$\theta$ denotes the reward model’s parameters, $r_{\theta}\left(\cdot\right)$
and $r_{\theta}\left(x,y_{r}\right)$ the rewards on the chosen and rejected
actions respectively. Our ablation study demonstrates the importance of both
$\mathcal{L}_{\text{BT}}$ and $\mathcal{L}_{\text{DR}}$.

##  4 Evaluation of Eurus-7B and Eurus-70B

Evaluation Setup. We consider both single-turn and multi-turn reasoning. For
single-turn evaluation, we consider HumanEval (Chen et al., 2021), MBPP
(Austin et al., 2021), and LeetCode (Guo et al., 2024a) for coding, GSM-Plus
(Li et al., 2024), MATH, TheoremQA (Chen et al., 2023), SVAMP (Patel et al.,
2021), and ASDiv (Miao et al., 2020) for math, and BBH-Hard (Suzgun et al.,
2022) for reasoning. We evaluate with pass@1 accuracy. We also use IFEval
(Zhou et al., 2023) to assess the instruction-following ability and report the
prompt-level loose score. For multi-turn evaluation, we adopt MINT (Wang et
al., 2023b) and only consider the coding and math problems. We report the
success rate at Turn 5. Please find further details on evaluation setups and
evaluations beyond reasoning in Appendix C.

As shown in Table 2, we compare our Eurus with general-purpose models, and
those specialized in coding and math of various sizes.

Table 2: Open-source LLM baselines that we compare to.

Type | Models
---|---
General Purpose |  Mistral-7B-Instruct-v0.2 (Jiang et al., 2023a), Zephyr-7B-$\beta$ (Tunstall et al., 2023), OpenChat-3.5-1210 (Wang et al., 2023a), Starling-LM-7B-$\alpha$ (Zhu et al., 2023), Mixtral-8x7B-Instruct (Jiang et al., 2023a), DeepSeek-LLM-67B-Chat (DeepSeek-AI, 2024), QWen1.5-72B-Chat (Bai et al., 2023)
Coding |  Magicoder-S-DS-6.7B (Wei et al., 2023), OpenCodeInterpreter (OpenCI for short, DS-6.7B/CL-70B) (Zheng et al., 2024), DeepSeek-Coder-33B-Instruct (Guo et al., 2024a), and CodeLLaMA-70B-Instruct(Roziere et al., 2023).
Math |  MAmmoTH-7B-Mistral (Yue et al., 2023), WizardMath-7B-v1.1 (Luo et al., 2023a), OpenMath (Mistral-7B/CodeLLaMA-70B) (Toshniwal et al., 2024).

We also summarize the results of GPT-3.5 Turbo and GPT-4 reported in previous
works.

Table 3: Overall performance. All test sets except MATH are out-of-
distribution to our models and most baselines. MAmmoTH, OpenChat, and
Starling-LM have been trained on TheoremQA test sets. We strikethrough the
contaminated numbers.

| Coding | Math | Reasoning | Ins-Following | Multi-Turn |
---|---|---|---|---|---|---
Model | HumanE. | MBPP | LeetC. | GSM-Plus | MATH | Theo.QA | SVAMP | ASDiv | BBH | IFEval | Code | Math | Avg.
$\sim$7B
Mistral-7B-Instruct-v0.2 | 39.0 | 30.8 | 6.1 | 15.7 | 9.5 | 8.5 | 42.9 | 49.5 | 62.4 | 44.4 | 7.4 | 26.2 | 28.5
Zephyr-7B-$\beta$ | 29.3 | 35.8 | 2.2 | 23.3 | 5.0 | 7.8 | 19.1 | 28.0 | 61.8 | 39.7 | 5.2 | 16.9 | 22.8
OpenChat-3.5-1210 | 64.0 | 61.7 | 11.7 | 46.7 | 28.1 | 19.1 | 75.4 | 77.0 | 67.0 | 50.3 | 21.3 | 32.4 | 46.2
Starling-LM-7B-$\alpha$ | 46.3 | 51.1 | 8.9 | 23.7 | 21.5 | 12.0 | 26.3 | 39.8 | 67.1 | 26.1 | 18.4 | 28.9 | 30.8
Magicoder-S-DS-6.7B | 75.6 | 70.4 | 23.9 | 16.4 | 19.9 | 13.1 | 61.6 | 62.8 | 57.0 | 21.1 | 27.9 | 8.0 | 38.1
OpenCI-DS-6.7B | 76.8 | 66.2 | 16.1 | 41.5 | 31.6 | 16.1 | 74.5 | 79.8 | 53.9 | 22.6 | 5.9 | 1.3 | 40.5
MAmmoTH-7B-Mistral | 24.4 | 42.4 | 7.2 | 40.1 | 36.0 | 26.3 | 60.7 | 72.3 | 57.7 | 34.9 | 3.7 | 6.7 | 34.4
WizardMath-7B-v1.1 | 50.0 | 53.9 | 6.7 | 54.6 | 30.0 | 16.5 | 57.8 | 73.5 | 64.4 | 22.6 | 16.2 | 8.9 | 37.9
OpenMath-Mistral-7B | 33.5 | 46.6 | 11.7 | 59.4 | 39.1 | 13.1 | 83.4 | 79.8 | 58.6 | 15.0 | 2.9 | 5.3 | 37.4
Eurus-7B-SFT | 55.5 | 59.1 | 20.0 | 52.1 | 32.6 | 20.0 | 82.2 | 84.1 | 64.6 | 44.0 | 15.4 | 28.4 | 46.5
\+ DPO | 50.6 | 52.1 | 8.3 | 51.0 | 28.3 | 20.9 | 78.7 | 83.8 | 65.0 | 42.5 | 20.6 | 32.4 | 44.5
\+ KTO | 56.1 | 58.6 | 18.9 | 55.0 | 33.2 | 20.6 | 84.4 | 85.0 | 67.6 | 43.1 | 19.1 | 43.6 | 48.8
\+ NCA | 55.5 | 60.2 | 14.4 | 54.9 | 34.2 | 20.9 | 84.6 | 85.4 | 64.3 | 42.7 | 21.3 | 38.7 | 48.1
$\sim$40B
Mixtral-8x7B-Instruct | 50.6 | 50.1 | 5.6 | 49.6 | 25.9 | 20.4 | 66.4 | 68.8 | 73.5 | 48.8 | 12.5 | 37.3 | 42.5
DeepSeek-Coder-33B-Ins | 82.3 | 73.9 | 27.8 | 29.5 | 20.2 | 21.9 | 75.2 | 85.0 | 61.5 | 26.1 | 35.3 | 21.8 | 46.7
$\sim$70B
CodeLLaMA-70B-Instruct | 56.7 | 58.6 | 14.4 | 34.9 | 12.0 | 8.4 | 63.5 | 70.1 | 74.5 | 24.0 | 3.7 | 14.2 | 36.3
DeepSeek-LM-67B-Chat | 70.7 | 65.7 | 20.0 | 65.0 | 41.0 | 17.9 | 74.0 | 84.0 | 78.9 | 52.7 | 30.9 | 41.8 | 53.5
QWen1.5-72B-Chat | 71.3 | 56.9 | 15.6 | 65.4 | 43.4 | 18.5 | 79.5 | 79.1 | 78.0 | 53.4 | 27.2 | 38.2 | 52.2
OpenCI-CL-70B | 77.4 | 71.7 | 20.0 | 46.1 | 29.2 | 18.8 | 76.1 | 79.4 | 66.7 | 26.8 | 30.9 | 12.0 | 46.3
OpenMath-CL-70B | 39.0 | 52.6 | 15.0 | 62.2 | 45.9 | 15.9 | 86.6 | 82.8 | 59.9 | 15.7 | 14.0 | 0.4 | 40.8
Eurus-70B-SFT | 75.6 | 74.2 | 33.3 | 58.1 | 40.6 | 28.0 | 86.3 | 88.5 | 79.9 | 49.2 | 31.6 | 40.4 | 57.1
\+ KTO | 76.8 | 68.2 | 26.1 | 62.2 | 41.3 | 30.6 | 90.4 | 89.0 | 80.8 | 46.4 | 39.0 | 49.8 | 58.4
\+ NCA | 79.3 | 71.9 | 33.3 | 62.8 | 41.7 | 32.6 | 89.5 | 90.3 | 80.0 | 49.2 | 38.2 | 39.6 | 59.0
Proprietary Models
GPT-3.5 Turbo | 76.8 | 82.5 | 23.3 | 61.2 | 37.8 | 35.6 | 83.0 | 90.6 | 70.1 | 56.6 | 29.4 | 36.9 | 57.0
GPT-4 | 85.4 | 83.5 | 41.8 | 85.6 | 69.7 | 52.4 | 94.8 | 92.6 | 86.7 | 79.7 | 59.6 | 65.8 | 74.8

###  4.1 Results

Results are shown in Table 3. We summarize the takeaways as follows:

Eurus, both the 7B and 70B variants, achieve the best overall performance
among open-source models of similar sizes. Eurus even outperform specialized
models in corresponding domains in many cases. Notably, Eurus-7B outperforms
baselines that are 5$\times$ larger and Eurus-70B achieves better performance
than GPT-3.5 Turbo. Eurus’s instruction-following performance is among the
best general-purpose models, substantially better than specialized ones.

Preference learning with UltraInteract can further improve the performance,
especially in math and the multi-turn ability. KTO and NCA consistently
improve the models’ performance in all five math benchmarks and mult-turn
evaluations, while their effects vary in others. Since SFT models only use the
single-turn data from UltraInteract while preference learning uses the multi-
turn ones, the improvements in interaction ability should also be attributed
to UltraInteract rather than the algorithms alone. Surprisingly, we observe
that DPO hurts model performance on most benchmarks. DPO training of our 70B
model fails since the rewards go down to $-\infty$. We analyze this phenomenon
in §6.1.

##  5 Evaluation of Eurus-RM-7B

Evaluation Setup. We evaluate Eurus-RM-7B on three RM benchmarks, RewardBench
(Lambert et al., 2024), AutoJ (Li et al., 2023a), and MT-Bench (Zheng et al.,
2023). Aiming for a more realistic OOD evalation, we exclude the “prior sets”
split from RewardBench, since many baselines train on the datasets that this
split contains. We compare with PairRM (Jiang et al., 2023b), Starling-
RM-7B/34B (Zhu et al., 2023), UltraRM-13B (Cui et al., 2023), GPT-3.5 Turbo,
and GPT-4. To further explore Eurus-RM-7B’s potential in improving models’
performance through reranking, we use it to rerank Mistral-7B-Instruct-v0.2’s
responses on HumanEval, MBPP, GSM8K, and MATH. We report the results of random
sampling, self-consistency, and Starling-RM-34B as baselines.

###  5.1 Results

Table 4 summarizes reward modeling performance, and Figure 4 plots some
reranking results with others in Appendix D.1.

Eurus-RM-7B stands out as the best 7B RM overall, and achieves similar or
better performance than much larger baselines. Particularly, it outperforms
GPT-4 in certain tasks.  Eurus-RM-7B achieves a better correlation with human
experts than all existing models on AutoJ and MT-Bench, and it achieves
comparable performance to the 5$\times$ larger Starling-RM-34B on RewardBench.
On RewardBench, Eurus-RM-7B outperforms all baselines on the “Chat-Hard” split
while achieving very competitive performance on the “Reasoning” split. Across
the AutoJ splits, Eurus-RM-7B outperforms nearly all existing models, with the
only exception being GPT-4’s results on Coding.

Our training objective is beneficial in improving RM performance on hard
problems and reasoning. Table 4 shows that optimizing
$\mathcal{L}_{\text{DR}}$ improves RM’s reasoning ability, but BT modeling is
still beneficial in equipping RM with abilities in general chatting as
suggested in the “Chat-Hard” column, though its effect on reasoning may vary.

UltraInteract is compatible with other datasets like UltraFeedback and
UltraSafety, and mixing these datasets can balance different RM abilities.
Improving RM’s capabilities in reasoning with UltraInteract does not sacrifice
others, which indicates that UltraInteract can be a great ingredient for the
training data mixture of reward models.

Figure 4: Results on reranking Mistral-7B-Instruct-v0.2’s responses. Full
results in Table 9.

Eurus-RM-7B improves LLMs’ reasoning performance by a large margin through
reranking. Eurus-RM-7B consistently improves pass@1 accuracy across all tasks
and performs better than 5$\times$ larger baseline Starling-RM-34B. Also,
Eurus-RM-7B’s reranking performance scales well with #responses per
instruction, except a slight decrease in HumanEval when increasing response
number form 8 to 16. In contrast, Starling-RM-34B suffers from severe
performance drop on HumanEval and it consistently hurts model accuracy on
MATH.

Table 4: Results on reward modeling benchmarks. UF: UltraFeedback; US:
UltraSafety. The best performance in each benchmark is in bold and the second
best one is underlined. Most baseline results are from Jiang et al. (2023b)
and Lambert et al. (2024).

Model | Reward Bench |  | AutoJ |  | MT-Bench
---|---|---|---|---|---
Chat | Chat-Hard | Safety | Reasoning | Avg. |  | Code | Math | Others | Overall |
PairRM | 90.2 | 53.0 | 31.5 | 60.0 | 58.7 |  | 58.3 | 52.8 | 58.9 | 59.1 |  | 59.0
Starling-RM-7B | 98.0 | 43.4 | 88.6 | 74.6 | 76.2 |  | 59.2 | 47.2 | 61.4 | 60.8 |  | 56.8
Starling-RM-34B | 96.9 | 59.0 | 89.9 | 90.3 | 84.0 |  | 65.8 | 54.2 | 62.3 | 62.6 |  | 60.4
UltraRM-13B | 96.1 | 55.3 | 45.8 | 82.0 | 69.8 |  | 55.0 | 43.1 | 59.6 | 59.9 |  | 56.0
GPT-3.5 Turbo | - | - | - | - | - |  | 36.6 | 40.3 | 41.2 | 42.7 |  | 57.1
GPT -4 | - | - | - | - | - |  | 69.2 | 51.4 | 61.4 | 61.9 |  | 63.9
Eurus-RM-7B | 96.5 | 65.3 | 80.7 | 87.0 | 82.4 |  | 67.5 | 62.5 | 63.6 | 64.5 |  | 72.9
w/o $\mathcal{L}_{\text{DR}}$ | 96.4 | 59.9 | 79.5 | 77.5 | 78.3 |  | 64.2 | 59.7 | 64.7 | 65.0 |  | 72.8
w/o $\mathcal{L}_{\text{BT}}$ | 96.8 | 58.5 | 83.8 | 84.2 | 80.8 |  | 67.5 | 66.7 | 64.8 | 65.6 |  | 72.6
w/o US | 96.5 | 66.2 | 67.7 | 81.7 | 73.3 |  | 66.7 | 61.1 | 65.0 | 65.7 |  | 72.6
w/o UF + US | 95.1 | 61.1 | 63.7 | 73.4 | 78.0 |  | 55.8 | 58.3 | 59.0 | 58.7 |  | 67.2

##  6 Analysis

Figure 5: Reward patterns of Eurus-7B preference learning with DPO, KTO, and
NCA. For all algorithms, the rewards of rejected data keep decreasing and the
margins between chosen and rejected data keep increasing. However, the rewards
of chosen data decrease below zero in DPO while keeping increasing and staying
positive in KTO and NCA. The absolute values of the reward in the last step
(in red) of the three algorithms positively correlate with their performance
in Table 3.

###  6.1 Explicit Reward as A Proxy? Hypothesis for Preference Learning in
Reasoning

We investigate the reason why DPO behaves differently than KTO and NCA. We
start by empirically inspecting the rewards throughout the preference learning
process, as shown in Figure 5. Rewards for chosen rejected data both keep
decreasing through DPO, though the rewards for chosen data is still higher
hence the loss decreases. In KTO and NCA, the rewards of chosen data keep
increasing with those of rejected data decreasing.

Therefore, we hypothesize it is the distinction in the trend of rewards that
leads to the performance gap between DPO and the other two algorithms. This
distinction can be attributed to that DPO, derived from the Bradley-Terry
model, only optimizes the relative differences between chosen and rejected
data overlooking the absolute values of the rewards. This is a non-issue in
alignment with general human values where preference is “relative” and there
can be many valid answers to the same input. However, in reasoning tasks, the
space of correct answers is much smaller than that of incorrect ones. Further,
we notice that the rewards of chosen data in the last training step follow the
ranking order of KTO $>$ NCA $>$ DPO, positively correlate with their
performance trends. Therefore, we believe that increasing the rewards of the
chosen data is especially beneficial in preference learning for reasoning
tasks.

###  6.2 Ablation Study

Table 5: Ablation study of SFT data.

Model | Coding | Math | BBH | IFEval | Avg.
---|---|---|---|---|---
Eurus-7B-SFT | 44.9 | 58.5 | 64.6 | 44.0 | 53.6
Ground-truth | 33.9 | 46.1 | 64.4 | 42.9 | 44.0
Open-source Only | 31.2 | 33.5 | 65.3 | 43.6 | 37.0
UltraInteract Only | 37.3 | 56.2 | 67.0 | 17.4 | 47.7

We study the impact of UltraInteract and other open-source alignment data on
Eurus-7B-SFT’s performance. We consider three settings: (1) With original
ground-truth answers, which replaces the generated actions with ground-truth
rationales and answers from the original datasets. If no rationales are
available, we use those from UltraInteract. (2) Open-source data only.
(3)UltraInteract only. We evaluate with the same setting as §4 and report the
averaged scores. See full results in Appendix E.

In Table 5, Eurus outperforms the “Grouth-truth” model on all tasks,
confirming the advantage of UltraInteract’s designs of divide-and-conquer and
code-as-action patterns, in line with conclusions of concurrent work (Chen et
al., 2024b; Wang et al., 2024). Training only on open-source data without
UltraInteract greatly hurts the reasoning performance, confirming the
effectiveness of UltraInteract. Meanwhile, training only on UltraInteract
suffers a performance drop except for BBH, especially in instruction
following. We attribute the performance drop to a worse instruction-following
ability. This suggests the necessity of mixing UltraInteract with other
alignment data for better all-around supervised fine-tuning.

##  7 Related Work

Open LLMs in Reasoning. Open-source LLMs have shown remarkable progress in
building specialists that excel in mathematics reasoning (Luo et al., 2023a;
Yue et al., 2023; Toshniwal et al., 2024) or coding abilities (Roziere et al.,
2023; Wei et al., 2023; Guo et al., 2024a; Zheng et al., 2024). On the
contrary, mastering general reasoning capabilities still challenges open
models, while the most advanced ones (DeepSeek-AI, 2024; Bai et al., 2023;
Touvron et al., 2023; Jiang et al., 2024) are well behind proprietary models.
More, these cutting-edge open general-purpose models maintain their alignment
recipes confidential, which further hinders the replication and development of
open-source reasoning models.

Preference Learning for Reasoning. Aligning language models from human or AI
preferences has emerged as a prevalent approach in the open-source community
(Tunstall et al., 2023; Bai et al., 2023) with the proposal of DPO (Rafailov
et al., 2023) and high-quality preference datasets (Cui et al., 2023; Zhu et
al., 2023). Different from open-domain chatbots, preference learning is
largely underexplored in complex reasoning. Recent research showed performance
degradation when applying DPO on reasoning tasks, but some newly proposed
algorithms demonstrated a positive effect (Ethayarajh et al., 2024; Chen et
al., 2024a; Mitra et al., 2024; Shao et al., 2024). However, a deep
understanding of preference learning, specifically its efficacy on complex
reasoning, is not yet established.

##  8 Conclusion

We strive to narrow the huge gap between open-source models and proprietary
models from the perspective of alignment. Our work pushes the boundaries of
open-source reasoning generalists by (1) releasing a high-quality multi-turn
reasoning dataset UltraInteract with preference trees, (2) introducing Eurus-
series LLMs which achieve new SOTA on challenging reasoning benchmarks and (3)
providing insights on preference learning for reasoning through analysis,
leading to new reward modeling objectives as well as a powerful reward model
for reasoning.

## References

  * Amini et al. (2019) Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi.  MathQA: Towards interpretable math word problem solving with operation-based formalisms.  In _Proc. of NAACL-HLT_ , 2019.
  * Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al.  Program synthesis with large language models.  _ArXiv preprint_ , abs/2108.07732, 2021.
  * Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu.  Qwen technical report.  _ArXiv preprint_ , abs/2309.16609, 2023.
  * Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, T. J. Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, Benjamin Mann, and Jared Kaplan.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _ArXiv preprint_ , abs/2204.05862, 2022.
  * Bradley & Terry (1952) Ralph Allan Bradley and Milton E. Terry.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  _Biometrika_ , 39, 1952.
  * Chen et al. (2024a) Huayu Chen, Guande He, Hang Su, and Jun Zhu.  Noise contrastive alignment of language models with explicit rewards.  _ArXiv preprint_ , abs/2402.05369, 2024a.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.  Evaluating large language models trained on code, 2021.
  * Chen et al. (2023) Wenhu Chen, Ming Yin, Max W.F. Ku, Yixin Wan, Xueguang Ma, Jianyu Xu, Tony Xia, Xinyi Wang, and Pan Lu.  Theoremqa: A theorem-driven question answering dataset.  _ArXiv preprint_ , abs/2305.12524, 2023.
  * Chen et al. (2024b) Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang, Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao.  Agent-flan: Designing data and methods of effective agent tuning for large language models.  volume abs/2403.12881, 2024b.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.  Training verifiers to solve math word problems.  volume abs/2110.14168, 2021.
  * Cui et al. (2023) Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun.  Ultrafeedback: Boosting language models with high-quality feedback.  _ArXiv preprint_ , abs/2310.01377, 2023.
  * DeepSeek-AI (2024) DeepSeek-AI.  Deepseek llm: Scaling open-source language models with longtermism.  _ArXiv preprint_ , abs/2401.02954, 2024.
  * Ding et al. (2023) Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou.  Enhancing chat language models by scaling high-quality instructional conversations.  In _Conference on Empirical Methods in Natural Language Processing_ , 2023.
  * Ethayarajh et al. (2024) Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela.  Kto: Model alignment as prospect theoretic optimization.  _ArXiv preprint_ , abs/2402.01306, 2024.
  * Fu et al. (2023) Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot.  Specializing smaller language models towards multi-step reasoning.  In _Proceedings of the International Conference on Machine Learning_ , 2023.
  * Geva et al. (2021) Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant.  Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies.  _Transactions of the Association for Computational Linguistics_ , 9, 2021.
  * Guo et al. (2024a) Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang.  Deepseek-coder: When the large language model meets programming - the rise of code intelligence.  _ArXiv preprint_ , abs/2401.14196, 2024a.
  * Guo et al. (2024b) Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Jiexin Wang, Huimin Chen, Bowen Sun, Ruobing Xie, Jie Zhou, Yankai Lin, Zhiyuan Liu, and Maosong Sun.  Controllable preference optimization: Toward controllable multi-objective alignment.  _ArXiv preprint_ , abs/2402.19085, 2024b.
  * Hendrycks et al. (2021a) Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al.  Measuring coding challenge competence with apps.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , 2021a.
  * Hendrycks et al. (2021b) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.  Measuring mathematical problem solving with the math dataset.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , 2021b.
  * Jiang et al. (2023a) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.  Mistral 7b.  _ArXiv preprint_ , abs/2310.06825, 2023a.
  * Jiang et al. (2024) Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, L’elio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mixtral of experts.  2024\.
  * Jiang et al. (2023b) Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin.  Llm-blender: Ensembling large language models with pairwise ranking and generative fusion.  In _Annual Meeting of the Association for Computational Linguistics_ , 2023b.
  * Lambert et al. (2024) Nathan Lambert, Valentina Pyatkin, Jacob Daniel Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Raghavi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hanna Hajishirzi.  Rewardbench: Evaluating reward models for language modeling.  2024\.
  * Li et al. (2023a) Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu.  Generative judge for evaluating alignment.  _ArXiv preprint_ , abs/2310.05470, 2023a.
  * Li et al. (2024) Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi.  Gsm-plus: A comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers.  _ArXiv preprint_ , abs/2402.19255, 2024.
  * Li et al. (2023b) Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li.  Taco: Topics in algorithmic code generation dataset.  volume abs/2312.14852, 2023b.
  * Li et al. (2022) Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals.  Competition-level code generation with alphacode.  volume abs/2203.07814, 2022.
  * Lian et al. (2023) Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and ”Teknium”.  Openorca: An open dataset of gpt augmented flan reasoning traces.  https://https://huggingface.co/Open-Orca/OpenOrca, 2023.
  * Liu et al. (2023) Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He.  What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning.  2023\.
  * Lu et al. (2023) Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan.  Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning.  In _Proceedings of ICLR_ , 2023.
  * Luo et al. (2023a) Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang.  Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct.  _ArXiv preprint_ , abs/2308.09583, 2023a.
  * Luo et al. (2023b) Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang.  Wizardcoder: Empowering code large language models with evol-instruct, 2023b.
  * Miao et al. (2020) Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su.  A diverse corpus for evaluating and developing English math word problem solvers.  In _Proc. of ACL_ , 2020.
  * Mishra et al. (2022) Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, and Ashwin Kalyan.  NumGLUE: A suite of fundamental yet challenging mathematical reasoning tasks.  In _Proc. of ACL_ , 2022.
  * Mitra et al. (2023) Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agrawal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, Hamid Palangi, Guoqing Zheng, Corby Rosset, Hamed Khanpour, and Ahmed Awadallah.  Orca 2: Teaching small language models how to reason.  _ArXiv preprint_ , abs/2311.11045, 2023.
  * Mitra et al. (2024) Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah.  Orca-math: Unlocking the potential of slms in grade school math.  _ArXiv preprint_ , abs/2402.14830, 2024.
  * OpenAI (2023) OpenAI.  Gpt-4 technical report, 2023.
  * Pasupat & Liang (2015) Panupong Pasupat and Percy Liang.  Compositional semantic parsing on semi-structured tables.  In _Proc. of ACL_ , 2015.
  * Patel et al. (2021) Arkil Patel, Satwik Bhattamishra, and Navin Goyal.  Are NLP models really able to solve simple math word problems?  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2021.
  * Qian et al. (2023) Cheng Qian, Chi Han, Yi Ren Fung, Yujia Qin, Zhiyuan Liu, and Heng Ji.  Creator: Tool creation for disentangling abstract and concrete reasoning of large language models.  In _Conference on Empirical Methods in Natural Language Processing_ , 2023.
  * Qin et al. (2023) Yujia Qin, Shi Liang, Yining Ye, Kunlun Zhu, Lan Yan, Ya-Ting Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Marc H. Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun.  Toolllm: Facilitating large language models to master 16000+ real-world apis.  _ArXiv preprint_ , abs/2307.16789, 2023.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  _ArXiv preprint_ , abs/2305.18290, 2023.
  * Roziere et al. (2023) Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al.  Code llama: Open foundation models for code.  _ArXiv preprint_ , abs/2308.12950, 2023.
  * Shao et al. (2024) Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, YK Li, Y Wu, and Daya Guo.  Deepseekmath: Pushing the limits of mathematical reasoning in open language models.  _ArXiv preprint_ , abs/2402.03300, 2024.
  * Suzgun et al. (2022) Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei.  Challenging big-bench tasks and whether chain-of-thought can solve them.  _ArXiv preprint_ , abs/2210.09261, 2022.
  * Toshniwal et al. (2024) Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman.  Openmathinstruct-1: A 1.8 million math instruction tuning dataset.  _arXiv preprint arXiv: Arxiv-2402.10176_ , 2024.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models.  _ArXiv preprint_ , abs/2307.09288, 2023.
  * Tunstall et al. (2023) Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al.  Zephyr: Direct distillation of lm alignment.  _ArXiv preprint_ , abs/2310.16944, 2023.
  * Wang et al. (2023a) Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu.  Openchat: Advancing open-source language models with mixed-quality data.  _ArXiv preprint_ , abs/2309.11235, 2023a.
  * Wang et al. (2023b) Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji.  Mint: Evaluating llms in multi-turn interaction with tools and language feedback.  _ArXiv preprint_ , abs/2309.10691, 2023b.
  * Wang et al. (2024) Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji.  Executable code actions elicit better llm agents.  _ArXiv preprint_ , abs/2402.01030, 2024.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou.  Chain of thought prompting elicits reasoning in large language models.  _ArXiv preprint_ , abs/2201.11903, 2022.
  * Wei et al. (2023) Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang.  Magicoder: Source code is all you need, 2023.
  * Weyssow et al. (2024) Martin Weyssow, Aton Kamanda, and Houari Sahraoui.  Codeultrafeedback: An llm-as-a-judge dataset for aligning large language models to coding preferences.  _ArXiv preprint_ , abs/2403.09032, 2024.
  * Xu et al. (2024) Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Yang Wang.  Perils of self-feedback: Self-bias amplifies in large language models.  _ArXiv preprint_ , abs/2402.11436, 2024.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In _Proc. of EMNLP_ , 2018.
  * Yu et al. (2020) Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng.  Reclor: A reading comprehension dataset requiring logical reasoning.  In _Proc. of ICLR_ , 2020.
  * Yuan et al. (2023) Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi Ren Fung, Hao Peng, and Heng Ji.  Craft: Customizing llms by creating and retrieving from specialized toolsets.  _ArXiv preprint_ , abs/2309.17428, 2023.
  * Yue et al. (2023) Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen.  Mammoth: Building math generalist models through hybrid instruction tuning.  _ArXiv preprint_ , abs/2309.05653, 2023.
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Haotong Zhang, Joseph Gonzalez, and Ion Stoica.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _ArXiv preprint_ , abs/2306.05685, 2023.
  * Zheng et al. (2024) Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue.  Opencodeinterpreter: Integrating code generation with execution and refinement.  _ArXiv preprint_ , abs/2402.14658, 2024.
  * Zhou et al. (2023) Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou.  Instruction-following evaluation for large language models.  _ArXiv preprint_ , abs/2311.07911, 2023.
  * Zhu et al. (2023) Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao.  Starling-7b: Improving llm helpfulness & harmlessness with rlaif, 2023.

##  Appendix A Additional Details in UltraInteract Construction

###  A.1 Dataset Details

Math. We adopt GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b),
MathQA (Amini et al., 2019), and NumGLUE (Mishra et al., 2022)for mathematic
reasoning, and include TabMWP (Lu et al., 2023) for tabular processing. We
retain all the instructions for all datasets except MathQA, NumGLUE, and
TabMWP. MathQA divides problems into different categories according to the
topics and annotates the formula that indicates the pattern needed to solve
each problem. We apply stratified sampling to sample at most five problems for
each pattern and prioritize the problems that come from the long-tail
category. Numglue contains eight different reasoning tasks and we discard Task
5 (Reading Comprehension + Explicit Numerical Reasoning), Task 6 (Reading
Comprehension + Implicit Numerical Reasoning), and Task 7 (Quantitative NLI)
due to the simplicity Mishra et al. (2022). For TabMWP, we only keep the
questions with difficulty levels 4 and 5 since the rest are too easy for
current state-of-the-art models.

Code. We focus on programming with Python for the simplicity of integration of
the interpreter. We use CodeContest (Li et al., 2022) and TACO (Li et al.,
2023b), two competition-level coding datasets collected from various online
platforms. We filter out the overlapped questions. Note that part of the
questions in TACO only contain ground-truth solutions and do not contain test
cases for evaluation, hence we apply GPT-4 to generate 12 test case inputs (4
basic inputs, 4 edge cases, and 4 large numbers) for each question and then
execute the ground-truth solution snippets to produce outputs. Given that the
two datasets mainly focus on competition problems that may deviate from real-
world daily uses, we exclusively adopt Magicoder-Evol-Instruct (Luo et al.,
2023b; Wei et al., 2023), the only dataset in our selection that does not
contain test cases or ground-truth solutions. We employ GPT-4 Turbo to judge
the correctness of generated code during interaction, and therefore we do not
use this dataset for preference learning since we cannot rigorously construct
pairs of correct and incorrect actions limited by the evaluation reliability.
We also include WikiTableQuestions (Pasupat & Liang, 2015) for table
processing with code.

Logical Reasoning. we use the multi-hop reasoning datasets HotpotQA (Yang et
al., 2018) and StrategyQA (Geva et al., 2021), and the logical reasoning
dataset ReClor (Yu et al., 2020). We follow the setting of Wang et al. (2023b)
and convert HotpotQA to a generation task, removing the contexts and requiring
LLMs to search relevant information using Wikipedia API.

Table 6: UltraInteract covers a diverse set of datasets spanning three tasks.

Task | Datasets
---|---
Math |  GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b), MathQA (Amini et al., 2019), NumGlue (Mishra et al., 2022), TabMWP (Lu et al., 2023)
Coding |  CodeContest (Li et al., 2022), TACO (Li et al., 2023b), WikiTableQuestions (Pasupat & Liang, 2015), Magicoder-Evol-Instruct (Luo et al., 2023b; Wei et al., 2023)
Logic |  ReClor (Yu et al., 2020), HotpotQA (Yang et al., 2018), StrategyQA (Geva et al., 2021)

###  A.2 Details on Preference Tree Construction

Models Adopted for Incorrect Action Sampling. We randomly sample one model
from Mistral-7B-Instruct-v0.2, DeepSeek-Coder-33B-Instruct,
Mixtral-8x7B-Instruct, and DeepSeek-LLM-67B-Chat to generate one incorrect
action to pair with each correct one.

Correct Action Generation Based on Ground Truth Annotations.

We adopt GPT-3.5 Turbo as the generator to generate correct actions based on
ground truth considering the instruction-following ability. We provide
different access to the ground truth information for different tasks,
specifically: (1) For coding, where test cases are black boxes to reference
solutions, we provide full access to the solution codes. The actor model will
add step marks and corresponding explanations to the ground-truth code to make
it easier to understand, or further refine the code for optimization. (2) For
tool-free math problems, to avoid the actor model directly copying the answers
to pass the correctness checking, we mask the answer numbers in the rationale
before providing it to LLMs. This approach can better ensure response quality
since it encourages LLMs to generate responses with complete reasoning chains
with each step clearly marked. (3) For program-enhanced math reasoning, we
first translate the textual rationale into code. Then, we either directly
provide it to the actor model to generate plans, or ask the actor model to
convert the code into modularization programming and then make plans to create
tools to solve problems.

###  A.3 Data Decomtamination

We conduct careful decontamination. Firstly, for LeetCode, we apply the Exact
Substring Matching Algorithm333https://github.com/bigcode-project/bigcode-
dataset/tree/main/decontamination to compare with each instruction in the
UltraInteract and find no overlaps. For others, we perform 8-gram exact
matching to compare UltraInteract instructions with test sets of the same
task. We remove those instructions that overlap 8 grams with any test sample.

###  A.4 Detailed Statistics

In total, UltraInteract has 86K instructions and 220K action pairs. The Total
# Pairs does not equal Total # Turns in UltraInteract, since we fail to
generate sufficient correct actions for every incorrect action in multi-turn
trajectories mainly due to a lack of sufficient ground truth annotations. The
total # pairs may not equal # correct answers, either, because it is also
difficult and unnecessary to sample incorrect actions for the correct ones for
some simple instructions. We present the specific information for each
dataset. In particular, we list information on human annotation in each
dataset, which plays an important role in correct action generation (§2.3 and
Appendix A.2). All three steps of correct action sampling methods mentioned in
§2.3 can be applied to datasets that have rationales, while for datasets only
containing answers, only the first two steps are applicable. We do not apply
any of the three-step methods to generate correct answers for Magicoder, the
only dataset without any human annotation, to construct preference pairs.

Table 7: Stats breakdown

Task | Dataset | w/ Tool? | # Prompts | # Pairs | # Correct Answers. | Avg. Length | Human Annotation
---|---|---|---|---|---|---|---
Has Answer? | Has Rationale?
Math | GSM8K | ✓ | 4,522 | 10,277 | 17,392 | 1,746.7 | ✓ | ✓
✗ | 7,257 | 10,879 | 15,752 | 823.3 | ✓ | ✓
MATH | ✓ | 7,474 | 22,905 | 34,667 | 1,189.0 | ✓ | ✓
✗ | 7,471 | 25,765 | 36,005 | 1,735.0 | ✓ | ✓
MathQA | ✓ | 7,552 | 15,079 | 20,328 | 2,338.5 | ✓ | ✓
✗ | 7,159 | 17,743 | 22,500 | 1,916.3 | ✓ | ✓
NumGLUE | ✓ | 3,020 | 3,601 | 5,717 | 1,474.6 | ✓ | ✗
✗ | 2,835 | 2,987 | 4,273 | 1,056.1 | ✓ | ✗
TabMWP | ✓ | 3,117 | 4,135 | 6,083 | 842.6 | ✓ | ✗
Coding | CodeContest | - | 8,167 | 44,319 | 44,666 | 2,061.7 | ✓ | ✓
TACO | - | 9,016 | 50,877 | 58,191 | 2,143.5 | ✓ | ✓
WikiTableQuestions | - | 1,401 | 1,544 | 1,738 | 1,794.8 | ✓ | ✗
Magicoder-Evol-Instruct | - | 10,374 | 0 | 10,238 | 687.1 | ✗ | ✗
Logic | Reclor | ✗ | 4,467 | 7,958 | 7,231 | 1,266.7 | ✓ | ✗
HotpotQA | ✓ | 1,182 | 1,009 | 1,230 | 1,333.2 | ✓ | ✗
StrategyQA | ✓ | 904 | 741 | 968 | 1,256.2 | ✓ | ✗

##  Appendix B Additional Details on Training Eurus Models

Supervised Fine-Tuning. We finetune base models for 1 epoch with a 2e-5
learning rate and 0.1 warmup ratio using a cosine scheduler. For Eurus-7B, we
mix 32K UltraChat, 30K ShareGPT, and 50K OpenOrca. For For Eurus-70B, we mix
63K UltraChat, 30K ShareGPT, and 70K OpenOrca.

Preference Learning. For hyperparameters, all $\beta$ is set to 0.1, and
$\lambda_{+}$/$\lambda_{-}$ in KTO is set to 1.33 as recommended. We finetune
models for 1 epoch with a 5e-7 learning rate and 0.1 warmup ratio using a
cosine scheduler.

Reward Modeling. We train RM for 1 epoch with lr=1e-5 learning rate. We also
use a cosine scheduler with a warmup ratio of 0.1.

Regarding pair augmentation, we scale up the pairs by matching every correct
action for each instruction with one incorrect action of other turns. This
leads to NxN pairs of single-turn actions for a trajectory of depth N. We
remove the action pairs consisting of nodes at the same turn, as they are
already part of the multi-turn trajectory pairs we included. Next, to avoid
overfitting on the training set, we only select instructions with NxN $\leq$
10, and for these instructions, we randomly sample at most 9 pairs with each
action occurring no more than 3 times. This leads to an augmentation of 240k
single-turn action pairs.

Table 8: MMLU and MT-Bench.

Model | MMLU | MT-Bench
---|---|---
$\sim$7B
Mistral-7B-Instruct-v0.2 | 58.9 | 7.60
Zephyr-7B-$\beta$ | 59.7 | 7.34
OpenChat-3.5-1210 | 63.4 | 7.81
Starling-LM-7B-$\alpha$ | 64.0 | 8.09
Magicoder-S-DS-6.7B | 37.1 | -
OpenCI-DS-6.7B | 37.2 | -
MAmmoTH-7B-Mistral | 56.2 | -
WizardMath-7B-v1.1 | 60.3 | -
OpenMath-Mistral-7B | 58.3 | -
Eurus-7B-SFT | 61.8 | 7.15
\+ DPO | 62.4 | 7.38
\+ KTO | 62.2 | 7.38
\+ NCA | 62.2 | 7.38
$\sim$40B
Mixtral-8x7B-Instruct | 70.3 | 8.30
DeepSeek-Coder-33B-Ins | 40.2 | -
$\sim$70B
CodeLLaMA-70B-Instruct | 55.1 | -
DeepSeek-LM-67B-Chat | 72.3 | -
QWen1.5-72B-Chat | 72.9 | 8.61
OpenCI-CL-70B | 52.4 | -
OpenMath-CL-70B | 60.2 | -
Eurus-70B-SFT | 59.1 | 7.69
\+ KTO | 59.5 | 7.93
\+ NCA | 59.4 | 7.54
Proprietary Models
GPT-3.5 Turbo | 70.0 | 7.94
GPT-4 | 86.4 | 8.96

##  Appendix C Additional Evaluation Results of Eurus

Detailed Setup in §4. For math, we test both textual reasoning and program-
enhanced settings and report the best performance of the two. All evaluations
are conducted in 0-shot CoT with two exceptions: BBH uses 3 shots and IFEval
does not use CoT. For MINT, we select MATH, TheoremQA, and MMLU-math from
“reasoning” as a new “math” split. We also evaluate 5-shot MMLU (Hendrycks et
al., 2021a) for STEM knowledge and MT-Bench (Zheng et al., 2023) for
conversation abilities to study whether Eurus needs to trade off other
capabilities for reasoning.

Results. Results are shown in Table 8.

On MMLU, Eurus outperforms baselines dedicated to coding and math, and
achieves higher results than Mistral-Instruct-v0.2 and CodeLLaMA-70B-Instruct,
the official aligned versions of our base model built by their authors.
Compared to general-purpose baseline models, Eurus-7B achieves comparable
performance with the top-performance OpenChat and Starling-LM, though
Eurus-70B does not achieve the same level of performance as other general-
purpose models, which is expected due to the gap in the base models since
CodeLLaMA-70B has not been intentionally optimized for knowledge.

On MT-Bench, we report baseline numbers from the official
leaderboard444https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard.
Eurus matches the performance of mainstream open-source general-purpose
models, and Eurus-70B-KTO further achieves the score of GPT-3.5 Turbo.

##  Appendix D Detailed Results on Reward Modeling

###  D.1 Additional Results on Reranking

Table 9: Detailed results of reranking Mistral-Instruct-v0.2’s responses on
coding and math.

Datasets | HumanEval | MBPP | GSM8K | MATH
---|---|---|---|---
N | 2 | 4 | 8 | 16 | 2 | 4 | 8 | 16 | 2 | 4 | 8 | 16 | 2 | 4 | 8 | 16
Random | 41.5 | 39.0 | 40.2 | 39.6 | 33.1 | 33.6 | 34.3 | 30.1 | 45.0 | 43.1 | 44.5 | 40.2 | 11.5 | 11.3 | 10.0 | 8.5
Top Logits | 43.3 | 43.3 | 43.3 | 43.3 | 35.3 | 35.3 | 35.3 | 35.3 | 45.7 | 45.7 | 45.7 | 45.7 | 12.1 | 12.1 | 12.1 | 12.1
Self-Consistency | 43.3 | 42.7 | 42.1 | 40.9 | 35.3 | 36.3 | 36.6 | 37.1 | 45.7 | 49.5 | 52.2 | 52.8 | 12.1 | 13.8 | 15.8 | 16.8
Starling-RM-34B | 47.6 | 47.0 | 49.4 | 45.7 | 37.8 | 38.8 | 39.6 | 40.4 | 49.1 | 52.8 | 56.0 | 56.5 | 6.5 | 7.2 | 7.7 | 7.7
Eurus-RM-7B | 44.5 | 45.7 | 47.6 | 47.0 | 39.3 | 42.6 | 43.4 | 43.9 | 49.8 | 53.7 | 56.3 | 57.3 | 14.3 | 16.2 | 17.1 | 17.3
w/o $\mathcal{L_{DR}}$ | 45.7 | 44.5 | 46.3 | 50.0 | 39.3 | 42.4 | 42.4 | 42.1 | 49.4 | 53.2 | 55.4 | 56.3 | 14.2 | 16.1 | 17.0 | 16.9
w/o $\mathcal{L_{BT}}$ | 45.1 | 44.5 | 47.0 | 48.2 | 38.6 | 40.6 | 39.6 | 40.1 | 49.1 | 52.5 | 55.2 | 57.8 | 14.3 | 16.3 | 17.2 | 17.1
w/o US | 45.7 | 47.0 | 49.4 | 50.6 | 39.3 | 41.1 | 41.4 | 42.9 | 49.4 | 53.8 | 57.4 | 58.7 | 14.5 | 16.6 | 17.2 | 17.5
w/o UF + US | 43.9 | 43.3 | 47.0 | 46.3 | 36.3 | 38.1 | 36.6 | 35.3 | 49.4 | 52.3 | 54.6 | 57.2 | 14.3 | 16.5 | 17.4 | 17.4
Pass@N | 62.8 | 73.8 | 88.4 | 92.7 | 42.4 | 48.1 | 52.6 | 58.6 | 54.9 | 64.1 | 73.2 | 80.4 | 16.9 | 22.7 | 28.9 | 35.5

We present the full results on reranking in Table 9, where the conclusions are
consistent with those drawn from §D: (1) Our reward models always achieve the
highest accuracy on all test sets across different N, except when N=2 on
HumanEval. (2) Both $\mathcal{L_{\text{BT}}}$ and $\mathcal{L_{\text{DR}}}$
consistently help improve reranking performance on three test sets except for
HumanEval, where removing either of the objectives can prevent the accuracy
from dropping when increasing N from 8 to 16. (3) Modeling safety hurts
reranking performance in reasoning. When removing UltraSafety from the
training data, the RM achieves higher accuracies than Eurus-RM-7B except on
MBPP.

##  Appendix E Detailed Ablation Results

We present the full results of §5 in Table 10, with detailed metrics on all
coding and math datasets.

Table 10: Ablation Study.

Model | Coding | Math | Reasoning | Ins-Following | Avg.
---|---|---|---|---|---
HumanEval | MBPP | LeetCode | GSM8K | MATH | TheoremQA | SVAMP | ASDiv | BBH | IFEval
Eurus-7B-SFT | 55.5 | 59.1 | 20.0 | 73.7 | 32.6 | 20.0 | 82.2 | 84.1 | 64.6 | 44.0 | 53.6
Ground-Truth | 46.3 | 46.4 | 8.9 | 62.2 | 15.0 | 9.6 | 75.1 | 68.8 | 64.4 | 42.9 | 44.0
Open-Source Only | 38.4 | 44.1 | 11.1 | 45.3 | 10.8 | 9.3 | 52.7 | 49.4 | 65.3 | 43.6 | 37.0
UltraInteract Only | 46.3 | 50.1 | 15.6 | 67.6 | 30.9 | 20.1 | 80.4 | 82.0 | 67.0 | 17.4 | 47.7