# Microstructure evolution and tensile behaviour of a cold rolled 8 wt% Mn
medium manganese steel

T. W. J. Kwok  P. Gong  X. Xu  J. Nutter  W. M. Rainforth  D. Dye  Department
of Materials, Royal School of Mines, Imperial College London, Prince Consort
Road, London, SW7 2BP, United Kingdom  Department of Materials Science and
Engineering, The University of Sheffield, Western Bank, Sheffield, S10 2TN,
United Kingdom

###### Abstract

A novel medium manganese steel named Novalloy with composition
Fe-8.3Mn-3.8Al-1.8Si-0.5C-0.06V-0.05Sn was developed and thermomechanically
processed through hot rolling and intercritical annealing. The steel possessed
a yield strength of 1 GPa, tensile strength of 1.13 GPa and ductility of 41%.
In order to study the effect of cold rolling after intercritical annealing on
subsequent tensile properties, the steel was further cold rolled up to 20%
reduction. After cold rolling, it was observed that the strain hardening rate
increased continuously with increasing cold rolling reduction but without a
significant drop in ductility during subsequent tensile tests. The
microstructural evolution with cold rolling reduction was analysed to
understand the mechanisms behind this enhanced TRIP effect. It was found that
cold rolling activated additional twinning systems which provided a large
number of potent nucleation sites for strain induced martensite to form during
subsequent tensile tests.

††journal: Acta Materialia

##  1 Introduction

Medium Mn steels (4$-$12 wt% Mn) are an emerging class of duplex
($\gamma+\alpha$) steels that have received considerable research attention.
Medium Mn steels have shown to exhibit a successive twinning and
transformation induced plasticity (TWIP$+$TRIP) effect if the Stacking Fault
Energy (SFE) and stability of the austenite phase are adjusted into the
correct regime during an Intercritical Annealing (IA) heat treatment [1, 2].
The TWIP$+$TRIP effect allowed some medium Mn steels to possess large
ductilities of 60$-$70% [2, 3, 4, 5] and therefore show great potential for
use in energy absorbing applications.

Many medium Mn steels which exhibit the TWIP$+$TRIP effect are manufactured by
a thermomechanical process composed of hot rolling followed by cold rolling
and then IA. The IA is a key step where the solute elements, such as Mn, Al
and Si, are able to partition into or out of the austenite phase and therefore
alter the SFE and also stability. IA is greatly enhanced with prior cold
rolling such that IA can be completed (i.e. reaching thermodynamic
equilibrium) within several minutes, allowing IA to be conducted on a
Continuous Annealing Line (CAL) [6] where the heating stage is usually between
2$-$3 minutes long. Otherwise, IA would have to be conducted in batch furnaces
for a minimum of several hours [7], severely reducing productivity.

One potential problem of cold rolling before IA is cracking. The hot rolled
microstructure of many medium Mn steels in the literature are either partially
or fully martensitic [8, 9, 2]. Therefore, a tempering step before cold
rolling is usually necessary as the hot rolled strip is potentially very hard
with poor ductility [10]. In conventional strip mills, hot rolled strip is
usually 2$-$3 mm thick which then needs to be cold rolled to 1$-$1.5 mm for
automotive applications. The rolling loads necessary to make the reduction in
medium Mn steel can be very large as shown in a study by Buchely et. al. [11].
However, it is anticipated that hot rolling capabilities to produce thinner
gauges will continue to improve. For example, recent advances in strip casting
technology have shown that medium Mn steel can be successfully cast into 2.5
mm thick strip [12]. With this in mind, reliance on cold rolling for heavy
gauge reduction can be reduced. Cold rolling can then be pushed to the end of
the processing route (i.e. hot rolling$-$IA$-$cold rolling) where it is still
useful for other purposes such as dimensional control. From an alloy design
perspective, element partitioning will no longer be able to rely on cold
rolling to improve partitioning kinetics. Ideally, the steel should still be
able to exhibit the TWIP$+$TRIP effect but through a simplified hot rolling
and IA thermomechanical process route without the need for cold rolling.
Several medium Mn steels in the literature have successfully demonstrated this
[13, 14, 9] but require IA durations between 10$-$30 min which may be
challenging in a CAL.

Studies on IA then cold rolled medium Mn steels are relatively sparse compared
to studies on cold rolled then IA steels. Sun et. al. [15] studied the cold
rollability of high Si, $\delta$-ferrite containing medium Mn steels and found
that brittle $\delta$-ferrite could lead to cracking during cold rolling but
more importantly that the TWIP$+$TRIP effect greatly enhanced the amount of
cold reduction without cracking as compared to another steel which only showed
the TRIP effect. Wang et. al. [5] and Li et. al. [16] studied the effect of
uniaxial pre-straining on the formation of Lüders type bands and concluded
that pre-straining increases the dislocation density in the austenite,
therefore delaying the onset of the TRIP effect which is known to cause the
formation of Lüders type bands [17]. Nevertheless the effect of cold rolling
on subsequent tensile behaviour and how it may affect the TWIP$+$TRIP effect
is relatively unknown.

This study therefore aims to accomplish two points. First to develop a cost
effective high strength medium Mn steel that exhibits the TWIP$+$TRIP effect
through hot rolling and IA only. Secondly, to investigate the effects of cold
rolling (up to 20% reduction) after IA on tensile behaviour.

##  2 Experimental

In order to minimise cost due to microalloying, a carbide optimisation study
was first conducted. Three steels with nominal compositions of
Fe-8.3Mn-3.8Al-1.8Si-0.5C-0.05Sn-(0, 0.04, 0.06)V in mass percent were arc
melted from pure elements to produce a 400 g ingot with dimensions of 60
$\times$ 23 $\times$ 23 mm. The chosen composition was based on a previous
alloy developed by Lee and DeCooman [14]. The ingot was sectioned into 4 bars
measuring 60 $\times$ 10 $\times$ 10 mm. The bars were quartz encapsulated in
low pressure Ar and homogenised at 1250 °C for 24 h before quenching in water.
Each bar was reheated to 1000 °C for 30 min prior to hot rolling.
Thermomechanical processing was conducted in 5 passes with decreasing
reductions from 1000 °C to 850 °C with a total reduction of approximately 85%.
The hot rolled strips were quenched immediately after the final pass. After
hot rolling, the strips were intercritically annealed at 750 °C for 5 min
before being allowed to air cool. The 0.06V steel was subsequently cold rolled
to 10% and 20% reduction.

Tensile samples with gauge dimensions of 19 $\times$ 1.5 $\times$ 1 mm were
machined via Electric Discharge Machining (EDM) from the rolled strips such
that the rolling direction was parallel to the tensile direction. Tensile
testing was conducted at a nominal strain rate of $10^{-3}$ s-1 with an
extensometer which was removed at 10% engineering strain.

Secondary Electron Microscopy (SEM), Electron Backscatter Diffraction (EBSD)
and Energy Dispersive Spectroscopy (EDS) were conducted on a Zeiss Sigma FE-
SEM equipped with a Bruker EBSD detector and Bruker XFlash 6160 EDS detector.
Transmission Electron Microscopy (TEM) was conducted on a JEOL JEM-F200
operated at an accelerating voltage of 200 kV. Phase and orientation mapping
was carried out using the NanoMEGAS ASTAR system, which uses Precession
Electron Diffraction (PED) to capture EBSD-like maps in the TEM. PED patterns
were collected with a precession angle of 0.7 °, a precession frequency of 100
Hz and a step size of 2.5 nm. The TEM was operated with a spot size of 7 and a
10 µm condenser aperture producing a beam size of minimal diameter ($<$2 nm
FWHM).

Samples for SEM, EBSD and EDS were mechanically ground and polished with an
OP-U suspension. Samples for TEM were cut via EDM from heat treated blanks,
mechanically ground to a thickness below 60 µm and electrolytically polished
in a Struers Tenupol twin-jet electropolishing unit using a solution
containing 5% perchloric acid, 35% butyl-alcohol and 60% methanol at a
temperature of $-40$ °C.

Figure 1: Comparison of SFE and Md30 of the austenite phase in several Medium
Mn steels (MMS) and High Mn TWIP steels (HMS) in the literature. Inset table:
Change in SFE and Md30 with different additions to austenite with an initial
composition of Fe-6.55Mn-1.33Al-1.19Si-0.48C, SFE of 20 mJ m-2 and Md30 of 275
°C [18]. Data from: [19, 20, 21, 22, 23, 18, 2, 10].

##  3 Results

###  3.1 Alloying concept

There are many considerations to make when designing a medium Mn steel
composition and its complementary thermomechanical process route. It is
therefore useful to first impose several boundary conditions both in terms of
composition and processing. A Mn content of 8$-$8.5 wt% was chosen as it
represented the upper limit of what can be tolerated in conventional secondary
steelmelting practice. Si content was chosen to be below 3 wt% as Si might
stabilise and embrittle $\delta$-ferrite at room temperature when added in
excess [24]. Although Al can significantly reduce the density of steel [25],
Al content was chosen to be kept below 4$-$5 wt% as Al is known to clog
continuous casting nozzles [26] and form brittle $\kappa$-carbides in medium
Mn steels when added in excessive amounts [25, 27]. V can also be added for
carbide formation and precipitation strengthening [28, 14] but its content
should ideally be kept below 0.1 wt% to minimise the alloying cost of the
steel. Lastly, even though Sn is a known tramp element in steel [29], 0.05 wt%
Sn was deliberately added to potentially improve the galvanisability of the
steel [30].

The next step is to consider the composition of the austenite phase. Figure 1
shows a comparison between the stacking fault energy and stability of the
austenite phase in several TRIP-type and TWIP$+$TRIP-type medium Mn steels as
well as high Mn TWIP steels from the literature. The SFE, if not reported by
the authors, was calculated according to the method by Sun et. al. [10].
Austenite stability, represented by Md30, is defined as the temperature where
half of the total austenite transforms to martensite at a strain of 30% and
can be calculated according to the equation [31, 32, 10]:

$\begin{split}\small
Md_{30}(\mathrm{\SIUnitSymbolDegree}C)=\,&551-462C-8.1Mn-9.2Si)\\\
&-1.42(-3.29-6.64\log_{10}d-8)\end{split}$ (1)

where compositions are given in wt% and $d$ is the grain size in µm. A
combination of a moderate SFE and low Md30 (i.e. higher austenite stability)
was chosen to promote twinning and a controlled TRIP effect respectively [33].
Since the TWIP$+$TRIP mechanism is only operative in the austenite phase, a
high austenite fraction is therefore desired ($\geq 0.5$) for greater
elongation and strain hardening rate.

Table 1: Bulk composition of 0.06V steel measured by ICP, and IGF for elements
marked by $\dagger$.

Mn Al Si C† V Sn N† P S† 8.29 3.78 1.81 0.467 0.06 0.047 0.001 $<$0.005 0.003

The thermodynamic software Thermo-Calc is often used to provide guidance on
some important processing parameters such as the optimal IA temperature,
precipitation temperatures, etc. Inductively Coupled Plasma (ICP) and Inert
Gas Fusion (IGF) were used to measure the bulk composition of the 0.06V steel
and the results are shown in Table 1. The corresponding Thermo-Calc property
diagram using the TCFE-7.0 database is shown in Figure 2. A wide austenitic
processing window between 880$-$1200 °C was available for hot rolling. The
$\delta$-ferrite phase was also not expected to be present at room temperature
in this alloy. VC was expected to precipitate at 940 °C while graphite and the
other MxCy carbides were expected to precipitate at temperatures 730 °C and
below. These carbides are thought to be deleterious to ductility and IA should
be conducted above 730 °C to avoid the formation of these unwanted carbides.
Therefore, the practical IA window was between 730$-$880 °C where the
austenite volume fraction would vary between 0.63$-$1.0 respectively.

Figure 2: Thermo-Calc property diagram of 0.06V steel.

###  3.2 Tensile properties

The tensile curves of the investigated steel with different V contents in the
as-annealed condition are shown in Figure 3. All three steels showed a high
yield strength and elongation but a low strain hardening rate. The steels also
showed continuous yielding with no Yield Point Elongation (YPE) and the
addition of 0.05 wt% Sn did not appear to deleteriously affect overall tensile
properties.

The addition of up to 0.06 wt% V resulted in an increase in yield strength
from 950 to 1005 MPa but did not affect the strain hardening rate or
elongation. Since the addition of 0.06 wt% V was able to result in a
GigaPascal-class steel [34], the 0.06V steel was chosen for further
processing.

Figure 3: Engineering tensile curves of the investigated steel with different
V content. Inset: early yielding behaviour. Figure 4: (a) Engineering tensile
curves and (b) true tensile curves and hardening rate of the as-annealed and
cold rolled 0.06V steel.

The 0.06V steel was further cold rolled to a reduction of 10% and 20% (hereby
known as CR10 and CR20 respectively), and the resulting tensile curves are
shown in Figure 4. In both the CR10 and CR20 samples, a pronounced YPE was
observed. The presence of YPE strongly indicates that TRIP rather than TWIP
was the dominant deformation mechanism at yielding [35, 17, 36].

Table 2: Tensile properties of the investigated steels. Alloy | $\sigma_{0.2}$ | $\sigma_{UTS}$ | $\epsilon$ | U30
---|---|---|---|---
(MPa) | (MPa) | (%) | (MJ m-3)
0V | 950 | 1100 | 41 | 361
0.04V | 985 | 1130 | 41 | 372
0.06V | 1005 | 1170 | 42 | 379
CR10 | 1110 | 1260 | 34 | 389
CR20 | 1250 | 1490 | 34 | 445

The yield strength, tensile strength and strain hardening rate increased
significantly with increasing cold rolling reduction. The strain hardening
rate in the as-annealed sample was not constant throughout the tensile curve,
most likely due to different deformation mechanisms being active at different
stages but averaged out to approximately 2 GPa. In the post-YPE section of the
tensile curve, the strain hardening rate of the CR10 sample was approximately
3 GPa. However, in the CR20 sample there was a sustained hardening rate of 3.5
GPa in the post-YPE region. It is also noteworthy that both CR10 and CR20
still retained a large ductility at such high strengths. A summary of the
tensile properties of the investigated steels is shown in Table 2 including
the energy absorption parameter U30 which is determined as the area under the
true tensile curve up to a true strain of 0.3 [35].

###  3.3 Carbide precipitation microstructure

The small strengthening increments shown in Figure 3 can be attributed to the
increase in volume fraction of VC within the microstructure. From the TEM-
Bright Field (BF) micrographs in Figure 5, small nanometre-sized carbides can
be seen in both ferrite and austenite grains and also along grain boundaries.
TEM-EDS revealed them to be V-rich in composition. To determine the
strengthening contribution of these carbides, the Ashby-Orowan equation after
Gladman [37] can be used:

$\Delta\sigma_{y}^{i}=\left(\frac{0.538Gbf^{0.5}}{X}\right)\,\ln\left(\frac{X}{2b}\right)$
(2)

where $\Delta\sigma_{y}^{i}$ is the change in yield strength of phase $i$
(MPa), $G$ is the shear modulus ($G_{\gamma}=75\,000$ MPa,
$G_{\alpha}=80\,000$ MPa), $b$ is the Burgers vector ($b_{\gamma}=0.250$ nm,
$b_{\alpha}=0.248$ nm) [38], $f$ is the volume fraction of carbides and $X$ is
the mean carbide diameter (nm). The overall increase in strength of the bulk
alloy can be described by the rule of mixtures:

$\Delta\sigma_{y}^{bulk}=\Delta\sigma_{y}^{\alpha}\,V_{f}^{\alpha}+\Delta\sigma_{y}^{\gamma}\,V_{f}^{\gamma}$
(3)

where $V_{f}^{\alpha}$ and $V_{f}^{\gamma}$ are the volume fractions of
ferrite and austentite respectively. Image analysis of TEM micrographs in the
0.06V sample revealed that the average VC diameter was $19\pm 4$ nm. VC size
and distribution appeared to be fairly equal between austentite and ferrite
grains. The volume fraction of carbides was determined by Thermo-Calc to be
$1.25\times 10^{-3}$. Assuming an equal volume fraction of VC in both
austenite and ferrite, the theoretical strength increment from the addition of
0.06 wt% V as calculated from Equations 2 and 3 was 70 MPa. However, the
theoretical strength increment was slightly larger than the experimentally
observed value of 55 MPa. The difference can be attributed to a slight loss of
VC volume fraction to intergranular precipitates at the grain boundaries.

Figure 5: (a) TEM-BF micrograph of 0.06V as-annealed sample. (b) Magnified
TEM-BF micrograph of the red square in (a) showing intergranular precipitates
(red arrows) and intragranular precipitates (green arrows). Diffraction
patterns obtained from (c) an austenite grain, $[100]_{\gamma}$ zone axis and
(d) a ferrite grain, $[111]_{\alpha}$ zone axis. Figure 6: EBSD Image quality
(IQ) + Phase Maps (PM) of 0.06V steel in the undeformed (a) as-annealed, (b)
CR10, (c) CR20 and deformed to failure (d) as-annealed, (e) CR10, (f) CR20
conditions. EBSD IQ+PM of the (g) as-annealed 0.06V steel deformed to true
strain of 0.26, red $-$ austenite, green $-$ ferrite or martensite. (h)
Corresponding EDS-Mn map with (h). (i) Magnified region from (g) showing both
ferrite and $\alpha$’-martensite. High angle grain boudaries in black and
austenite $\Sigma$3 boundaries in white. (j) Corresponding magnified region
from (h). (k) Austenite IPF-X, (l) ferrite IPF-X maps and (m) pole figures of
the 0.06V as-annealed sample.

###  3.4 EBSD observations

The EBSD maps of the as-annealed, CR10 and CR20 0.06V steel are shown in
Figure 6 and a summary of the phase fractions is shown in Table 3 and Figure
7. The microstructure of the as-annealed sample did not resemble the typical
lath-type microstructure in hot rolled, quenched and IA medium Mn steels [14].
Instead, the microstructure can be described as pancaked austenite grains with
larger ferrite grains on the austenite grain boundaries and finer ferrite
grains in the austenite grain interior. The phase fractions obtained were also
reasonably close to that obtained by Thermo-Calc (Table 3) which may suggest
that the equilibrium phase fraction had been attained.

The texture of the as-annealed steel is shown in Figure 6k-m. The steel was
highly textured which is typical of steels rolled below the non-
recrystallisation temperature ($T_{nr}$). While annealing twins could be found
in multiple austenite grains, the slight curvature of the annealing twins and
the lack of a recrystallisation texture imply that the as-annealed
microstructure was recovered instead of recrystallised and that the annealing
twins formed at higher temperatures during hot rolling.

Due to the difficulty in distinguishing martensite from BCC ferrite in EBSD,
the martensite fraction from any particular map was determined by summing the
BCC and Non-Indexed (NI) fractions and subtracting 32.5%, where 32.5% is the
sum of BCC and NI fractions in the undeformed as-annealed sample (Figure 6a)
and taken to be the baseline. From correlative EBSD and EDS-Mn maps in Figures
6i-j, most NI areas correspond to a region with high Mn concentration, i.e.
austenite. This suggests that the NI regions were highly deformed but have the
same composition as austenite and therefore may be assumed to be martensite
which forms upon deformation. Nevertheless, it is acknowledged that there will
be some margin of error with this indirect method of measurement.

From Figure 7, it is evident that the martensite fraction increased in the
cold rolled samples and also in subsequent tensile tests. When cold rolled to
10% reduction (CR10), the martensite fraction increased slightly, indicating
that the TRIP effect was not yet significant. However when cold rolled to 20%
reduction (CR20), the martensite fraction increased signifcantly. A tensile
sample made from the as-annealed steel was interupted at a true strain of
0.26, hereby known as TS26, such that it had the same equivalent strain
($\epsilon_{eq}$) as the CR20 sample [39]. It was found that the TS26 sample
had a slightly lower martensite fraction than CR20. When the cold rolled
samples were tensile tested to failure, it was also found that the final
martensite fraction also increased with prior cold reduction. The increase in
final martensite fraction occured most significantly in the first 10% of cold
rolling reduction (CR10) but began to saturate in the next 10% (CR20).

Figure 7: Evolution of martensite fraction with cold rolling reduction before
and after a uniaxial tensile test.

###  3.5 TEM observations

While EBSD is helpful in understanding the microstructural changes on a
macroscopic scale, it is necessary to probe the deformation structures on a
much finer scale using TEM. Figure 8 shows the microstructure after 10% cold
rolling reduction, taken from the CR10 sample. The microstructure comprised of
austenite, ferrite and also martensite. At a higher magnification, austenite
twins were observed in one grain (Figure 8b) and two sets of twins from two
twinning systems were observed in an adjacent austenite grain (Figure 8c). The
martensite grain in Figure 8d had a diffraction pattern illustrating the K-S
orientation relationship with the adjacent austenite grain in region (b). The
martensite was therefore likely to be Strain Induced Martensite (SIM) and
denoted as blocky-SIM because of its blocky morphology.

Figure 8: Deformation structures in CR10 sample with beam direction parallel
to $[110]_{\gamma}$. (a) TEM-BF of the general microstructure. (b) STEM-BF of
region b showing one active twinning system. (c) STEM- BF of region c showing
two twinning systems. (d) STEM-BF of region d showing strain-induced
martensite. Figure 9: Deformation structures in CR20 sample with beam
direction parallel to $[110]_{\gamma}$. (a) STEM-BF micrograph of the general
microstructure. (b) STEM-BF micrograph showing one active twinning system and
surrounding high dislocation density. (c) STEM-BF micrograph of a region with
two active twinning systems. (d) Diffraction pattern obtained from region in
(b). (e) TEM-BF micrograph of an austenite grain showing long lath-structures.
(f) Magnified TEM-BF micrograph from (e). (g) TEM-Dark Field (DF) obtained
from martensite spot indicated by white arrow in (h). (h) Diffraction pattern
obtained from (f). Figure 10: Strain-induced martensite in CR20 sample (a)
TEM-BF of two twinning systems and (b) corresponding NanoMegas phase map. (c)
TEM-BF of a wide twin and (d) corresponding NanoMegas phase map. Beam
direction parallel to $[110]_{\gamma}$

The microstructure after a 20% cold rolling reduction obtained from the CR20
sample is shown in Figure 9. The microstructure showed that both austenite and
martensite phases had a higher dislocation density than the CR10 sample.
Austenite grains with a high density of twins from either one twinning system
(Figure 9b) or two twinning systems (Figure 9c) were observed. The diffraction
pattern in 9d was obtained from the single twinning region in Figure 9b. The
NanoMEGAS phase map in Figure 10b revealed some very fine martensite grains in
a region similar to Figure 9c containing intersecting twins from two active
twinning systems. These martensite grains most likely nucleated at the twin
intersections through the mechanism proposed by Olsen and Cohen [40, 41] and
is widely reported in TWIP$+$TRIP-type medium Mn steels [2, 10].

From Figure 9e, a deformation structure that was not present in the CR10
sample was observed in the CR20 sample. The structure comprised of long wavy
laths which spanned the entire grain. Figures 9f-h showed that the wavy laths
were most likely kinked austenite twins which contained very fine SIM within
the twin boundaries. The SIM was also found to obey the K-S orientation
relationship with the parent austenite. However, according to the intersecting
shear mechanism by Olson and Cohen [42], it would not be possible to nucleate
martensite within a single twin. But if the twins were kinked then it might be
possible that the deformation may have provided a suitable secondary shear to
nucleate martensite within the twin [43, 44]. In many Metastable Austenitic
Stainless Steels (MASS), martensite can also be found in curved twins without
the need for an intersection with another twin [45, 46, 47]. MASS are an older
class of steels which have been extensively studied. They share a large number
of similarities with medium Mn steels such as the TWIP$+$TRIP effect and are
therefore a very useful guide to the understanding of medium Mn steels. The
SIM which forms at twin intersections and within curved twins shall be denoted
as fine-SIM due to its small and particulate morphology.

Another grain in the CR20 sample which also contained similar wavy lath-like
structures is shown in Figure 10c-d. However, the NanoMEGAS phase map revealed
a different relationship where austenite appears to have nucleated inside a
lath within a martensitic matrix. This effect was likely the result of three
stages of deformation. Firstly, the formation and thickening of an austenite
twin. Secondly, the nucleation of fine-SIM within the twin but not consuming
the entire twin. Thirdly, the austenite matrix transforms to martensite,
likely due to the growth of blocky-SIM. However, because the austenite phase
within the twin was of a different orientation to the austenite matrix, it
remains untransformed. This also shows that blocky-SIM grains in the CR20
sample were also growing with increasing cold rolling reduction.

Table 3: Phase fractions (%) of the annealed and cold rolled steels before and after tensile testing. $\dagger$ N.I. $-$ Non-indexed percentage. $\ddagger$ Martensite fraction $=$ (BCC + NI) $-32-0.5$. | FCC | BCC |  NI† | $\alpha^{\prime}\ddagger$
---|---|---|---|---
Thermo-Calc (750 °C) | 71.2 | 28.8 | - | -
0.06V - undeformed | 67.5 | 32 | 0.5 | 0
0.06V - deformed to $0.26$ | 45.3 | 48.2 | 6.5 | 22.2
0.06V - deformed to failure | 31.2 | 59.6 | 9.2 | 36.3
CR10 - undeformed | 62.1 | 35.2 | 2.7 | 5.4
CR10 - deformed to failure | 21.8 | 55.8 | 22.4 | 45.7
CR20 - undeformed | 41.2 | 47.9 | 10.9 | 26.3
CR20 - deformed to failure | 17.8 | 61.3 | 20.9 | 49.7

The microstructure of the TS26 sample is shown in Figure 11 and was very
different from the CR20 sample even though they have the same equivalent
strain. Three regions from the TS26 sample were chosen in Figure 11 to
illustrate these differences. In Region 1 (Figure 11a-d), large martensite
plates were observed to be propagating outwards in the direction of the dashed
green arrows from the interphase grain boundary marked by the dashed green
line. The martensite plates appear to be able to propagate through several
austenite twins in Figure 11b but with some distortion along the edges of the
martensite plate. This suggests that twins are not a strong obstacle to the
propagation of blocky-SIM which would render these twins ineffective to the
future formation of fine-SIM. Intersecting twins or secondary twins from a
second twinning system were not observed in the TS26 sample.

In Region 2 (Figure 11e-h), the martensite adopted a different morphology with
a curved interface as it appears to grow into the surrounding austenite grain.
Finally, in Region 3 (Figure 11i-k), twinned martensite or Stress-Assisted
Martensite (SAM) was observed. SAM has been shown to be able to form after
plastic yielding when the stress is high enough to contribute to a
sufficiently large driving force for the martensitic transformation in the
absence of alternative nucleation sites for SIM [42, 48, 49]. SAM was
therefore likely able to form in TS26 due to a combination of a low density of
twins (i.e. SIM nucleation sites) and high applied tensile stress (1425 MPa).

Figure 11: Deformation structures in TS26 sample. From region 1, (a) general
microstructure (STEM-BF), (b) magnified view showing martensite and austenite
twins (STEM-BF), NanoMegas (c) phase map and (d) IPF from red square in (b).
From region 2, (e) general microstructure (STEM-BF), (b) magnified view of
austenite and martensite regions (STEM-BF), NanoMegas (g) phase map and (h)
IPF from red square in (f). From region 3, TEM-BF micrograph of (i) entirely
martensitic region, (j) magnified view showing martensitic twins and (k)
diffraction pattern from (j).

###  3.6 Composition

The composition of each phase of the 0.06V steel was measured by SEM and TEM-
EDS and compared against the Thermo-Calc simulation in Table 4. The C content
in the austenite phase was determined by the lever rule using the phase
fractions obtained by EBSD in Table 3 and assuming negligible C solubility in
the ferrite phase.

Compositions measured by both SEM and TEM-EDS showed that Mn partitioned to
austenite, whereas Al and Si partitioned to ferrite, which is well reported in
the literature [50]. However, the degree of partitioning measured by SEM-EDS
was much lower than TEM-EDS. This is largely due to a limitation of the large
spot size of SEM-EDS in measuring the composition of a fine-grained material.
The TEM-EDS results were therefore taken to be more reliable. While the
compositions measured by TEM-EDS do not match up exactly with those predicted
by Thermo-Calc, the degree of partitioning was such that the resulting SFE was
within the known twinning regime in both high Mn TWIP steels and medium Mn
steels. With reference to the Thermo-Calc simulation, the 0.06V steel was at
phase equilibrium but not at composition equilibrium, most likely due to the
short IA duration.

##  4 Discussion

###  4.1 Effect of cold rolling on microstructure

The TEM observations of the CR10, CR20 and TS26 samples revealed a variety of
deformation structures such as twins and martensite in the microstructure. By
first comparing the CR20 and TS26 sample, it is evident that the strain path,
uniaxial tension in TS26 and plane strain compression in CR10 and CR20,
strongly influenced the presence and density of the deformation structures.
The effects of deformation mode on microstructure has been studied extensively
in MASS [39, 51, 45, 52, 53, 54] and largely show that as the mode of
deformation changes from uniaxial to multiaxial, more slip systems and also
twinning systems are activated. The same principle also appears to hold true
in the investigated steel. While both CR20 and TS26 samples had the same
equivalent strain, CR20 had a larger density of twins and even secondary twins
(Figure 9) compared to TS26 which only had a low density of primary twins
(Figure 11). This shows that cold rolling is able to produce a higher twin
density than can be achieved under uniaxial tension. However, the lack of a
high density of twins in TS26 was surprising because its SFE as calculated in
Table 4 was well within the known twinning region of TWIP steels [55].
Nevertheless, when compared to other TWIP$+$TRIP-type medium Mn steels in
Figure 1, the investigated steel had the highest SFE. This suggests that
twinning kinetics in medium Mn steel are slower and may not be directly
comparable with fully austenitic TWIP steels.

Table 4: Composition of the bulk and individual phases in the 0.06V steel measured by ICP, SEM-EDS and TEM-EDS. Uncertainties given in parantheses. $\dagger$C content in SEM and TEM-EDS determined by lever rule based on phase fractions obtained from EBSD. $\ddagger$ Grain sizes calculated from the Bruker ESPRIT software. | (wt%) | dgrain‡ | SFE | Md30
---|---|---|---|---
| Mn | Al | Si | C† | V | Sn | (µm) | (mJ m-2) | ( °C)
Bulk (ICP) | 8.29 | 3.78 | 1.8 | 0.467 | 0.06 | 0.047 | - | - | -
$\gamma$ (Thermo-Calc) | 9.38 | 3.69 | 1.98 | 0.7 | trace | - | - | 36.6 | -
$\gamma$ (SEM-EDS) | 8.8 (0.7) | 4.0 (0.6) | 1.9 (0.4) | 0.69 | - | - | 4.2 | 38.2 | 165
$\gamma$ (TEM-EDS) | 9.5 (0.4) | 3.2 (0.1) | 1.8 (0.1) | 0.69 | 0.07 (0.03) | 0.03 (0.06) | 4.2 | 32.9 | 161
$\alpha$ (Thermo-Calc) | 4.76 | 4.65 | 2.09 | trace | trace | - | - | - | -
$\alpha$ (SEM-EDS) | 7.3 (1.2) | 4.2 (0.9) | 1.9 (0.3) | 0 | - | - | 1.8 | - | -
$\alpha$ (TEM-EDS) | 6.5 (0.2) | 4.0 (0.1) | 2.3 (0.1) | 0 | 0.04 (0.02) | 0.04 (0.08) | 1.8 | - | -

Twin intersection density in the microstructure is known to affect the
subsequent TRIP effect by providing potent nucleation sites for SIM nucleation
[2, 23, 38]. Two types of SIM were observed in the investigated steel: fine-
SIM and blocky-SIM. Both fine-SIM and blocky-SIM were observed to have
nucleated on defects as per the definition by Olson and Cohen [42] but were
distinguished on the basis of morphology and type of nucleating defect. Fine-
SIM was only observed in the CR20 sample where it nucleates at twin
intersections and in curved twins (Figures 9 and 10). Fine-SIM also tends to
remain within the twin boundaries even as they grow [51].

On the other hand, blocky-SIM was observed to nucleate at grain boundaries,
especially interphase grain boundaries (Figures 8 and 11) and can grow to very
large sizes, even consuming the entire austenite grain (Figure 6d-f). Blocky-
SIM was observed in all three samples CR10, CR20 and TS26 suggesting that the
formation of blocky-SIM was unavoidable, regardless of strain path or strain
level. However, the existence of blocky-SIM at low strains, i.e. CR10 (Figure
8), suggests that grain boundaries, particularly interphase boundaries, can
also act as potent nucleation sites for martensite nucleation, allowing
martensite to form without the need for twin intersections, contrary to what
was suggested by Lee and De Cooman [23]. This supports the observation by Sohn
et. al. [4] that TWIP and TRIP (in the form of blocky-SIM) can occur
simultaneously. During the plastic deformation of medium Mn steels, the
austenite-ferrite interphase boundary experiences a large amount of strain due
to the difference in operating slip systems between the two phases [56, 36].
Therefore, the large strain at the interphase grain boundaries could provide a
large driving force for the nucleation of blocky-SIM.

Since twinning and martensite transformation have been shown to occur
simultaneously, then TWIP and TRIP can be said to be competitive processes in
the investigated alloy. At low strains (e.g. CR10) where the plastic mismatch
between ferrite and austenite is low [56], only a small fraction of austenite
has transformed in the form of blocky-SIM, allowing twins to form in the
untransformed austenite. However, at higher strains (e.g. CR20 and TS26), a
larger fraction of austenite would have transformed to blocky-SIM, with less
untransformed austenite available for twinning. Cold rolling therefore
promotes twin formation in the austenite before blocky-SIM begins to dominate
the microstructure at higher strains, making twinning more difficult.

A summary of the microstructural evolution of the austenite phase is shown in
Figure 12. At 10% cold reduction, the austenite phase first shows twins and
blocky-SIM. Secondary twins may also be present in certain grains. When cold
reduction was increased to 20%, twinning becomes more extensive and numerous
twin intersections have developed. Fine-SIM has started to nucleate in curved
twins and at twin intersections while blocky-SIM would also have grown larger
and more numerous. Alternatively, at 0.26 true tensile strain, the
microstructure contains of a low density of austenite twins, a large fraction
of blocky-SIM and also SAM.

Figure 12: Schematic of the deformation structures in the austenite phase as
seen in the various samples. CR10 showing austenite twins, as well as and
blocky-SIM. CR20 showing kinked twins with fine-SIM within, intersecting twins
and larger-SIM. TS26 showing only one active twinning system, extensive large-
SIM and also twinned martensite (SAM).

###  4.2 Effect of cold rolling on tensile behaviour

The effect of cold rolling as explained in the previous section was to
introduce a variety of deformation structures into the microstructure. When
the cold rolled samples were subsequently tensile tested, these deformation
structures will certainly be expected to influence the tensile behaviour. The
effect of deformation twins is well known and is considered to be beneficial
as it strengthens the austenite through the dynamic Hall-Petch effect [55] and
also provides nucleation sites for fine-SIM [2]. Fine SIM, especially twin
intersection-type SIM, is also considered beneficial and its effect on tensile
properties is well documented and modelled in both medium Mn steels and MASS
[23, 38, 57, 58]. Steels which can form a higher density of twin intersections
and therefore form a larger volume fraction of fine-SIM were able to exhibit
higher strain hardening rates. However, the effects of blocky-SIM are not as
well studied but it is postulated that because of its coarse nature, would not
provide as large a benefit to strain hardening or ductility enhancement
compared to fine-SIM. SAM which shares a similar blocky and coarse morphology
as blocky-SIM is similarly not expected to aid in ductility enhancement [48].

From Figures 4 and 11, the as-annealed sample likely deformed via simultaneous
TWIP and TRIP (blocky-SIM) while also forming SAM at later stages. The lack of
fine-SIM and predominance of blocky-SIM in the as-annealed 0.06V steel may
explain the lower strain hardening rate as compared to other medium Mn steels
in the literature [2, 23, 4]. On the other hand, the cold rolled samples
deformed predominantly via TRIP only. The YPE in the cold rolled tensile
samples strongly suggests that twinning was no longer active during the early
stage of deformation. It was likely that the uniaxial tensile strain was not
able to activate new twinning systems nor increase twin density than what was
already formed by cold rolling.

The presence of a high density of potent fine-SIM nucleation sites in the cold
rolled samples indicates that during uniaxial tension, the TRIP effect would
comprise of the nucleation of both fine-SIM and blocky-SIM. The additional
formation of fine-SIM would greatly improve the strain hardening rate and
tensile strength which was observed in Figure 4. Since the CR20 sample
contained a higher twin density and therefore nucleation sites for fine-SIM,
the strain hardening rate and tensile strength in the CR20 tensile sample was
consequently higher than CR10. The higher martensite fractions in the cold
rolled and tensile tested samples shown in Figure 7 may also be attributed to
the nucleation and growth of fine-SIM that would otherwise not have formed
under pure uniaxial tension. The phenomena of increased martensite fraction in
cold rolled then tensile tested samples was also observed in MASS [59].

It is also noteworthy that the CR10 and CR20 tensile samples both failed at
the same elongation. While the increased final martensite fraction in the cold
rolled and tensile tested samples may have contributed to ductility
enhancement, Sun et. al. [24] found that in a TWIP$+$TRIP-type medium Mn
steel, failure was driven by void formation at $\alpha/\alpha^{\prime}$
interfaces, most likely due to the large plasticity mismatch between the two
phases. A similar situation may also be occuring in the CR10 and CR20 tensile
samples where the interface strength was the limiting factor, since neither
sample failed by necking.

These findings suggest that the cold rolled samples were no longer
TWIP$+$TRIP-type but deformed only via the TRIP mechanism. It can be viewed
that by cold rolling the steel, the TWIP effect was consumed to produce an
enhanced TRIP effect during subsequent uniaxial tensile testing. This implies
that the strain hardening rate of medium Mn steels can be varied by simply
adjusting the cold rolling reduction after the IA heat treatment. Future alloy
design for cold rolled medium Mn steels should aim to reduce the SFE of the
austenite to promote twinning during cold rolling and therefore increase the
density of nucleation sites for fine-SIM.

###  4.3 Industrial relevance

In our previous work [35], we introduced a medium Mn steel of composition
Fe-12Mn-4.8Al-2Si-0.32C-0.3V and found that the high price of V greatly
increased the alloying cost of the steel. Many researchers have also added V
to medium Mn steels and achieved large amounts of precipitation strengthening
[60, 14]. Hu et. al. [60] was able to improve the yield strength by 650 MPa
with 0.7 wt% V through a warm rolling and IA processing route. However,
relying on V for strength is a costly route. In this study, a carbide
optimisation study was conducted and found that 0.06 wt% V was able to
increase the yield strength by 55 MPa and sufficient to raise the yield
strength of the as-annealed steel to 1 GPa.

Figure 13 shows an updated figure from our previous work [35] which includes
the investigated 0.06V steel (Novalloy), several new literature medium Mn
steels and also two grades of Quenching and Partitioning (QP) steels from
Baosteel. The alloying cost of novel and commercial steels were estimated
based on pure metal and ferroalloy additions to steel scrap. By reducing the
Mn and V content from our previous alloy, the alloying cost was significantly
reduced by 160 USD/tonne (prices correct as of May 2019) while maintaining a
similar U30. Novalloy in its annealed condition is price competitive with
commercially available DP steels yet being two to three times more energy
absorbing. When in the cold rolled condition (i.e. CR20), the U30 was
significantly increased, presenting new opportunities for down gauging in
automotive steel. This shows that a holistic alloy design concept can result
in a price-competitive high strength and high energy absorbing steel.

Figure 13: U30 diagram comparing alloying cost and energy absorption between
several medium Mn steels in the literature and commerical steels. [61, 62, 12,
5, 3, 38, 2, 63, 7, 13, 64, 65, 35, 60, 66, 67]. *ThyssenKrupp dual phase
steels.

##  5 Conclusions

The effects of cold rolling after IA were investigated in a novel steel named
Novalloy and it was found that the tensile properties, especially the strain
hardening rate, could be improved with increasing cold reduction. The
microstructural evolution with increasing cold rolling reduction was
investigated to understand the mechanisms behind the improvement in tensile
properties. The key findings from this study are summarised as follows:

  1. 1.

The microstructural evolution of medium Mn steels is strain path dependent.
Cold rolled specimens showed higher twin densities and active secondary twin
systems compared to limited twinning under uniaxial tension without prior cold
rolling.

  2. 2.

The defects introduced by cold rolling promote the formation of fine-SIM which
would lead to an increased strain hardening rate during uniaxial tension, i.e.
an enhanced TRIP effect.

  3. 3.

TWIP and TRIP effects were likely occuring simultaneously during the
deformation of the investigated steel, regardless of strain path.

  4. 4.

By keeping the V content low, the investigated steel is price competitive with
many commercial steels but with a significantly greater energy absorption
capability.

##  6 Acknowledgements

TWJK would like to acknowledge the provision of a studentship by A*STAR,
Singapore. DD, WMR and XX would like to acknowledge funding support from the
UK EPSRC (Designing alloys for resource efficiency, EP/L025213/1).

##  7 References

## References

  * [1] Lee S, De Cooman BC.  On the selection of the optimal intercritical annealing temperature for medium Mn TRIP steel.  Metallurgical and Materials Transactions A: Physical Metallurgy and Materials Science 2013;44:5018
  * [2] Lee S, De Cooman BC.  Annealing Temperature Dependence of the Tensile Behavior of 10 pct Mn Multi-phase TWIP-TRIP Steel.  Metallurgical and Materials Transactions A: Physical Metallurgy and Materials Science 2014;45:6039
  * [3] Hu B, Luo H.  A strong and ductile 7Mn steel manufactured by warm rolling and exhibiting both transformation and twinning induced plasticity.  Journal of Alloys and Compounds 2017;725:684
  * [4] Sohn SS, Choi K, Kwak JH, Kim NJ, Lee S.  Novel ferrite-austenite duplex lightweight steel with 77% ductility by transformation induced plasticity and twinning induced plasticity mechanisms.  Acta Materialia 2014;78:181
  * [5] Wang JJ, Hui WJ, Xie ZQ, Zhang YJ, Zhao XL.  Influence of pre-strain on microstructural characteristics and tensile deformation behaviour of a cold-rolled Al-containing medium Mn steel.  Journal of Materials Science 2020;55:5296
  * [6] Lee S, Lee SJ, Santhosh Kumar S, Lee K, De Cooman BC.  Localized deformation in multiphase, ultra-fine-grained 6 Pct Mn transformation-induced plasticity steel.  Metallurgical and Materials Transactions A: Physical Metallurgy and Materials Science 2011;42:3638
  * [7] Savic V, et al.  Development of a Lightweight Third-Generation Advanced High-Strength Steel (3GAHSS) Vehicle Body Structure.  SAE International Journal of Materials and Manufacturing 2018;11:2018
  * [8] Sun B, et al.  Discontinuous strain-induced martensite transformation related to the Portevin-Le Chatelier effect in a medium manganese steel.  Scripta Materialia 2017;133:9
  * [9] Kim MT, Park TM, Baik KH, Choi WS, Han J.  Effects of cold rolling reduction ratio on microstructures and tensile properties of intercritically annealed medium-Mn steels.  Materials Science and Engineering A 2019;752:43
  * [10] Sun B, Fazeli F, Scott C, Brodusch N, Gauvin R, Yue S.  The influence of silicon additions on the deformation behavior of austenite-ferrite duplex medium manganese steels.  Acta Materialia 2018;148:249
  * [11] Buchely MF, Field DM, Van Aken DC.  Analysis of Hot- and Cold-Rolled Loads in Medium-Mn TRIP Steels.  Metallurgical and Materials Transactions B: Process Metallurgy and Materials Processing Science 2019;50:1180
  * [12] Wang H, et al.  A medium-Mn steel processed by novel twin-roll strip casting route.  Materials Science and Technology 2019;35:1227
  * [13] Lee YK, Han J.  Current opinion in medium manganese steel.  Materials Science and Technology 2015;31:843
  * [14] Lee S, De Cooman BC.  Tensile behavior of intercritically annealed ultra-fine grained 8% Mn multi-phase steel.  Steel Research International 2015;86:1170
  * [15] Sun B, et al.  Critical role of strain partitioning and deformation twinning on cracking phenomenon occurring during cold rolling of two duplex medium manganese steels.  Scripta Materialia 2017;130:49
  * [16] Li ZC, Ding H, Misra RD, Cai ZH.  Deformation behavior in cold-rolled medium-manganese TRIP steel and effect of pre-strain on the Lüders bands.  Materials Science and Engineering A 2017;679:230
  * [17] Zhang M, et al.  Temperature-dependent micromechanical behavior of medium-Mn transformation-induced-plasticity steel studied by in situ synchrotron X-ray diffraction.  Acta Materialia 2017;141:294
  * [18] Lee S, Lee K, De Cooman BC.  Observation of the TWIP+TRIP Plasticity-Enhancement Mechanism in Al-Added 6 Wt Pct Medium Mn Steel.  Metallurgical and Materials Transactions A: Physical Metallurgy and Materials Science 2015;46:2356
  * [19] Field DM, Qing J, van Aken DC.  Chemistry and Properties of Medium-Mn Two-Stage TRIP Steels.  Metallurgical and Materials Transactions A: Physical Metallurgy and Materials Science 2018;49:1
  * [20] Luo H, Shi J, Wang C, Cao W, Sun X, Dong H.  Experimental and numerical analysis on formation of stable austenite during the intercritical annealing of 5Mn steel.  Acta Materialia 2011;59:4002
  * [21] Dumay A, Chateau JP, Allain S, Migot S, Bouaziz O.  Influence of addition elements on the stacking-fault energy and mechanical properties of an austenitic Fe-Mn-C steel.  Materials Science and Engineering A 2008;483-484:184
  * [22] Kim JK, De Cooman BC.  Stacking fault energy and deformation mechanisms in Fe-xMn-0.6C-yAl TWIP steel.  Materials Science and Engineering A 2016;676:216
  * [23] Lee S, De Cooman BC.  Tensile Behavior of Intercritically Annealed 10 pct Mn Multi-phase Steel.  Metallurgical and Materials Transactions A 2014;45A:709
  * [24] Sun B, et al.  Revealing fracture mechanisms of medium manganese steels with and without delta-ferrite.  Acta Materialia 2019;164:683
  * [25] Sohn SS, Lee BJ, Lee S, Kim NJ, Kwak JH.  Effect of annealing temperature on microstructural modification and tensile properties in 0.35 C-3.5 Mn-5.8 Al lightweight steel.  Acta Materialia 2013;61:5050
  * [26] Vermeulen Y, Coletti B, Blanpain B, Wollants P, Vleugels J.  Material evaluation to prevent nozzle clogging during continuous casting of Al killed steels.  ISIJ International 2002;42:1234
  * [27] Song H, et al.  Improvement of tensile properties in (austenite+ferrite+$\kappa$-carbide) triplex hot-rolled lightweight steels.  Materials Science and Engineering: A 2018;730:177
  * [28] Lee S, Estrin Y, De Cooman BC.  Constitutive modeling of the mechanical properties of V-added medium manganese TRIP steel.  Metallurgical and Materials Transactions A: Physical Metallurgy and Materials Science 2013;44:3136
  * [29] Zhang X, Ma G, Liu M, Li Z.  Removal of residual element tin in the ferrous metallurgy process: A review.  Metals 2019;9:1
  * [30] Pourmajidian M, McDermid JR.  On the reactive wetting of a medium-Mn advanced high-strength steel during continuous galvanizing.  Surface and Coatings Technology 2019;357:418
  * [31] Angel T.  Formation of martensite in austenitic stainless steels.  J Iron Steel Inst 1954;:165
  * [32] Nohara K, Ono Y, Ohashi N.  Composition and Grain Size Dependencies of Strain-induced Martensitic Transformation in Metastable Austenitic Stainless Steels.  J Iron Steel Inst 1977;63:772
  * [33] Herrera C, Ponge D, Raabe D.  Design of a novel Mn-based 1 GPa duplex stainless TRIP steel with 60 % ductility by a reduction of austenite stability.  Acta Materialia 2011;59:4653
  * [34] Keeler S, Kimchi M, J Mooney P.  Advanced High-Strength Steels Apllication Guidelines Version 6.0.  Technical report, WorldAutoSteel 2017
  * [35] Kwok TWJ, et al.  Design of a High Strength, High Ductility 12 wt% Mn Medium Manganese Steel With Hierarchical Deformation Behaviour.  Materials Science & Engineering A 2020;
  * [36] Sun B, et al.  Macroscopic to nanoscopic in situ investigation on yielding mechanisms in ultrafine grained medium Mn steels: Role of the austenite-ferrite interface.  Acta Materialia 2019;178:10
  * [37] Gladman T.  Precipitation hardening in metals.  Materials Science and Technology 1999;15:30
  * [38] Lee S, Woo W, de Cooman BC.  Analysis of the Tensile Behavior of 12 pct Mn Multi-phase ($\alpha$ \+ $\gamma$) TWIP + TRIP Steel by Neutron Diffraction.  Metallurgical and Materials Transactions A: Physical Metallurgy and Materials Science 2016;47:2125
  * [39] Nakada N, Ito H, Matsuoka Y, Tsuchiyama T, Takaki S.  Deformation-induced martensitic transformation behavior in cold-rolled and cold-drawn type 316 stainless steels.  Acta Materialia 2010;58:895
  * [40] Olsen GB, Cohen M.  A mechanism for the strain-induced nucleation of martensitic transformations.  Journal of the Less-Common Metals 1972;28:107
  * [41] Olson GB, Cohen M.  A General Mechanism of Martensitic Nucleation- Part II. FCC —,-BCC and Other Martensitic Transformations.  Metallurgical Transactions A 1976;7A:1905
  * [42] Olson GB, Cohen M.  A mechanism for the strain-induced martensitic transformations* nucleation of.  Journal of the Less-Common Metals 1972;28:107
  * [43] Bogers AJ, Burgers WG.  Partial dislocations on the {110} planes in the B.C.C. lattice and the transition of the F.C.C. into the B.C.C. lattice.  Acta Metallurgica 1964;12:255
  * [44] Higo Y, Lecroisey F, Mori T.  Relation between applied stress and orientation relationship of $\alpha$’ martensite in stainless steel single crystals.  Acta Metallurgica 1974;22:313
  * [45] Tsakiris V, Edmonds DV.  Martensite and deformation twinning in austenitic steels.  Materials Science and Engineering A 1999;273-275:430
  * [46] Tian Y, Borgenstam A, Hedström P.  Comparing the deformation-induced martensitic transformation with the athermal martensitic transformation in Fe-Cr-Ni alloys.  Journal of Alloys and Compounds 2018;766:131
  * [47] Chen M, Terada D, Shibata A, Tsuji N.  Identical area observations of deformation-induced martensitic transformation in SUS304 austenitic stainless steel.  Materials Transactions 2013;54:308
  * [48] Maxwell PC, Goldberg A, Shyne JC.  Stress-Assisted and strain-induced martensites in FE-NI-C alloys.  Metallurgical Transactions 1974;5:1305
  * [49] Yen HW, et al.  Role of stress-assisted martensite in the design of strong ultrafine-grained duplex steels.  Acta Materialia 2015;82:100
  * [50] Hu B, Luo H, Yang F, Dong H.  Recent progress in medium-Mn steels made with new designing strategies, a review.  Journal of Materials Science and Technology 2017;33:1457
  * [51] Murr LE, Staudhammer KP, Hecker SS.  Effects of Strain State and Strain Rate on Deformation-Induced Transformation in 304 Stainless Steel: Part II. Microstructural Study.  Metallurgical Transactions A 1982;13:627
  * [52] Beese AM, Mohr D.  Effect of stress triaxiality and Lode angle on the kinetics of strain-induced austenite-to-martensite transformation.  Acta Materialia 2011;59:2589
  * [53] Sohrabi MJ, Naghizadeh M, Mirzadeh H.  Deformation-induced martensite in austenitic stainless steels: A review.  Archives of Civil and Mechanical Engineering 2020;20:1
  * [54] Shrinivas V, Varma SK, Murr LE.  Deformation-induced martensitic characteristics in 304 and 316 stainless steels during room-temperature rolling.  Metallurgical and Materials Transactions A 1995;26:661
  * [55] De Cooman BC, Estrin Y, Kim SK.  Twinning-induced plasticity (TWIP) steels.  Acta Materialia 2018;142:283
  * [56] Lee S, Woo W, De Cooman BC.  Analysis of the Plasticity-Enhancing Mechanisms in 12 pctMn Austeno-ferritic Steel by In Situ Neutron Diffraction.  Metallurgical and Materials Transactions A: Physical Metallurgy and Materials Science 2014;45:5823
  * [57] Lebedev AA, Kosarchuk VV.  Influence of phase transformations on the mechanical properties of austenitic stainless steels.  International journal of plasticity 2000;16:749
  * [58] Martin S, Wolf S, Martin U, Krüger L, Rafaja D.  Deformation Mechanisms in Austenitic TRIP/TWIP Steel as a Function of Temperature.  Metallurgical and Materials Transactions A: Physical Metallurgy and Materials Science 2016;47:49
  * [59] Milad M, Zreiba N, Elhalouani F, Baradai C.  The effect of cold work on structure and properties of AISI 304 stainless steel.  Journal of Materials Processing Technology 2008;203:80
  * [60] Hu B, He BB, Cheng GJ, Yen HW, Huang MX, Luo HW.  Super-high-strength and formable medium Mn steel manufactured by warm rolling process.  Acta Materialia 2019;174:131
  * [61] Luo H, Dong H.  New ultrahigh-strength Mn-alloyed TRIP steels with improved formability manufactured by intercritical annealing.  Materials Science and Engineering A 2015;626:207
  * [62] Shao C, Hui W, Zhang Y, Zhao X, Weng Y.  Microstructure and mechanical properties of hot-rolled medium-Mn steel containing 3% aluminum 2017
  * [63] De Cooman BC, Lee SJ, Shin S, Seo EJ, Speer JG.  Combined Intercritical Annealing and Q&P Processing of Medium Mn Steel.  Metallurgical and Materials Transactions A: Physical Metallurgy and Materials Science 2017;48:39
  * [64] Lee D, Kim JK, Lee S, Lee K, De Cooman BC.  Microstructures and mechanical properties of Ti and Mo micro-alloyed medium Mn steel.  Materials Science and Engineering A 2017;706:1
  * [65] Zhu Y, Hu B, Luo H.  Influence of Nb and V on Microstructure and Mechanical Properties of Hot-Rolled Medium Mn Steels.  Steel Research International 2017;1700389:1
  * [66] He BB, Huang MX.  Strong and ductile medium Mn steel without transformation-induced plasticity effect.  Materials Research Letters 2018;6:365
  * [67] Sohn SS, Song H, Jo MC, Song T, Kim HS, Lee S.  Novel 1.5 GPa-strength with 50%-ductility by transformation-induced plasticity of non-recrystallized austenite in duplex steels.  Scientific Reports 2017;7:1


11institutetext:  Paris-Saclay University, France
11email<EMAIL_ADDRESS>22institutetext: Lincoln Agri-Robotics,
University of Lincoln, United Kingdom
22email<EMAIL_ADDRESS>33institutetext: Department of Computer Science,
University of York, United Kingdom
33email<EMAIL_ADDRESS>
# Learning cooperative behaviours in adversarial multi-agent systems††thanks:
This work benefited from the advice of Assistant Professor Shinkyu Park at
King Abdullah University of Science and Technology (KAUST).

Ni Wang 11 0000-0002-5867-0176 Gautham P. Das 22 0000-0001-5351-9533 Alan G.
Millard 33 0000-0002-4424-5953

###### Abstract

This work extends an existing virtual multi-agent platform called RoboSumo to
create TripleSumo — a platform for investigating multi-agent cooperative
behaviors in continuous action spaces, with physical contact in an adversarial
environment. In this paper we investigate a scenario in which two agents,
namely ‘Bug’ and ‘Ant’, must team up and push another agent ‘Spider’ out of
the arena. To tackle this goal, the newly added agent ‘Bug’ is trained during
an ongoing match between ‘Ant’ and ‘Spider’. ‘Bug’ must develop awareness of
the other agents’ actions, infer the strategy of both sides, and eventually
learn an action policy to cooperate. The reinforcement learning algorithm Deep
Deterministic Policy Gradient (DDPG) is implemented with a hybrid reward
structure combining dense and sparse rewards. The cooperative behavior is
quantitatively evaluated by the mean probability of winning the match and mean
number of steps needed to win.

###### Keywords:

multi-agent cooperation reinforcement learning.

##  1 Introduction

With the success of AlphaGo, AI in games has been gaining increasing attention
from researchers around the world [1]. In the physical world, agents exhibit
intelligent behaviours at multiple spatial and temporal scales [2]. In [3], a
multi-agent platform RoboSumo111 https://github.com/openai/robosumo was
designed to investigate the potential of continuous adaptation in non-
stationary and competitive multi-agent environments through meta-learning. In
that platform, two agents, either homogeneous or heterogeneous, learn to play
a ‘sumo’ game against each other. The one that successfully pushes its
opponent off the square arena wins the match.

While RoboSumo allows multi-agent physical interaction to be investigated in
adversarial scenarios, the platform does not offer support for exploring
cooperative behaviors among agents. In this paper, we extend RoboSumo such
that a new agent is added and can team up with one of the existing agents.
This new agent must learn a policy to cooperate with its pre-defined partner,
and play against their opponent. We train the system with Deep Deterministic
Policy Gradient (DDPG), a reinforcement learning algorithm which learns a
Q-function with off-policy data and the Bellman equation, and concurrently
learns a policy using the Q-function [4]. The training result is evaluated
through both qualitative observations of the agents’ behaviors in simulation,
and two quantitative parameters – ‘mean winning rate’ and ‘steps needed to
win’. The code developed for training, testing, and evaluation is open for
public access222 https://github.com/niart/triplesumo . The major contributions
of this work are: to establish a virtual platform that allows both cooperative
and competitive interactions to be explored in physical contact-rich
scenarios, and to report baseline results for the two evaluation metrics after
training the system with DDPG.

The next section of this paper reviews related work on multi-agent games based
on virtual platforms; Section 3 describes our extension of RoboSumo and
establishes TripleSumo; Section 4 details our methodology for training the
agent with the DDPG algorithm, followed by an evaluation of training results.
The final section summarises our findings and outlines plans for future work.

##  2 Related Work

Games provide challenging environments to quickly test new algorithms and
ideas of reinforcement learning (RL) in a safe and reproducible manner [5].
Therefore, recent years have witnessed the development of a series of novel
virtual game platforms that have fuelled reinforcement learning research.
While some research focuses on a single agent tackling a task, many cases are
based on adversarial models, where agents compete against each other to
improve the overall outcome of a system.

In 2020, ‘Google Research Football’[5] was designed as an open-source platform
where agents are trained to play football in simulated physics-based 3D scene.
Three scenarios were provided with varying levels of difficulties. Three RL
algorithms, namely Importance Weighted Actor-Learner Architecture (IMPALA),
Proximal Policy Optimization Algorithms (PPO), and Ape-X DQN, were implemented
by the authors to report baselines. This popular platform was demonstrated to
be useful in developing AI methods, for example, ‘TiKick’ [6]. However,
‘Google Research Football’ assumes the actions are synchronously executed in
multi-agent settings, limiting its utility. In response to this, ‘Fever
Basketball’ [7] was developed — an asynchronous sports game environment for
multi-agent reinforcement learning.

Similarly based on virtual football games, [2] studied integrated decision-
making at multiple scales in a physically embodied multi-agent system by
developing a method that combines imitation learning, single- and multi-agent
reinforcement learning, and population-based training, making use of
transferable representations of behaviour. This research evaluated agent
behaviours using several analysis techniques, including statistics from real-
world sports analytics.

[8] introduced a ‘hide-and-seek’ game to investigate agents learning tool use.
Through training in this environment, agents build a series of six distinct
strategies and counter strategies. This work suggested a promising future of
multi-agent co-adaptation, which could produce complex and intelligent
behaviors.

When it comes to RL-based methods for multi-agent cooperation, [9] tackled the
limitation of Q-learning in a non-stationary environment, resulting in
variance of the policy gradient as the number of agents grows. They presented
an adaptation of actor-critic methods that consider action policies of other
agents, and a training regimen utilising an ensemble of policies for each
agent that leads to more robust multi-agent policies.

CollaQ [10] decomposes the Q-function of each agent into a ‘self term’ and an
‘interactive term’, with a Multi-Agent Reward Attribution (MARA) loss that
regularises the training. This method was validated in the ‘StarCraft multi-
agent challenge’ and was demonstrated to outperform existing state-of-the-art
techniques.

In a multi-agent pursuit-evasion problem, [11] used shared experience to train
a policy with curriculum learning for a given number of pursuers, to be
executed independently by each agent at run-time. They designed a reward
structure combining individual and group rewards to encourage good formation.

Similarly in the pursuit-evasion problem, [12] presented a new geometric
approach of learning cooperative behaviours in a 2-pursuer single evader
scenario to reduce the capture time of the evader. This method was shown to be
scalable thanks to categorisation and removal of redundant pursuers.

These virtual games offer useful experimental platforms and learning methods
for multi-agent teamwork in adversarial environments. However, physical
contact between interactive agents in continuous domain is rarely
investigated. Our work meets this need by developing a platform that
facilitates research into multi-agent cooperation in physical contact-rich
environments. This paper introduces TripleSumo, which extends the RoboSumo
platform, and presents reinforcement-learning of cooperative behaviours in an
adversarial multi-agent setting.

##  3 TripleSumo

Based on the RoboSumo framework built with the software toolkit OpenAI/Gym
[13] and the MuJoCo[14] physics engine, TripleSumo adds one more agent to the
system (see Figure 1).

Figure 1: Competitive relation between agent ‘Spider’ (green, right) and the
pair of ‘Bug’ (blue, middle) and ‘Ant’ (red, left) Figure 2: The simulated
virtual platform ‘TripleSumo’, where two agents (red and blue) play against
the green agent on a Tatami.

In this scenario, ‘Ant’ (red) and ‘Bug’ (blue) team up and play against their
opponent ‘Spider’ (green) on a square arena (‘Tatami’) (see Figure 2). Agent
behaviors are trained and observed in designated continuous action spaces. To
simplify the interfaces, this preliminary work sets up all agents to be four-
legged and the same size. However, the three agents differ from one another in
contact force (see Table 1). Morphological and physical features of the agents
are subject to free choices according to future research demands. Once the
game starts, the three agents interact through physical contact and the match
lasts until the centre of mass of any of the three agents falls outside the
edge of the arena. The two agents ‘Ant’ and ‘Spider’ have been pre-trained
through DDPG to create an ongoing game (see supplementary video 1). ‘Spider’
will win the game if it manages to push either ‘Ant’ or ‘Bug’ off the arena.
Alternatively, the team of ‘Ant’ and ‘Bug’ will win the game if they manage to
push off ‘Spider’.

| Control range of contact force (in Newtons)
---|---
‘Spider’ (green) | [-0.22, 0.22]
‘Ant’ (red) | [-0.20, 0.20]
‘Bug’ (blue) | [-0.18, 0.18]
Table 1: Control range of contact force of heterogeneous agents

##  4 Experimental Results

This section implements a commonly used reinforcement learning algorithm DDPG
to train the new agent. In order to infer the strategies of both its opponent
and partner, the agent ‘Bug’ is trained to learn an action policy during an
ongoing match333 supplementary video 1:
https://www.youtube.com/watch?v=nxzi7Pha2GU , where ‘Ant’ and ‘Spider’ are
playing against each other. The ongoing game is created by training ‘Ant’ for
3,000 epochs with DDPG first, and afterwards training ‘Bug’ for 20,000 epochs
(Figure 3), with reward structures similar to equations (1), (2), but in
different directions.

Figure 3: Mean total rewards received by ‘Ant’ over 3,000 epochs of training
(left); Mean total rewards received by ‘Spider’ over 20,000 epochs of training
(right); a rolling mean filter of 200 applied to each graph.

###  4.1 Reward Shaping

The reward function used in this experiment consists of two parts:

$Reward\ =\ Dense\ Reward\ +\ Sparse\ Reward$ (1)

In the ‘Dense’ part, each single step of the agent’s movement is associated
with a value which adds up to the total reward of the current epoch. This
‘Dense Reward’ is decomposed into four terms — ‘opponent velocity reward’,
‘partner velocity reward’, ‘self velocity reward’, and a constant punishment
(Equation 2), where $C_{1}$, $C_{2}$ and $C_{3}$ are constant coefficients
inserted to each term of dense reward function.

$\begin{split}Dense\ reward=\ &C_{1}\ *\ opponent\ velocity\ reward\ \\\ +\
&C_{2}\ *\ self\ velocity\ reward\ \\\ +\ &C_{3}\ *\ partner\ velocity\
reward\ \\\ +\ &still\ punishment\end{split}$ (2)

In each step, the agent receives an ‘opponent velocity reward’ which is
decomposed to an X component and a Y component. Its X component has an
absolute value proportional to the opponent’s speed along the X-direction, and
is assigned to be positive if the opponent is moving backwards from the agent,
and negative if the opponent moves towards the agent. The Y component of
‘opponent velocity reward’ is defined in a similar way in the Y direction.
‘Self velocity reward’ and ‘partner velocity reward’ have absolute values
proportional to the agent’s and its partner’s speeds respectively, assigned to
be positive if the agent or its partner moves towards its opponent, while
assigned to be negative if they move away from their opponent. That is, the
agent will be rewarded if itself or its partner moves towards their opponent,
or if the opponent moves backwards from the agent, and will be punished on the
contrary. The ‘still punishment’ is a negative constant to ensure the agent
will be punished if it remains stationary. Figure 4 shows the direction of
allocating positive dense reward to the agent.

Figure 4: Direction of allocating positive dense reward: A. opponent velocity
reward; B. partner velocity reward; C. self velocity reward.

On the other hand, the ‘Sparse’ part of the reward function is associated with
the result of game. A score of +500 or -400 will be allocated if the team wins
or loses the game respectively (see Algorithm 1). The sparse reward will be 0
by default if the epoch exceeds the maximum number of steps without a side
winning.

Algorithm 1 Sparse part of reward function

while An epoch is ongoing do

if The opponent falls off Tatami then

reward += 500

else if ‘Bug’ or ‘Ant’ falls off Tatami then

reward -= 400

end if

end while

Hyper-parameter |  Value
---|---
Optimiser |  Adam
Actor Learning rate |  1e-4
Critic Learning rate |  1e-3
Discount factor ($\gamma$) |  0.99
Target Update Factor ($\tau$) |  0.01
Steps per epoch |  500
Batch size |  64
Maximum epochs |  20,000
Replay buffer size |  1,000,000
Update stride |  30
Table 2: Hyper-parameters in DDPG implementations

Important hyper-parameters are summarised in Table 2. Over the course of
training for 20,000 epochs, the agent demonstrates increasing mean values of
both dense and sparse rewards, as shown in Figure 6.

Figure 5: Mean of Total rewards (orange), Dense reward (blue), and Sparse
reward (green) received by ‘Bug’ in each epoch during training. A rolling mean
filter of 200 is applied. The final dense and total rewards stay below zero
due to a constant ‘still punishment’ term in reward function.

Figure 6: Mean of Total rewards (orange), Dense reward (blue), and Sparse
reward (green) received by ‘Bug’ in a test of 20,000 epochs. A rolling mean
filter of 200 is applied.

Figure 7:  MWR of the team increases as training progresses (blue). The MWR
rising from negative to positive indicates the team ends up with stronger
competency than its opponent. MWR of the team remains positive (between 0.15
and 0.47) in testing of 20,000 epochs (orange). A rolling mean filter of 200
was applied to each graph.

Figure 8: Mean value of number of steps needed to win the game decreases over
the course of training. A rolling mean filter of 200 is applied. Only games
won by the team are counted.

###  4.2 Performance Evaluation

The action policy of ‘Bug’ resulting from this training is observed in a test
across 20,000 epochs444 supplementary video 2:
https://www.youtube.com/watch?v=C4xGuIyeY5A , where the agents demonstrate
cooperative behavior and stable values of both dense and sparse rewards (see
Figure 6). Additionally, in order to evaluate the efficiency of agents’
cooperation, we used the two metrics: MWR and ‘steps needed to win’ to analyse
the training result.

Result of an epoch |  Local winning rate
---|---
Team wins |  +1
Team loses |  -1
Time expires without result |  0
Table 3: Definition of local winning rate at a certain epoch

A metric ‘winning rate’ (WR) is defined as the probability of winning the game
at a certain phase during training or testing. During training, the local WR
at a certain epoch is assigned to be ‘+1’ if the team wins the game; assigned
to be ‘-1’ if the team loses the game; and assigned to be ‘0’ if the epoch
reaches maximum number of steps without winning or losing. The definition of
local WR is described in Table 3. A ‘mean WR’ is calculated as the rolling
mean value of local winning rate when the training or testing reaches a
certain epoch. Intuitively, a positive WR indicates the team is stronger than
opponent, while a negative WR indicates the team is weaker than its opponent,
and a zero WR indicates the two sides have similar levels of competency. With
a rolling mean filter of $\lambda$, the mean WR (denoted as MWR) at epoch
$\alpha$ (denoted as $E_{\alpha}$) is:

$MWR_{\alpha}\ =\frac{sum\ of\ all\ WRs\ in\ recent\ \lambda\ epochs\ until\
E_{\alpha}}{\lambda}$ (3)

The variations of MWR during training and testing are plotted in Figure 8,
which indicate that the team’s competency improves as the training progresses,
and remains stable in the resulting policy. In other words, the agent ‘Bug’
has learned to cooperate with ‘Ant’ and contribute to the teamwork.

Another metric ‘number of steps needed to win a game’ is defined to reflect
the cost of time for a successful teamwork. Its rolling mean (computed in a
way similar to Equation 3) reflects the variation of teamwork efficiency
during the training process. In Figure 8, the plot indicates that the team
reaches higher levels of efficiency in teamwork as they are trained, resulting
in reducing the amount of steps or time needed to win the game.

##  5 Discussion & Future work

This preliminary work extended an existing virtual multi-agent platform
RoboSumo into TripleSumo, which contains three players in a sumo game. Two
agents are predefined to team up and play against the other agent. As a
baseline, cooperative behaviors were investigated by training the newly added
agent with the reinforcement learning algorithm DDPG, using a hybrid reward
structure during an ongoing match. Both the sparse and dense parts of mean
rewards are demonstrated to increase and eventually converge as the training
process progresses. The teamwork increases in competency and efficiency, as
reflected by an increasing mean winning rate and a decreasing mean number of
steps needed to win a game, which indicates successful cooperation between the
agents in this adversarial environment. While this work has focussed
exclusively on DDPG, our future research will investigate the peformance of
other RL algorithms with respect to learning cooperative strategies in multi-
agent systems.

The scenario presented in this paper is similar to learning cooperative
behaviours in a 2-pursuer single evader scenario to reduce the capture time of
the evader, with direct applications in swarm robotic systems. A natural
extension is to increase the complexity of the environment, to include more
pursuing agents making it a M-Pursuer single-evader scenario and incorporating
probabilistic observations about the prey positions. It would then be possible
to evolve different categorical behaviours such as ‘interceptor’, ‘escort’,
and ‘redundant’ for the pursuer agents.

The next step of this research is to investigate a more complex scenario of
non-predefined pairing for cooperation in TripleSumo. With three or more
agents playing against one another, the game will continue until only one
agent remains on the arena. All agents will learn to freely choose to team up
with a non-predefined partner in order to remain as long as possible in the
game.

## References

  * [1] Q. Yin, J. Yang, W. Ni, B. Liang, and K. Huang, “AI in Games: Techniques, Challenges and Opportunities,” _ArXiv_ , vol. abs/2111.07631, 2021.
  * [2] S. Liu, G. Lever, Z. Wang, J. Merel, S. M. A. Eslami, D. Hennes, W. M. Czarnecki, Y. Tassa, S. Omidshafiei, A. Abdolmaleki, N. Siegel, L. Hasenclever, L. Marris, S. Tunyasuvunakool, H. F. Song, M. Wulfmeier, P. Muller, T. Haarnoja, B. D. Tracey, K. Tuyls, T. Graepel, and N. M. O. Heess, “From motor control to team play in simulated humanoid football,” _ArXiv_ , vol. abs/2105.12196, 2021.
  * [3] M. Al-Shedivat, T. Bansal, Y. Burda, I. Sutskever, I. Mordatch, and P. Abbeel, “Continuous adaptation via meta-learning in nonstationary and competitive environments,” _ArXiv_ , vol. abs/1710.03641, 2018.
  * [4] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. M. O. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” _CoRR_ , vol. abs/1509.02971, 2016.
  * [5] K. Kurach, A. Raichuk, P. Stanczyk, M. Zajac, O. Bachem, L. Espeholt, C. Riquelme, D. Vincent, M. Michalski, O. Bousquet, and S. Gelly, “Google research football: A novel reinforcement learning environment,” in _AAAI_ , 2020.
  * [6] S. Huang, W. Chen, L. Zhang, Z. Li, F. Zhu, D. Ye, T. Chen, and J. Zhu, “TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations,” _ArXiv_ , vol. abs/2110.04507, 2021.
  * [7] H. Jia, Y. Hu, Y. Chen, C. Ren, T. Lv, C. Fan, and C. Zhang, “Fever basketball: A complex, flexible, and asynchronized sports game environment for multi-agent reinforcement learning,” _ArXiv_ , vol. abs/2012.03204, 2020\.
  * [8] B. Baker, I. Kanitscheider, T. Markov, Y. Wu, G. Powell, B. McGrew, and I. Mordatch, “Emergent tool use from multi-agent autocurricula,” _ArXiv_ , vol. abs/1909.07528, 2020.
  * [9] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” _ArXiv_ , vol. abs/1706.02275, 2017.
  * [10] T. Zhang, H. Xu, X. Wang, Y. Wu, K. Keutzer, J. Gonzalez, and Y. Tian, “Multi-agent collaboration via reward attribution decomposition,” _ArXiv_ , vol. abs/2010.08531, 2020.
  * [11] C. de Souza, R. Newbury, A. Cosgun, P. Castillo, B. Vidolov, and D. Kulić, “Decentralized multi-agent pursuit using deep reinforcement learning,” _IEEE Robotics and Automation Letters_ , vol. 6, pp. 4552–4559, 2021.
  * [12] A. Von Moll, D. W. Casbeer, E. Garcia, and D. Milutinović, “Pursuit-evasion of an evader by multiple pursuers,” in _2018 International Conference on Unmanned Aircraft Systems (ICUAS)_ , 2018, pp. 133–142.
  * [13] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” _ArXiv_ , vol. abs/1606.01540, 2016.
  * [14] E. Todorov, T. Erez, and Y. Tassa, “MuJoCo: A physics engine for model-based control,” in _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2012, pp. 5026–5033.