In general, we note that photometric redshift algorithms are by nature more
efficient than colours alone for accurate selection of galaxy samples. As an
alternative to the proposed colour selection criteria, one could make efforts
to fine-tune LePhare (or any other SED fitting tool) to assign more weight to
the $Y_{\scriptscriptstyle\rm E}$ band or the $(Y_{\scriptscriptstyle\rm
E}-J_{\scriptscriptstyle\rm E})$ colour, thereby optimising the algorithm to
further lift the degeneracy between intermediate- and high-redshift galaxies.
More work is warranted to investigate how to optimise SED fitting codes to
improve the discrimination between true $z>6$ and intermediate-$z$
interlopers.

In addition, machine learning methods as alternatives to traditional SED
fitting are becoming increasingly popular, especially with the advent of large
sky surveys such as the Euclid Survey. Recent works using mock Euclid
photometry have demonstrated how machine learning approaches typically
outperform template-fitting algorithms to retrieve the photometric redshift
and galaxy classification at $z<1$ (Euclid Collaboration: Desprez et al. 2020;
Humphrey et al. 2021 in prep). At higher redshifts, machine learning methods
become increasingly unreliable because extensive spectroscopic training
samples are lacking. However, Euclid will obtain spectroscopy and high-quality
photometry in several reference fields that already contain deep ancillary
data. Therefore, machine learning methods for selecting $z>6$ galaxies are
likely to become more viable in the future and will possibly reduce
contamination of intermediate-$z$ galaxies further.

###  6.4 Possible caveats in this work

As a few final words of caution, a possible caveat in this work is how the
Euclid (+ancillary) photometry is simulated from best-fit SED templates, as
compared to extracting fluxes from real Euclid (+ancillary) images or even
simulated images that emulate the real data more directly. By sampling the
photometry directly from models, we do not take into account factors that
could degrade the quality of the photometric measurements, such as telescope
defects and source confusion. We do note, however, that Euclid will have an
excellent (HST-like) angular resolution, and, thus, source blending is
unlikely to be a major problem, considering the depths of the Euclid Deep
Fields. Nonetheless, the simulated data in this work are idealised to a
certain degree, and as such photometric redshift degeneracies could be more
prevalent for the real data. Moreover, we used the same template set to
simulate the Euclid (+ancillary) photometry and to recompute the photometric
redshifts based on these simulated data. The template fluxes were randomised
as we described in Sect. 3, but nonetheless, our results on the contamination
fractions could be affected by this.

Second, throughout this work we have not distinguished between normal galaxies
and galaxies that host an active galactic nucleus (AGN), even though we know
that these types of galaxies are present in the COSMOS field (Brusa et al.
2010; Delvecchio et al. 2017; Chang et al. 2017). We cross-correlated our
UltraVISTA DR4 catalogue with the C-COSMOS X-ray catalogue (Civano et al.
2016) and identified 996 X-ray AGN sources with host galaxy redshifts between
$z=1$–5.3. These sources are rare, as they constitute only 0.6 % of the total
galaxy population between $z=1$–5.3. Moreover, we find that the simulated
redshifts of these sources lie below $z=6$ in any scenario of Euclid
(+ancillary) data. Therefore, we conclude that our X-ray selected AGNs do not
contaminate the Euclid high-redshift galaxy sample. However, alternative
methods to identify AGNs exist, so more work regarding this AGN contamination
of $z>6$ galaxies is desired.

Lastly, throughout this work we have assumed the expected full depths for the
Euclid and Rubin photometry, finalised once the nominal missions have been
completed. With this assumption, we have demonstrated how ancillary photometry
is only marginally helpful to prevent intermediate-$z$ contaminants from
entering the high-redshift sample, and how $z>6$ galaxies are well-recovered
from Euclid data alone for both bright and faint galaxies. This does not imply
that the ancillary data have an insignificant role in the study of high-$z$
galaxies. The efforts to gather ancillary imaging in the Euclid Deep Fields
are very important. For a long time the ancillary data in the optical bands
will be deeper than that provided by Euclid $I_{\scriptscriptstyle\rm E}$. In
fact, the final $I_{\scriptscriptstyle\rm E}$ depth assumed here will not be
achieved until late stages of the Euclid operations. Given that the Euclid and
Rubin missions will run over many years, interim datasets at intermediate
depths will be released. Therefore, ancillary imaging will be very important
for all the studies of high-$z$ galaxies performed in the first years of
Euclid data analysis. In general, we conclude that more work is required to
analyse the contamination fraction estimates at intermediate Euclid and Rubin
depths.

##  7 Summary and conclusions

We have investigated the contamination from intermediate-$z$ interlopers
($z=1$–5.8) of the $z>6$ galaxy population as expected for the Euclid Deep
Fields. Our tests are based on $\sim$176 000 real galaxies at $z=1$–8 in a
$\sim$0.7 deg2 area selected from the UltraVISTA ultra-deep survey in the
COSMOS field and an additional sample of $\sim$96 000 mock galaxies with
$25.3\leq H<27.0$ that follow a CANDELS-like photometric redshift and $H$
magnitude distribution. For both datasets, we simulated Euclid and ancillary
photometry from fiducial 28-band photometry and subsequently re-derived
photometric redshifts for eight scenarios of data availability: (i) Euclid
data only; (ii) Euclid and Rubin $ugrizy$ data; (iii) Euclid and ultra-deep
Rubin $ugrizy$ data from the Rubin DDF; (iv) Euclid and CFHT $u$ and Subaru
HSC $griz$ data from the H20 survey; (v) Euclid and Spitzer IRAC 3.6 and 4.5
$\mathrm{\SIUnitSymbolMicro m}$ data; (vi) Euclid, Rubin, and Spitzer data;
(vii) Euclid, Rubin DDF, and Spitzer data; and (viii) Euclid, H20, and Spitzer
data. We emphasise that the findings presented below are only representative
of galaxies up to $z=8$ due to the limitations of our fiducial sample and that
we cannot assess the photometric redshift recovery of Euclid $z>8$ galaxies.

  1. 1.

We determined the fraction of intermediate-$z$ contaminants (fiducial
$z=1$–5.8) amongst the apparent $z>6$ population as identified from Euclid
(+ancillary) data. Based on the bright UltraVISTA-like sample, we estimate the
contamination fraction of $z>6$ galaxies observed with Euclid to be
$0.18^{+0.07}_{-0.06}$. Contrarily, when we consider the faint mock sample,
the contamination fraction of $z>6$ galaxies with Euclid is significantly
higher at $0.39^{+0.10}_{-0.11}$.

  2. 2.

Based on the bright UltraVISTA-like sample, we find that the contamination
fraction is reduced to $0.13^{+0.04}_{-0.05}$ when including Spitzer IRAC
photometry, and to $0.04^{+0.03}_{-0.02}$ when including Rubin DDF photometry.
In our most optimistic scenario, where we combined Euclid, Rubin DDF, and
Spitzer photometry, virtually any possible contamination from intermediate-$z$
interlopers is ruled out; the contamination fraction is
$0.01^{+0.0006}_{-0.0001}$. Conversely, when we consider the faint mock
sample, the contamination fraction is only significantly reduced once we
include the ultra-deep Rubin DDF photometry, to $0.07^{+0.03}_{-0.03}$.

  3. 3.

We replicated our analysis of the bright UltraVISTA-like sample on the
COSMOS2020 galaxy catalogue, which contains independent photometry over the
same field, in combination with various SED fitting routines. We find
generally good agreement between the contamination fraction estimated from our
own photometry and those derived using LePhare/LePhare++ with the COSMOS2020
catalogue. However, we obtain different results using the EAZY code, which
returns a lower fraction of contaminants as well as a lower completeness in
the recovery of $z>6$ galaxies.

  4. 4.

The contaminants of $z>6$ galaxies have distinctly different
$(I_{\scriptscriptstyle\rm E}-Y_{\scriptscriptstyle\rm E})$ colours from true
$z>6$ galaxies, so colour selection criteria can be used to separate these
populations a posteriori of obtaining the Euclid-like photometric redshifts.
However, many contaminants have unconstrained $(I_{\scriptscriptstyle\rm
E}-Y_{\scriptscriptstyle\rm E})$ colours and as such cannot be included in
this analysis. Regardless, we have presented a grid of
$(I_{\scriptscriptstyle\rm E}-Y_{\scriptscriptstyle\rm E})$ &
$(Y_{\scriptscriptstyle\rm E}-J_{\scriptscriptstyle\rm E})$ colour cuts that
balance the contamination fraction and $z>6$ galaxy completeness. For the
bright UltraVISTA-like sample, the colour cut $(I_{\scriptscriptstyle\rm
E}-Y_{\scriptscriptstyle\rm E})>2.8$ & $(Y_{\scriptscriptstyle\rm
E}-J_{\scriptscriptstyle\rm E})<1.4$ when applied to the apparent $z>6$
galaxies (that have a flux measurement in the $I_{\scriptscriptstyle\rm E}$
and/or $Y_{\scriptscriptstyle\rm E}$ bands; identified from Euclid data)
reduces the contamination fraction to 0.01 while preserving 81 % of the
fiducial $z>6$ galaxies (i.e. galaxies at fiducial $z=6$–8 that are recovered
at $z>6$ from Euclid photometry). Considering the faint mock sample, these
proposed colour selection criteria are not useful, given that the majority of
contaminants are undetected in both the $I_{\scriptscriptstyle\rm E}$ and
$Y_{\scriptscriptstyle\rm E}$ bands.

  5. 5.

Alternatively, we find that imposing a $5\sigma$ detection threshold
requirement in at least one of the Euclid NIR bands is useful for obtaining a
purer apparent $z>6$ sample, specifically for the faint mock sample. By
employing this detection threshold requirement, the contamination fraction of
the faint mock $z>6$ galaxies is $0.25^{+0.07}_{-0.07}$ (Euclid data alone),
and we maintain a $z>6$ completeness of 91 %. As such, the identification of
faint high-redshift sources in the Euclid Deep Fields will be very efficient,
with a moderate contamination of intermediate-$z$ galaxies.

  6. 6.

We investigated if one can select a pure sample of $z>6$ galaxies from Euclid
colours alone. Based on the bright UltraVISTA-like sample, we have shown how a
strict colour cut at $(I_{\scriptscriptstyle\rm E}-Y_{\scriptscriptstyle\rm
E})>3.4$ & $(Y_{\scriptscriptstyle\rm E}-J_{\scriptscriptstyle\rm E})<0.9$
could possibly select a pure sample of high-redshift galaxies, whilst
maintaining a $z>6$ completeness of 58 %. However, this cannot be achieved for
the faint mock sample at all, and therefore we conclude that deriving
photometric redshifts cannot be circumvented.

  7. 7.

We compared the fiducial physical properties (based on 28 bands in COSMOS) of
the intermediate-redshift galaxies that are $z>6$ contaminants and stable
intermediate-$z$ galaxies (i.e. $z=1$–6 galaxies that stay in the same
redshift bin when observed with Euclid (+ancillary) data). The analysis of the
physical parameters is solely based on the bright UltraVISTA-like sample. We
find that the $z>6$ contaminants reside primarily at fiducial redshifts
$z_{\mathrm{fid}}\sim 1$–3 and $z_{\mathrm{fid}}\sim 4.5$–6 and are primarily
faint, regardless of which ancillary data are added to the Euclid photometry.
We identify three distinct populations: (i) moderately reddened galaxies at
$z_{\mathrm{fid}}=$1–3.5 that are misidentified through the typical confusion
of the Lyman-$\alpha$ break at $\lambda=1216\,\AA$ and the $4000\,\AA$ break;
(ii) a group of contaminants at $z_{\mathrm{fid}}=3.5$–5 that are strongly
dust-reddened; and (iii) a group at $z_{\mathrm{fid}}=5$–6 that have typically
ill-defined, flat, fiducial SEDs and are mistaken for $z\sim 6$ galaxies.

  8. 8.

Moreover, we compared the physical properties as derived with Euclid
(+ancillary) photometry of $z>6$ contaminants and true $z>6$ galaxies. We have
demonstrated how Spitzer photometry is essential for recovering the stellar
masses of $z>6$ galaxies. The $z>6$ contaminants are most separated from true
$z>6$ galaxies based on their colour excess, as the latter have virtually no
dust attenuation. Although the parameter distributions of true $z>6$ galaxies
and $z>6$ are generally statistically different, we find no selection criteria
that effectively separate them based on their physical parameters.

Overall, we conclude that the Euclid high-redshift recovery will be excellent
for bright $z=6$–8 galaxies, and successful for faint galaxies as well. In
addition, ultra-deep ancillary photometry is highly effective at reducing
contamination from intermediate-$z$ interlopers to the $z>6$ galaxy sample.

###### Acknowledgements.

Based on data products from observations conducted with ESO Telescopes at the
Paranal Observatory under ESO program ID 179.A-2005 and on data products
produced by TERAPIX and the Cambridge Astronomy Survey Unit on behalf of the
UltraVISTA consortium. Also based in part on observations carried out with the
Spitzer Space Telescope, which is operated by the Jet Propulsion Laboratory,
California Institute of Technology under a contract with NASA. Also based on
observations carried out by NASA/ESA Hubble Space Telescope, obtained and
archived at the Space Telescope Science Institute; and the Subaru Telescope,
which is operated by the National Astronomical Observatory of Japan. This
research has made use of the NASA/IPAC Infrared Science Archive, which is
operated by the Jet Propulsion Laboratory, California Institute of Technology,
under contract with NASA.
SvM and KC acknowledge funding from the European Research Council through the
award of the Consolidator Grant ID 681627-BUILDUP. PD acknowledges support
from the European Research Council’s starting grant ERC StG-717001 (DELPHI),
from the NWO grant 016.VIDI.189.162 (ODIN) and the European Commission’s and
University of Groningen’s CO-FUND Rosalind Franklin program. The Euclid
Consortium acknowledges the European Space Agency and a number of agencies and
institutes that have supported the development of Euclid, in particular the
Academy of Finland, the Agenzia Spaziale Italiana, the Belgian Science Policy,
the Canadian Euclid Consortium, the French Centre National d’Etudes Spatiales,
the Deutsches Zentrum für Luft- und Raumfahrt, the Danish Space Research
Institute, the Fundação para a Ciência e a Tecnologia, the Ministerio de
Economia y Competitividad, the National Aeronautics and Space Administration,
the National Astronomical Observatory of Japan, the Netherlandse
Onderzoekschool Voor Astronomie, the Norwegian Space Agency, the Romanian
Space Agency, the State Secretariat for Education, Research and Innovation
(SERI) at the Swiss Space Office (SSO), and the United Kingdom Space Agency. A
complete and detailed list is available on the Euclid web site
(http://www.euclid-ec.org). We thank Smaran Deshmukh for useful discussions on
the SMUVS catalogue photometry. We thank Marc Sauvage for carefully reading
the manuscript and providing constructive comments for the Euclid Consortium
internal review.

## References

  * Aihara et al. (2019) Aihara, H., AlSayyad, Y., Ando, M., et al. 2019, PASJ, 71, 114
  * Arnouts et al. (1999) Arnouts, S., Cristiani, S., Moscardini, L., et al. 1999, MNRAS, 310, 540
  * Arnouts et al. (2013) Arnouts, S., Le Floc’h, E., Chevallard, J., et al. 2013, A&A, 558, A67
  * Ashby et al. (2018) Ashby, M. L. N., Caputi, K. I., Cowley, W., et al. 2018, ApJS, 237, 39
  * Ashby et al. (2015) Ashby, M. L. N., Willner, S. P., Fazio, G. G., et al. 2015, ApJS, 218, 33
  * Atek et al. (2015) Atek, H., Richard, J., Kneib, J.-P., et al. 2015, ApJ, 800, 18
  * Atek et al. (2011) Atek, H., Siana, B., Scarlata, C., et al. 2011, ApJ, 743, 121
  * Bertin & Arnouts (1996) Bertin, E. & Arnouts, S. 1996, A&AS, 117, 393
  * Bisigello et al. (2020) Bisigello, L., Kuchner, U., Conselice, C. J., et al. 2020, MNRAS, 494, 2337
  * Bouwens et al. (2012) Bouwens, R. J., Illingworth, G. D., Oesch, P. A., et al. 2012, ApJ, 754, 83
  * Bouwens et al. (2010) Bouwens, R. J., Illingworth, G. D., Oesch, P. A., et al. 2010, ApJ, 709, L133
  * Bouwens et al. (2021) Bouwens, R. J., Oesch, P. A., Stefanon, M., et al. 2021, AJ, 162, 47
  * Bowler et al. (2015) Bowler, R. A. A., Dunlop, J. S., McLure, R. J., et al. 2015, MNRAS, 452, 1817
  * Bowler et al. (2014) Bowler, R. A. A., Dunlop, J. S., McLure, R. J., et al. 2014, MNRAS, 440, 2810
  * Bowler et al. (2020) Bowler, R. A. A., Jarvis, M. J., Dunlop, J. S., et al. 2020, MNRAS, 493, 2059
  * Brammer et al. (2008) Brammer, G. B., van Dokkum, P. G., & Coppi, P. 2008, ApJ, 686, 1503
  * Brusa et al. (2010) Brusa, M., Civano, F., Comastri, A., et al. 2010, ApJ, 716, 348
  * Bruzual & Charlot (2003) Bruzual, G. & Charlot, S. 2003, MNRAS, 344, 1000
  * Burgasser (2014) Burgasser, A. J. 2014, in Astronomical Society of India Conference Series, Vol. 11, Astronomical Society of India Conference Series, 7–16
  * Calzetti et al. (2000) Calzetti, D., Armus, L., Bohlin, R. C., et al. 2000, ApJ, 533, 682
  * Caputi et al. (2015) Caputi, K. I., Ilbert, O., Laigle, C., et al. 2015, ApJ, 810, 73
  * Caputi et al. (2006) Caputi, K. I., McLure, R. J., Dunlop, J. S., Cirasuolo, M., & Schael, A. M. 2006, MNRAS, 366, 609
  * Castellano et al. (2016) Castellano, M., Dayal, P., Pentericci, L., et al. 2016, ApJ, 818, L3
  * Chabrier (2003) Chabrier, G. 2003, PASP, 115, 763
  * Chang et al. (2017) Chang, Y.-Y., Le Floc’h, E., Juneau, S., et al. 2017, ApJS, 233, 19
  * Civano et al. (2016) Civano, F., Marchesi, S., Comastri, A., et al. 2016, ApJ, 819, 62
  * Conroy & Gunn (2010) Conroy, C. & Gunn, J. E. 2010, ApJ, 712, 833
  * Conroy et al. (2009) Conroy, C., Gunn, J. E., & White, M. 2009, ApJ, 699, 486
  * Cropper et al. (2016) Cropper, M., Pottinger, S., Niemi, S., et al. 2016, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 9904, Space Telescopes and Instrumentation 2016: Optical, Infrared, and Millimeter Wave, ed. H. A. MacEwen, G. G. Fazio, M. Lystrup, N. Batalha, N. Siegler, & E. C. Tong, 99040Q
  * Dahlen et al. (2013) Dahlen, T., Mobasher, B., Faber, S. M., et al. 2013, ApJ, 775, 93
  * de Barros et al. (2014) de Barros, S., Schaerer, D., & Stark, D. P. 2014, A&A, 563, A81
  * Delvecchio et al. (2017) Delvecchio, I., Smolčić, V., Zamorani, G., et al. 2017, A&A, 602, A3
  * Deshmukh et al. (2018) Deshmukh, S., Caputi, K. I., Ashby, M. L. N., et al. 2018, ApJ, 864, 166
  * Duncan et al. (2014) Duncan, K., Conselice, C. J., Mortlock, A., et al. 2014, MNRAS, 444, 2960
  * Ellis et al. (2013) Ellis, R. S., McLure, R. J., Dunlop, J. S., et al. 2013, ApJ, 763, L7
  * Euclid Collaboration: Desprez et al. (2020) Euclid Collaboration: Desprez, G., Paltani, S., Coupon, J., et al. 2020, A&A, 644, A31
  * Euclid Collaboration: Moneti et al. (2022) Euclid Collaboration: Moneti, A., McCracken, H. J., Shuntov, M., et al. 2022, A&A, 658, A126
  * Euclid Collaboration: Scaramella et al. (2022) Euclid Collaboration: Scaramella, R., Amiaux, J., Mellier, Y., et al. 2022, A&A, 662, A112
  * Euclid Collaboration: Schirmer et al. (2022) Euclid Collaboration: Schirmer, M., Jahnke, K., Seidel, G., et al. 2022, A&A, 662, A92
  * Fazio et al. (2004) Fazio, G. G., Hora, J. L., Allen, L. E., et al. 2004, ApJS, 154, 10
  * Finkelstein et al. (2015) Finkelstein, S. L., Ryan, Russell E., J., Papovich, C., et al. 2015, ApJ, 810, 71
  * Foley et al. (2018) Foley, R. J., Koekemoer, A. M., Spergel, D. N., et al. 2018, arXiv e-prints, arXiv:1812.00514
  * Grogin et al. (2011) Grogin, N. A., Kocevski, D. D., Faber, S. M., et al. 2011, ApJS, 197, 35
  * Hsieh et al. (2012) Hsieh, B.-C., Wang, W.-H., Hsieh, C.-C., et al. 2012, ApJS, 203, 23
  * Huang et al. (2015) Huang, X., Zheng, W., Wang, J., et al. 2015, ApJ, 801, 12
  * Ilbert et al. (2006) Ilbert, O., Arnouts, S., McCracken, H. J., et al. 2006, A&A, 457, 841
  * Ilbert et al. (2009) Ilbert, O., Capak, P., Salvato, M., et al. 2009, ApJ, 690, 1236
  * Inoue et al. (2014) Inoue, A. K., Shimizu, I., Iwata, I., & Tanaka, M. 2014, MNRAS, 442, 1805
  * Ivezić et al. (2019) Ivezić, Ž., Kahn, S. M., Tyson, J. A., et al. 2019, ApJ, 873, 111
  * Jarvis et al. (2013) Jarvis, M. J., Bonfield, D. G., Bruce, V. A., et al. 2013, MNRAS, 428, 1281
  * Kennicutt (1998) Kennicutt, R. C. 1998, ApJ, 498, 541
  * Koekemoer et al. (2011) Koekemoer, A. M., Faber, S. M., Ferguson, H. C., et al. 2011, ApJS, 197, 36
  * Laigle et al. (2016) Laigle, C., McCracken, H. J., Ilbert, O., et al. 2016, ApJS, 224, 24
  * Laureijs et al. (2011) Laureijs, R., Amiaux, J., Arduini, S., et al. 2011, arXiv e-prints, arXiv:1110.3193
  * Maciaszek et al. (2016) Maciaszek, T., Ealet, A., Jahnke, K., et al. 2016, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 9904, Space Telescopes and Instrumentation 2016: Optical, Infrared, and Millimeter Wave, ed. H. A. MacEwen, G. G. Fazio, M. Lystrup, N. Batalha, N. Siegler, & E. C. Tong, 99040T
  * McCracken et al. (2012) McCracken, H. J., Milvang-Jensen, B., Dunlop, J., et al. 2012, A&A, 544, A156
  * Nayyeri et al. (2017) Nayyeri, H., Hemmati, S., Mobasher, B., et al. 2017, ApJS, 228, 7
  * Oesch et al. (2014) Oesch, P. A., Bouwens, R. J., Illingworth, G. D., et al. 2014, ApJ, 786, 108
  * Oesch et al. (2012) Oesch, P. A., Bouwens, R. J., Illingworth, G. D., et al. 2012, ApJ, 745, 110
  * Oke & Gunn (1983) Oke, J. B. & Gunn, J. E. 1983, ApJ, 266, 713
  * Pentericci et al. (2014) Pentericci, L., Vanzella, E., Fontana, A., et al. 2014, ApJ, 793, 113
  * Polletta et al. (2007) Polletta, M., Tajer, M., Maraschi, L., et al. 2007, ApJ, 663, 81
  * Prevot et al. (1984) Prevot, M. L., Lequeux, J., Maurice, E., Prevot, L., & Rocca-Volmerange, B. 1984, A&A, 132, 389
  * Roberts-Borsani et al. (2022) Roberts-Borsani, G., Morishita, T., Treu, T., Leethochawalit, N., & Trenti, M. 2022, ApJ, 927, 236
  * Salmon et al. (2020) Salmon, B., Coe, D., Bradley, L., et al. 2020, ApJ, 889, 189
  * Schlafly & Finkbeiner (2011) Schlafly, E. F. & Finkbeiner, D. P. 2011, ApJ, 737, 103
  * Scoville et al. (2007) Scoville, N., Aussel, H., Brusa, M., et al. 2007, ApJS, 172, 1
  * Smirnov (1939) Smirnov, N. V. 1939, Bull. Math. Univ. Moscou, 2, 3
  * Song et al. (2016) Song, M., Finkelstein, S. L., Ashby, M. L. N., et al. 2016, ApJ, 825, 5
  * Stefanon et al. (2019) Stefanon, M., Labbé, I., Bouwens, R. J., et al. 2019, ApJ, 883, 99
  * Steidel et al. (1996) Steidel, C. C., Giavalisco, M., Pettini, M., Dickinson, M., & Adelberger, K. L. 1996, ApJ, 462, L17
  * Stern et al. (2007) Stern, D., Kirkpatrick, J. D., Allen, L. E., et al. 2007, ApJ, 663, 677
  * Vulcani et al. (2017) Vulcani, B., Trenti, M., Calvi, V., et al. 2017, ApJ, 836, 239
  * Weaver et al. (2022) Weaver, J. R., Kauffmann, O. B., Ilbert, O., et al. 2022, ApJS, 258, 11
  * Werner et al. (2004) Werner, M. W., Roellig, T. L., Low, F. J., et al. 2004, ApJS, 154, 1
  * Willott et al. (2013) Willott, C. J., McLure, R. J., Hibon, P., et al. 2013, AJ, 145, 4

##  Appendix A COSMOS2020

In order to verify the validity of our estimates on the fraction of
intermediate-$z$ interlopers amongst the Euclid-observed $z>6$ galaxy
population, we have repeated our analysis of the bright UltraVISTA-like
sources on a different photometric catalogue and different SED fitting codes.

We have used photometry from the COSMOS2020 catalogue (Weaver et al. 2022;
version of December 2020; hereafter W21), which comprises the latest data
release of COSMOS (Scoville et al. 2007). This catalogue includes new ultra-
deep optical data from DR2 of the HSC Subaru Strategic Program (Aihara et al.
2019), infrared data from the fourth release of the UltraVISTA survey
(McCracken et al. 2012), and all Spitzer data in four IRAC channels ever
obtained in the COSMOS field (Euclid Collaboration: Moneti et al. 2022). Two
independent versions of the COSMOS2020 catalogue using different techniques of
measuring photometry were constructed, and in this work we made use of the so-
called Classic catalogue. For this version, source detection was performed
with SourceExtractor on the combined $izYJHK_{\rm s}$ detection image, after
which photometry was measured in $2^{\prime\prime}$ and $3^{\prime\prime}$
diameter apertures on the PSF-homogenised images in 40 optical and NIR bands.
The IRAC fluxes were extracted separately using the IRACLEAN software (Hsieh
et al. 2012). The final catalogue consists of roughly $1\,700\,000$ sources.
We refer the reader to W21 for a complete description of the COSMOS2020
Classic photometric catalogue. Throughout this paper we globally refer to the
COSMOS2020 Classic photometry as the COSMOS2020 aperture photometry.

In addition, W21 presented two independent sets of photometric redshifts for
all sources in the COSMOS2020 Classic catalogue, derived from two different
template-fitting routines, namely LePhare and EAZY (again, we refer the reader
to the original publication for the full details on their methodology). Here
we make use of both sets of photometric redshifts.

For our tests, we only considered sources classified as galaxies (type=0) from
this catalogue. In addition, we considered only the very deepest parts of
COSMOS2020 catalogue and therefore cut the catalogue to ensure its area
overlaps exactly with our own UltraVISTA catalogue (which comprises three out
of the four ultra-deep stripes). These measures reduced the sample to 316 698
galaxies. We used the aperture and Galactic extinction corrections from W21
(the latter based on dust maps from Schlafly & Finkbeiner 2011, consistent
with our own analysis) to obtain total, dust-corrected photometry.

Our goal is to derive Euclid-like photometry and photometric redshifts based
on the COSMOS2020 catalogue, in three independent tests where we use different
SED fitting algorithms.

###  A.1 COSMOS2020 with the LePhare++ routine

For our first test, we simply replicated the SED fitting process with LePhare
as described in W21 to the best of our ability. We used an updated version of
LePhare called LePhare++, which is based on the Fortran version of the code
but has been migrated to C++. The set of galaxy templates included 19
empirical elliptical and spiral templates from Polletta et al. (2007), 12
star-forming galaxy models from the Bruzual & Charlot (2003, hereafter BC03)
library and two BC03 templates with exponentially declining star-formation
rates. The considered attenuation curves are from Calzetti et al. (2000),
Prevot et al. (1984), and two realisations of the Calzetti law including the
$2175\,\AA$ bump (Prevot et al. 1984) at different strengths. Dust extinction
was varied between $E(B-V)=0.0$ and 0.5 in 0.05 steps and the considered
redshift range was $z=0$–10 with $\Delta z=0.01$ increments. Emission lines
were included in the fit following the original recipe from LePhare (Ilbert et
al. 2009), but using a new functionality in LePhare++, we allowed the modelled
Oiii $\lambda\lambda 4931,5007$ flux to vary between 50 % and 150 % (in 25 %
steps) of the theoretical line flux.

This run was based on 34 bands from the COSMOS2020 photometric catalogue:
GALEX NUV; CFHT MegaCam u and u∗; Subaru HSC $g$, $r$, $i$, $z$, and $y$;
Subaru Suprime-Cam $B$, $V$, $r^{+}$, $i^{+}$, $z^{++}$, and 14 intermediate-
and narrow bands; VISTA $Y$, $J$, $H$, $K_{\rm s}$, and $NB118$; and IRAC
channels 1 and 2. We applied photometric offsets as presented in Table 2 in
W21 to the fluxes and added 0.02 mag errors in quadrature to the photometric
errors in the optical bands, 0.05 mag errors to the VISTA $J$, $H$, and
$K_{\rm s}$ bands and IRAC channel 1 and the three narrow bands, and 0.1 mag
to IRAC channel 2.

Once we derived the photometric redshifts for the COSMOS2020 galaxies in this
way, we fixed their redshifts to these values and used LePhare++ with an
alternate template library to derive their physical properties and best-fit
SEDs from which we measure Euclid-like photometry. The two-step strategy
follows W21 and is necessary as the empirical galaxy templates cannot be used
to derive physical parameters. In this second run, we used a set of 11 BC03
templates with nine exponentially declining and two delayed exponentially
declining SFHs, assuming a Chabrier (2003) IMF. We adopted two metallicities:
solar ($Z=0.02$) and half-solar ($Z=0.008$). The dust extinction was allowed
to vary between $E(B-V)=0.0$ and 0.7 in 0.1 steps. The extinction laws used
were the Calzetti et al. (2000) law and a curve that has a slope in between
the Calzetti et al. (2000) law and the Prevot et al. (1984) curve
($\lambda^{0.9}$; Appendix A of Arnouts et al. 2013). Again, we add emission
lines using the recipe from Ilbert et al. (2009), but employ no dispersion of
the Oiii doublet. We emphasise again that this two-step method is directly
taken from W21.

Once we obtained the best-fit SEDs, we derived the Euclid (+ancillary)
photometry in precisely the same manner as presented in Sect. 3. An important
caveat is that the source detection of the COSMOS2020 catalogue was performed
using the UltraVISTA DR4 mosaics, which means the catalogue is deeper than our
own DR3-derived UltraVISTA catalogue. Given that we want to exclude sources
that lie beyond the detection abilities of Euclid, we restrained the
COSMOS2020 sample to sources that have at least a $3\sigma$ detection in both
the simulated $J_{\scriptscriptstyle\rm E}$ and $H_{\scriptscriptstyle\rm E}$
bands.

In total, the COSMOS2020 photometry processed with the above described SED
fitting routine, after applying the $J_{\scriptscriptstyle\rm E}$\- and
$H_{\scriptscriptstyle\rm E}$-band requirements, yields 164 800 galaxies with
fiducial redshifts $z_{\mathrm{fid}}=1$–8, of which 598 are at
$z_{\mathrm{fid}}=6$–8. Out of this sample, 127 985 galaxies are detected in
at least IRAC channel 1 or channel 2, and 342 of these sources are at
$z_{\mathrm{fid}}=6$–8.

For each of the eight Euclid and ancillary data availability scenarios, we re-
derived the photometric redshifts of the COSMOS2020 galaxies from their
simulated Euclid (+ancillary) photometry alone, using LePhare++ with the same
set of empirical and BC03 templates as described above in the first step of
the two-step SED fitting strategy. Following the method outlined in Sect. 4.2,
we calculated the high-redshift completeness and fraction of intermediate-$z$
interlopers amongst $z>6$ galaxies. The results are shown in Table 7.

###  A.2 COSMOS2020 with the LePhare routine used in this paper

For our second test, we used the COSMOS2020 aperture photometry in combination
with LePhare (the Fortran version), using the SED fitting settings we adopted
for our own UltraVISTA catalogue (Sect. 2). As a reminder, our SED fitting
routine considers among others a smaller redshift range ($z=0$–9), templates
from the BC03 library only and a broader dust extinction range, that is,
$E(B-V)=0$–1 with the Calzetti et al. (2000) reddening law. We based this test
on the same 34 optical and infrared band as listed for the previous test. We
did not derive the fiducial photometric redshifts from scratch, instead fixing
the redshift to those derived by W21 with LePhare. However, given that for
this test, our LePhare parameters are not identical to those of W21, we
minimised any discrepancies between our resulting best-fit SED and the fixed
photometric redshift by deriving photometric offsets. This was done
iteratively, until the difference between the input photometry and template
fluxes converged. Subsequently, we sample the Euclid (+ancillary) photometry
from the final best-fit SED.

With this second SED fitting method, the COSMOS2020 photometry yields 165 190
galaxies at fiducial redshifts $z_{\mathrm{fid}}=1$–8 that exceed the
$3\sigma$ flux in the $J_{\scriptscriptstyle\rm E}$ and
$H_{\scriptscriptstyle\rm E}$ bands, of which 466 are at
$z_{\mathrm{fid}}=6$–8. Out of this sample, 124 520 galaxies are detected in
at least IRAC channel 1 or channel 2, and 253 of these are at
$z_{\mathrm{fid}}=6$–8.

Subsequently, photometric redshifts based on the Euclid (+ancillary)
photometry were recomputed using our own LePhare fitting routine. We derived
the contamination fraction and $z>6$ completeness for each combination of
Euclid and ancillary photometry based on this second test with the COSMOS2020
catalogue, and show the results in Table 8.

###  A.3 COSMOS2020 with the EAZY code

For our third test, we used the COSMOS2020 aperture photometry but now with
another SED fitting code, that is, the newly updated version of the EAZY code
(Brammer et al. 2008), which has been migrated to Python (version 0.5.2.dev7).
Similar to LePhare, EAZY is a template-fitting algorithm that assigns
photometric redshifts to multi-wavelength photometry. This code was used by
W21 to provide a second set of photometric redshifts based on the COSMOS2020
data. Therefore, we reproduce their process as closely as possible in order to
recover the EAZY best-fit templates for the COSMOS2020 Classic catalogue.

Following W21, EAZY uses a set of 17 templates derived from the Flexible
Stellar Population Synthesis models (Conroy et al. 2009; Conroy & Gunn 2010)
with a range of dust attenuations and SFHs, and fits linear combinations of
these templates to the observed photometry. The redshift is allowed to range
from $z=0$–12 and a $K_{\rm s}$ apparent magnitude prior is applied in the
process. We include the photometric offsets computed by W21 in the fit,
derived iteratively from a subsample of sources with spectroscopic redshifts.
Multiplicative factors as presented in W21 were applied to the photometric
errors, and the same magnitude errors as used for LePhare were added in
quadrature. We selected a sample of $z=1$–8 galaxies from the EAZY COSMOS2020
catalogue and derived our own fiducial photometric redshifts with this method.
EAZY was used to retrieve the observed frame fluxes in the Euclid (+ancillary)
filters.

Our strategy to ensure realistic depths for the simulated photometry is
slightly different. With LePhare, one is able to set flux upper limits such
that any template that produces fluxes higher than the upper limits in bands
with non-detections is immediately discarded. EAZY performs the template
fitting in linear flux density units, so it naturally accounts for the
$\sigma$ flux limits we derive from the expected Euclid (+ancillary) depths
when we set the flux uncertainty in each band to half the $2\sigma$ flux.
Subsequently, we re-derived the photometric redshifts based on the simulated
photometry using the templates and settings as used for the COSMOS2020
photometry, although we did not employ the apparent magnitude prior.

The COSMOS2020 photometry with EAZY yields 160 260 galaxies at fiducial
$z_{\mathrm{fid}}=1$–8 that satisfy the $J_{\scriptscriptstyle\rm E}$\- and
$H_{\scriptscriptstyle\rm E}$-band criteria we imposed, of which 369 are at
$z_{\mathrm{fid}}=6$–8. We calculated the contamination fraction and high-
redshift completeness for three scenarios of Euclid and ancillary photometry:
Euclid, Euclid+H20, and Euclid+Rubin DDF. The results are shown in Table 9.
For our test with EAZY, we chose to investigate only these three scenarios as
they reflect well the different levels of depth expected from the ancillary
data.

Lastly, we note that uncertainties on the contamination fraction were derived
for all three tests, using the exact same method adopted for our photometric
catalogue (see Sect. 4.2).

Table 7: Contaminants amongst $z>6$ galaxies using COSMOS2020 with LePhare++.

Filters | True $z>6$ | Contaminants | Contaminant Fraction | Completeness
---|---|---|---|---
Euclid | 527 | 90 | $0.15^{+0.03}_{-0.04}$ | 88 %
Euclid \+ Rubin | 531 | 98 | $0.16^{+0.05}_{-0.06}$ | 89 %
Euclid \+ Rubin DDF | 561 | 30 | $0.05^{+0.02}_{-0.02}$ | 94 %
Euclid \+ H20 | 541 | 86 | $0.14^{+0.04}_{-0.04}$ | 90 %
Euclid \+ Spitzer | 292 | 55 | $0.16^{+0.05}_{-0.05}$ | 85 %
Euclid \+ Rubin + Spitzer | 301 | 57 | $0.16^{+0.04}_{-0.04}$ | 88 %
Euclid \+ Rubin DDF + Spitzer | 317 | 18 | $0.05^{+0.02}_{-0.01}$ | 93 %
Euclid \+ H20 + Spitzer | 300 | 54 | $0.15^{+0.04}_{-0.04}$ | 88 %

888Number of true $z>6$ galaxies and $z>6$ contaminants, using the COSMOS2020
Classic catalogue from Weaver et al. (2022). The fiducial photometric
redshifts and Euclid (+ancillary) photometry were derived using the methods
described in Weaver et al. (2022). Only sources with a $3\sigma$ detection in
both the $J_{\scriptscriptstyle\rm E}$ and $H_{\scriptscriptstyle\rm E}$ bands
are considered here.

Table 8: Contaminants amongst $z>6$ galaxies using COSMOS2020 with LePhare.

Filters | True $z>6$ | Contaminants | Contaminant Fraction | Completeness
---|---|---|---|---
Euclid | 418 | 82 | $0.16^{+0.05}_{-0.04}$ | 90 %
Euclid \+ Rubin | 414 | 81 | $0.16^{+0.06}_{-0.05}$ | 89 %
Euclid \+ Rubin DDF | 424 | 18 | $0.04^{+0.02}_{-0.02}$ | 91 %
Euclid \+ H20 | 418 | 58 | $0.12^{+0.04}_{-0.04}$ | 90 %
Euclid \+ Spitzer | 227 | 43 | $0.16^{+0.06}_{-0.06}$ | 90 %
Euclid \+ Rubin + Spitzer | 227 | 47 | $0.17^{+0.06}_{-0.05}$ | 88 %
Euclid \+ Rubin DDF + Spitzer | 228 | 8 | $0.03^{+0.01}_{-0.01}$ | 90 %
Euclid \+ H20 + Spitzer | 228 | 28 | $0.11^{+0.05}_{-0.04}$ | 90 %

999Number of true $z>6$ galaxies and $z>6$ contaminants, using the COSMOS2020
Classic catalogue from Weaver et al. (2022). The fiducial photometric
redshifts and Euclid (+ancillary) photometry were derived using the same
method adopted for the UltraVISTA/SMUVS photometry, described in Sect. 2.2.
Only sources with a $3\sigma$ detection in both the $J_{\scriptscriptstyle\rm
E}$ and $H_{\scriptscriptstyle\rm E}$ bands are considered here.

Table 9: Contaminants amongst $z>6$ galaxies using COSMOS2020 with EAZY.

Filters | True $z>6$ | Contaminants | Contaminant Fraction | Completeness
---|---|---|---|---
Euclid | 299 | 30 | $0.09^{+0.05}_{-0.05}$ | 81 %
Euclid \+ Rubin DDF | 263 | 2 | $0.01^{+0.002}_{-0.006}$ | 71 %
Euclid \+ H20 | 165 | 0 | $0.0^{+0.00}_{-0.00}$ | 45 %
101010Number of true $z>6$ galaxies and $z>6$ contaminants, using the
COSMOS2020 Classic catalogue from Weaver et al. (2022). The fiducial
photometric redshifts and Euclid (+ ancillary) photometry were derived with
EAZY, following the methods outlined in Weaver et al. (2022). Only sources
with a $3\sigma$ detection in both the $J_{\scriptscriptstyle\rm E}$ and
$H_{\scriptscriptstyle\rm E}$ bands are considered here.


# PaDeLLM-NER: Parallel Decoding in Large Language Models for Named Entity
Recognition

Jinghui Lu 1 Ziwei Yang 1 Yanjie Wang 1 Xuejing Liu 2 Brian Mac Namee 3 Can
Huang∗ 1
1 ByteDance
2 University of Chinese Academy of Sciences  China
3 School of Computer Science  University College Dublin
{lujinghui, yangziwei.1221, wangyanjie.prince<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

In this study, we aim to reduce generation latency for Named Entity
Recognition (NER) with Large Language Models (LLMs). The main cause of high
latency in LLMs is the sequential decoding process, which autoregressively
generates all labels and mentions for NER, significantly increase the sequence
length. To this end, we introduce Parallel Decoding in LLM for NER (PaDeLLM-
NER), a approach that integrates seamlessly into existing generative model
frameworks without necessitating additional modules or architectural
modifications. PaDeLLM-NER allows for the simultaneous decoding of all
mentions, thereby reducing generation latency. Experiments reveal that
PaDeLLM-NER significantly increases inference speed that is 1.76 to 10.22
times faster than the autoregressive approach for both English and Chinese.
Simultaneously it maintains the quality of predictions as evidenced by the
performance that is on par with the state-of-the-art across various datasets.

PaDeLLM-NER: Parallel Decoding in Large Language Models for Named Entity
Recognition

**footnotetext: Corresponding author.

##  1 Introduction

Variant | Input Unstructured Text | Output Structured Label-mention String
---|---|---
Augmented Language Paolini et al. (2020); Das et al. (2023) |  Japan, co-hosts of the World Cup in 2002 and ranked 20th in the world by FIFA, are favourites to regain their title here. |  [Japan — LOC], co-hosts of the [World Cup — MISC] in 2002 and ranked 20th in the world by [FIFA — ORG], are favourites to regain their title here.
Structured Annotation Lu et al. (2022, 2023a); Wang et al. (2023c) |  Cuttitta announced his retirement after the 1995 World Cup , where he took issue with being dropped from the Italy side that faced England in the pool stages. |  ((PER): (Cuttitta), (MISC): (1995 World Cup), (LOC): (Italy), (LOC): (England), (ORG): (NULL))

Table 1: Structured output string format used in the literature. The examples
come from CoNLL2003 dataset.

Named Entity Recognition (NER), a fundamental task in Natural Language
Processing (NLP), aims to extract structured information from unstructured
text data. This includes identifying and categorizing key elements such as
Organization, Geopolitical Entity and so on (referred to as “labels”) in
inputs, and pairing them with relevant text spans extracted from the text
(termed “mentions”). Conventionally, NER tasks are carried out through an
extractive paradigm that entails token-level classification and the subsequent
extraction of identified tokens Ma et al. (2020); Liu et al. (2021).

Recent advancements in Large Language Models (LLMs) Raffel et al. (2020a);
Muennighoff et al. (2022); Touvron et al. (2023a, b); Bai et al. (2023); Gao
et al. (2023); Liu et al. (2023d); Yang et al. (2023a) have revolutionized
numerous foundational tasks in NLP, including NER tasks Paolini et al. (2020);
Lu et al. (2022); Das et al. (2023); Lu et al. (2023a); Wang et al. (2023c),
through the adoption of a generative paradigm. This paradigm involves
instruction-tuning a sequence-to-sequence (seq2seq) model. The model takes a
sequence of unstructured text as input and produces a sequence of structured
label-mention pairs as output. Generally, the output structured string should
be formatted to meet two criteria: (1) it should have a clear and
straightforward structure that facilitates post-processing for label and
mention extraction, and (2) it needs to be generated fluidly and efficiently
from the perspective of language models Wang et al. (2023b).

In Table 1, we list two typically used autoregressive output formats found in
the literature : (1) accommodate original input text to contain label
information, which is referred to as “augmented language” Paolini et al.
(2020); Das et al. (2023); (2) directly using a customized, easily-parsed
structured format to output all labels and mentions, which is called
“structured annotation” Lu et al. (2022, 2023a); Wang et al. (2023c). These
formats present certain challenges. For example, augmented language
necessitates duplicating all original input text, thereby increasing output
length and resulting in inference inefficiency. While structure annotation
avoids replicating the entire input, it produces all labels and mentions in an
autoregressive manner. This implies that each subsequently generated pair
depends on its preceding pairs, and when the number of label-mention pairs is
large, it will lead to longer sequences. As demonstrated in Chen et al.
(2023c); Ning et al. (2023), high latency in LLMs mainly stems from lengthy
sequence generation, we believe that by reducing the length of sequence, a
more efficient inference scheme can be provided for NER tasks.

In light of this, we propose Parallel Decoding in LLM for NER (PaDeLLM-NER), a
novel approach to accelerate the inference of NER tasks for LLMs. PaDeLLM-NER
empowers the model with the capability to predict a single label-mention pair
within a single sequence, subsequently aggregating all sequences to generate
the final NER outcome. Specifically, in the training phase, we reconstruct the
instruction tuning tasks, enabling LLMs to predict the count of mentions for a
specific label and to identify the $n^{th}$ mention within the entire input
for that label (Figure 1). In the inference phase, LLMs first predict the
number of mentions for all labels, then predict all label-mention pairs in
parallel (Figure 2). Finally, results from all sequences are aggregated and
duplicate mentions across labels are eliminated based on prediction
probability. This approach results in a more efficient inference method for
NER tasks, producing shorter output strings and enabling parallel decoding
label-mention pairs in batches.

Comprehensive experiments have been conducted, demonstrating that PaDeLLM-NER
effectively reduces the number of tokens produced in each sequence, thereby
decreasing inference latency. Additionally, it maintains or even enhances
prediction quality in both flat and nested NER for English and Chinese
languages, compared to existing methods in the literature. To conclude, our
contributions are as follows:

  * •

We present PaDeLLM-NER, a novel approach tailored for NER using LLMs. This
approach can predict all label-mention pairs in parallel, effectively reducing
inference latency.

  * •

Extensive experiments have been conducted, revealing that PaDeLLM-NER
significantly improves inference efficiency. By completely decoupling the
generation of label-mention pairs, the average sequence length is reduced to
around 13% of that produced by conventional autoregressive methods.
Correspondingly, the inference speed is 1.76 to 10.22 times faster than these
previous approaches.

  * •

Comprehensive experiments demonstrate that, in addition to its enhanced
prediction speed, PaDeLLM-NER also maintains or surpasses the prediction
quality of conventional autoregressive methods, on par with state-of-the-art
performance on many NER datasets.

To the best of our knowledge, our techinique stands as a pioneering approach
in accelerating NER inference in LLMs by parallel decoding all label-mention
pairs. This unique characteristic makes it complementary to other inference
acceleration methods such as LLM.int8() Dettmers et al. (2022) and speculative
sampling Chen et al. (2023a); Leviathan et al. (2023). Thus, it can be
efficiently integrated with these methods.***Code is available at URL masked
for anonymous review.

##  2 Related Work

###  2.1 Generative Models for NER

Before the era of LLMs, most research approached NER as a sequence labeling
task, where each token is assigned a pre-defined tag (e.g., BIO scheme). In
this line of work, usually pre-trained transformer-based language models Ma et
al. (2020); Liu et al. (2021) is combined with a tailored prediction head to
perform a token-level classification, followed by the extraction of identified
tokens.

Encouraged by the success of unifying multiple NLP tasks into a single seq2seq
paradigm Brown et al. (2020); Lester et al. (2021), especially with the
evolution of LLMs Raffel et al. (2020b); Achiam et al. (2023); Touvron et al.
(2023a); Yang et al. (2023a), the trend of applying seq2seq models to NER
tasks is gaining momentum Xu et al. (2023), with both inputs and outputs being
represented as sequences of text Paolini et al. (2020); Lu et al. (2022); Das
et al. (2023); Lu et al. (2023a); Wang et al. (2023c). Recently, the focus of
work on NER using LLMs has shifted towards zero-shot Xie et al. (2023); Sainz
et al. (2023) or few-shot learning Ashok and Lipton (2023); Chen et al.
(2023b); Das et al. (2023); Wang et al. (2023b), utilizing in-context learning
Chen et al. (2023b); Wang et al. (2023b), self-consistency Wang et al.
(2022b); Xie et al. (2023) or learning programming Friedman et al. (2023);
Sainz et al. (2023).

Unlike previous studies emphasizing few-shot performance with training-free
prompt learning Liu et al. (2023c); Lu et al. (2023b), our work focus on a
fully supervised setting. More importantly, our primary objective is to speed
up NER inference.

###  2.2 Inference Speedup in LLMs

Modern LLMs employ a sequential decoding strategy for token generation, which
poses a significant challenge in terms of parallelization, especially as model
size and sequence length increase Ning et al. (2023). There is plenty of work
in the literature to address this challenge Wang et al. (2021); Frantar et al.
(2023); Santilli et al. (2023); Xiao et al. (2023). One line of work falls
into training-free category such as introducing extra modules for speculative
sampling Chen et al. (2023a); Leviathan et al. (2023). Another approaches
explore modifying model architecture to accelerate inference, such as exiting
at earlier layer Elbayad et al. (2019); Schuster et al. (2022), or designing
entirely different training and inference mechanisms Lan et al. (2023); Yang
et al. (2023b); Zhang et al. (2023). Different from previous works, we focus
on exploring the inference speedup in LLMs with a focus on the NER task
without the change of model architecture or modules.

##  3 Method

Figure 1: PaDeLLM-NER training paradigm: texts within frames of the same color
represents one training example, where texts inside the solid-line frame are
the input, and those inside the dashed-line frame are the output. Italic texts
are prompt templates. The “entity type” signifies the label being predicted.
The “¡num¿” indicates count of mentions for that label, and “¡mention n¿”
refers to the $n^{th}$ mention of a label in the input.  Figure 2: PaDeLLM-NER
inference paradigm: texts enclosed in frames with identical colors indicate
sequences of the same label. Specifically, the texts within solid-lined frames
represent the added templates, while those within dashed-lined frames denote
the prediction. In Step 1, the model predicts the number of mentions for all
labels while in Step 2, it predicts the mentions. By aggregating mentions and
labels from all sequences, the final NER results are obtained. Duplicate
mentions appearing in different labels are resolved using prediction
probabilities.

In this section, we delve into the details of PaDeLLM-NER. First, we focus on
reframing the instruction tuning tasks as outlined in Section 3.1. Second, we
explore the two-step inference process, detailed in Section 3.2. Finally, we
discuss the aggregation of results and the technique for eliminating duplicate
mentions across labels, which is elaborated in Section 3.3. An illustration of
PaDeLLM-NER is shown in Figure 1 and Figure 2.

###  3.1 Reframing of Instruction Tuning

Illustration of the reframing is presented in Figure 1. As an example, we use
a case from the CoNLL2003 dataset including four labels: person (PER),
miscellaneous (MISC), location (LOC), and organization (ORG). The specifics of
the input text and the corresponding ground truth are provided in the second
row of Table 1.

During reformulation, a single unstructured text containing all label-mention
pairs is split into several sequences. Each new sequence’s output includes the
count of mentions for a specified label (denoted as “entity type”), followed
by the $n^{th}$ mention of that label (denoted as “¡mention n¿”). Note that
the count of mentions and their respective indices are represented using
corresponding digit tokens from the LLM’s vocabulary. Specifically, if there
are no mentions, the model is trained to immediately predict the “¡eos¿”
token, bypassing the need to predict mentions.

Therefore, in this example, one original training data is transformed into
five new training data entries. These include two for predicting “LOC” (with 2
mentions), one for predicting “MISC” (with 1 mention), one for predicting
“PER” (with 1 mention), and one for predicting “ORG” (with 0 mentions,
directly predicting “¡eos¿”). Moreover, the number of mentions for each label
and the text corresponding to each mention index can be easily obtained from
the original ground truth, meaning that the number of new examples depends on
the ground truth of that particular example.

With the newly reformulated training examples, we then apply the standard
instruction tuning procedure. The model takes a sequence of text
$t_{1},t_{2},\ldots,t_{T}$ consisting of input unstructured text and output
structured label-mention pair. The optimization objective is cross-entropy
loss $\mathcal{L}$ which can be defined as follows:

$\mathcal{L}=-\frac{1}{T}\sum_{i=1}^{T}\log P\left(t_{i}\mid
t_{1},t_{2},\ldots,t_{i-1}\right)$ (1)

where $P\left(t_{i}\mid t_{1},t_{2},\ldots,t_{i-1}\right)$ represents the
probability of $i^{th}$ token $t_{i}$ given the sequence of preceding tokens
$t_{1},t_{2},\ldots,t_{i-1}$, as predicted by the model. Note that loss
calculation begins from the number of mention tokens (i.e., texts enclosed by
dashed-line frames). Theoretically, loss from text spans such as “¡mention n¿”
could be ignored during this calculation, since they simply prompt the
mention’s order, which does not necessarily need to be generated by the model.
However, our ablation studies show that ignoring these texts has negligible
impact on model performance, a point further discussed in Section 4.3.
Therefore, we adhere to the standard instruction tuning procedure.

This reformulation allows the model to focus one label-mention pair at a time,
shortening the generated length per sequence. More details are shown in
Appendix B.

###  3.2 Inference of Label-Mention Pairs

Given a trained LLM, we propose a two-step inference approach: firstly, to
predict the number of mentions for a specific label based on the prompt; and
secondly, given the label and provided index to precisely identify the
corresponding mention.

Figure 2 shows the overview of PaDeLLM-NER inference. In Step 1, the model
predicts the total number of mentions for each label in the input, based on
the label prompt. A separate token “\n” signals the completion of this count
prediction. If no mentions of the given label exist, the model generates an
“¡eos¿” token, skipping Step 2 for that label. In Step 2, following adding the
predicted mention count to the input, mention indexes templates are appended.
Formally, if the predicted number of mention is $m$, then “¡mention n¿”,
indicating the $n^{th}$ mention of the specified label, is appended for each
$n$ within the set $\\{1,2,3,\ldots,m\\}$ and $n$ is an integer. Subsequently,
the corresponding mention is generated by the model conditioned on preceding
tokens. Note that the decoding of all label-mention pairs occurs in parallel,
allowing for their simultaneous generation.

In practice, if there are sufficient GPU resources, the inference for the
number of mentions for each label, as well as the subsequent inference for the
mention text spans, can be allocating on separate GPUs. If GPU resources are
limited, the inference can also be deployed on a single GPU using batch
inference, facilitating parallel decoding. Using Figure 2 as an example, in
Step 1, the batch size is four, as there are four labels in the dataset. In
Step 2, the batch size is five, reflecting the five label-mention pairs
determined in Step 1 (i.e., 1 in “PER”, 2 in “MISC”, 2 in “LOC”). This
parallel decoding strategy is effective in reducing inference latency,
especially in scenarios where inputs are received in a streaming manner.

###  3.3 Removal of Duplicate Mentions

Unlike autoregressive decoding, where subsequent label-mention pairs can
attend preceding ones, PaDeLLM-NER generates each label-mention pair
independently. This inference strategy means that the model might generate
mentions erroneously repeated in multiple labels. As exemplified in Figure 2,
the model correctly predicts the first mention of “LOC” as “Italy”, but it
also incorrectly predicts the second mention of “MISC” as “Italy”.

To address the issue of duplicate mentions, we suggest employing prediction
probability to remove repeated mentions. Specifically, we calculate the
prediction probability for each instance of the mention. This is done using
the formula: $P=\prod_{i=b}^{e}P(t_{i}|t_{1},t_{2},\ldots,t_{i-1})$ where $b$
represents the starting token index of the mention text, and $e$ denotes the
ending token index. Then, for a mention that appears in multiple labels, the
mention instance with the highest probability will be preserved. As
illustrated in Figure2, “Italy” is categorized as “MISC” with only a 0.61
probability, which is lower than that for “LOC”, resulting in its removal. In
practice, the probability of each token can be calculated concurrently with
token generation. Consequently, this method enables an efficient and accurate
identification of duplicate mentions without incurring additional costs. The
effectiveness of this de-duplication approach is further explored in Section
4.3.

##  4 Experiments

In this section, we showcase the effectiveness of PaDeLLM-NER in terms of
prediction quality and inference acceleration through experiments.

###  4.1 Setup

#### Datasets

We evaluate our method on English and Chinese NER datasets. English datasets
include the general domain flat NER CoNLL2003 Tjong Kim Sang and De Meulder
(2003), the nested NER ACE2005 Kirkpatrick (2010), and the biomedical nested
NER GENIA Ohta et al. (2002). Chinese datasets include four commonly used
general domain flat NER benchmarks Resume Zhang and Yang (2018), Weibo Peng
and Dredze (2015), MSRA Levow (2006) and Ontonotes 4.0 Pradhan et al. (2013)
and two vertical industrial domain flat NER datasets YouKu Jie et al. (2019)
and Ecommerce Ding et al. (2019). The statistics of all datasets are shown in
Appendix A.

#### Training setup

We employ pre-trained version of Llama2-7b Touvron et al.
(2023b)***https://huggingface.co/meta-llama/Llama-2-7b and Baichuan2-7b Yang
et al. (2023a)***https://huggingface.co/baichuan-inc/Baichuan2-7B-Base as base
models for English and Chinese study respectively. Additional implementation
details are in Appendix C.

#### Inference setup

For all generative models, we use greedy search with a beam size of $1$, a
maximum of $512$ new tokens, and a temperature of $1.0$. As described in
Section 3.2, for PaDeLLM-NER, we adopt two inference settings: (1) each
example is inferred on multiple GPUs to implement parallel decoding (i.e.,
each sequence is assigned on one GPU), termed as PaDeLLMMulti; and (2) each
example is inferred on a single GPU, employing batch decoding for parallel
decoding, termed as PaDeLLMBatch. Note that for PaDeLLMMulti, we sequentially
predict each sequence of one example to simulate parallel decoding on multiple
GPUs.

#### Baselines

As the primary focus of this work is on reducing inference latency in NER
tasks using LLMs, we compare our method, PaDeLLM-NER, with traditional
autoregressive approaches. As mentioned in Section 1, the main points of
comparison are autoregressive structured output formats used in Paolini et al.
(2020); Das et al. (2023) and Lu et al. (2022, 2023a); Wang et al. (2023c),
referred to respectively as AutoRegAug and AutoRegStruct, as these are the
approaches very close to our system. We reimplemented these methods for both
English and Chinese datasets, utilizing the same pre-trained LLMs as in
PaDeLLM-NER. More details on the reimplementation are provided in Appendix C.
Besides, we compare our approach with other recent state-of-the-art methods,
including BINDER Zhang et al. (2022a), Gollie Sainz et al. (2023), and
DeepStruct Wang et al. (2022a) for English benchmarks, as well as W2NER Li et
al. (2022), NEZHA-BC Zhang et al. (2022b), and SSCNN Zhang and Lu (2023) for
Chinese benchmarks, to show PaDeLLM-NER’s efficacy in prediction quality.

#### Evaluation

Our evaluation encompasses two dimensions: prediction quality and acceleration
of NER inference. For assessing prediction quality, in line with Lu et al.
(2022); Wang et al. (2023c), we employ the micro F-score.

Regarding inference acceleration, as per Ning et al. (2023), we evaluate using
latency (in milliseconds). We record the latency using the following method:
start = time.time(); model.generate(); latency = time.time() - start. In
PaDeLLM-NER, we add the latency of mention counting and label-mention pair
generation as the latency of each sequence. The final latency for the example
is determined by the highest latency across sequences, as the user can only
obtain the result of an example when the slowest sequence is generated. We
conduct experiments three times and use the average result to alleviate the
effect of randomness. We also report the average sequence length (tokenized)
to clearly demonstrate the extent of sequence length reduction in Appendix D.
Evaluations of all models were performed on the same NVIDIA A100 GPU.

###  4.2 Main Results

| English Dataset | Chinese Dataset |
---|---|---|---
AutoReg | CoNLL03 | ACE05 | GENIA | Weibo | MSRA | Onto4 | Resume | Youku | Ecom | Mean
AutoRegAug | 992.70 | 944.90 | 1,515.35 | 1,276.32 | 812.78 | 1,009.68 | 982.39 | 579.99 | 845.42 | 995.50
AutoRegStruct | 753.36 | 1,293.87 | 1,266.31 | 1,630.62 | 609.34 | 783.28 | 1,462.56 | 598.59 | 738.20 | 1,015.12
Ours |  |  |  |  |  |  |  |  |  |
PaDeLLMMulti | 229.74 | 255.53 | 316.90 | 159.57 | 143.47 | 171.67 | 238.27 | 203.63 | 293.40 | 223.57
PaDeLLMBatch |  333.89 |  498.50 |  616.01 |  344.75 |  204.24 |  288.43 |  459.20 |  241.25 |  419.40 |  378.40

Table 2: Comparison of inference latency (in milliseconds) between PaDeLLM-NER
and baseline methods. Underscored font is the second-best method, while a bold
font is the best method, also applied to subsequent tables.

(a) ARAug vs. PDLMMulti

(b) ARStruct vs. PDLMMulti

(c) ARAug vs. PDLMBatch

(d) ARStruct vs. PDLMBatch

Figure 3: Speedup of PaDeLLM-NER compared to Autoregressive methods.

| English Dataset | Chinese Dataset |
---|---|---|---
AutoReg | CoNLL03 | ACE05 | GENIA | Weibo | MSRA | Onto4 | Resume | Youku | Ecom | Mean
AutoRegAug | 93.08 |  83.04 | 70.16 |  59.04 | 95.56 | 79.20 | 95.80 | 86.07 | 76.02 | 81.99
AutoRegStruct | 91.87 | 82.99 | 77.90 | 56.07 | 90.92 | 80.97 |  95.74 |  86.85 |  81.57 |  82.76
Ours |  |  |  |  |  |  |  |  |  |
PaDeLLM-NER |  92.52 | 85.02 | 77.66 | 67.36 |  95.03 |  80.81 | 94.98 | 87.91 | 81.85 | 84.79

Table 3: Comparison of prediction quality between PaDeLLM-NER and baseline
methods.

#### Latency evaluation

We investigate how PaDeLLM-NER reduces the end-to-end latency compared to
baseline methods. Table 2 presents the average latency for each method across
all datasets. First, it’s clear that both PaDeLLMMulti and PaDeLLMBatch
significantly reduce inference latency when compared to baseline methods, as
highlighted by the substantial reduction in mean latency. For example, the
mean latency reduction achieved between PaDeLLMMulti and AutoRegStruct stands
at an impressive $791.55$ ms, underscoring the significant improvement.

To more intuitively quantify the latency reduction of PaDeLLM-NER, we break
down its speedup across different datasets in comparison to baseline methods
in Figure 3. The speedup is computed by dividing the latency of baselines by
the latency of PaDeLLM-NER. We can observe that PaDeLLM-NER consistently show
a speedup over baseline methods across all datasets. The highest speedup is
observed in the Weibo dataset when comparing AutoRegStruct vs. PaDeLLMMulti,
with a speedup of $10.22$x. When we narrow our focus to the comparison between
PaDeLLMBatch and the baseline methods, considering these methods utilize a
single GPU for inference, we can still observe substantial speedup ranging
from $1.76$x to $4.73$x. The speedup factor varies across different datasets,
suggesting that the efficiency gains of PaDeLLM-NER may be influenced by the
characteristics of each dataset. Interestingly, we can observe that the
PaDeLLMBatch is slower than PaDeLLMMulti ($378.40$ ms vs. $223.57$ ms), more
analysis about this is shown in Section 5.

Overall, the Table 2 and Figure 3 suggest that PaDeLLM-NER significantly
reduces latency compared to autoregressive methods, though the extent of this
reduction varies by dataset and the specific baseline method it’s compared to.

#### Prediction quality evaluation

Table 3 shows the micro F-score of PaDeLLM-NER and other baseline methods.
It’s noteworthy that the micro F-scores for PaDeLLMMulti and PaDeLLMBatch are
identical. Overall, PaDeLLM-NER outperforms baselines with the highest mean
F-score of $84.79$ and emerging as the best-performing method in 4 out of the
9 datasets. It excels particularly in the Weibo, Youku and ACE2005 datasets
and is competitive in others. We believe that the improved prediction could be
attributed to shorter sequences, which lessen the challenge of long-range
dependencies and enhance prediction quality. More analysis is discussed in
Appendix E.

In summary, the results presented in Tables 2 and 3, demonstrate that our
approach not only maintains superior prediction quality but also significantly
reduces inference latency.

#### Comparison to state-of-the-art methods

In Tables 4 and 5, we compare the micro F-scores of PaDeLLM-NER with other
recent state-of-the-art methods. Our findings indicate that PaDeLLM-NER
performs comparably to these methods across most datasets except for Weibo.
Notably, in the Youku dataset, PaDeLLM-NER outperforms the previously best-
performing method by an improvement of 1.81%. Note that the primary focus of
our work is on accelerating prediction latency using LLMs compared to baseline
autoregressive methods, rather than achieving state-of-the-art results.

State-of-the-art Method | CoNLL03 | ACE05 | GENIA
---|---|---|---
BINDER Zhang et al. (2022a) | 93.33 | 89.50 | 80.50
Gollie Sainz et al. (2023) | 93.10 | 89.60 | -
DeepStruct Wang et al. (2022a) | 93.00 | 86.90 | 80.80
Our Method |  |  |
PaDeLLM-NER | 92.52 | 85.02 | 77.66

Table 4: Comparing PaDeLLM-NER Performance with recent state-of-the-art methods on English datasets. State-of-the-art Method | Weibo | MSRA | Onto4 | Resume | Youku | Ecom
---|---|---|---|---|---|---
NEZHA-BC Zhang et al. (2022b) | - | - | - | - | - | 82.98
SSCNN Zhang and Lu (2023) | 71.81 | - | 82.99 | 96.40 | 86.10 | 81.80
W2NER Li et al. (2022) | 72.32 | 96.10 | 83.08 | 96.65 | - | -
Our Method |  |  |  |  |  |
PaDeLLM-NER | 67.36 | 95.03* | 80.81 | 94.98 | 87.91 | 81.85
Table 5: Comparing PaDeLLM-NER Performance with recent state-of-the-art
methods on Chinese datasets. “*” indicates that results are not directly
comparable.

###  4.3 Ablation study

Variant | CoNLL03 | ACE05 | GENIA | Mean
---|---|---|---|---
PaDeLLM-NER | 92.52 | 85.02 | 77.66 | 85.06
\+ Loss ignoring | 92.01 | 85.18 | 73.47 | 83.55
\- De-duplication |  92.44 |  84.80 |  77.54 |  84.92
\+ De-duplicationReverse | 92.38 | 84.44 | 77.38 | 84.73

Table 6: Ablations on ignoring loss and de-duplication.

In this section, we set out to investigate the effects of the different
aspects of PaDeLLM-NER.

#### Ignoring text spans in loss

As discussed in Section 3.1, during training, it is permissible to overlook
the loss of text span “¡mention n¿”, as the model does not need to generate
this specific text, which is appended during inference. However, as shown in
Table 6 illustrate, omitting these texts has minimal impact on prediction
quality.

One possible explanation is that during training, the more significant
challenge for LLMs lies in predicting the appropriate mention texts, rather
than their format. As the model can readily learns to correctly position the
format “¡mention n¿”, this aspect contributes minimally to the loss
computation in training. In this case, computing the loss for all text is
almost equivalent to ’neglecting’ the computation of loss for “¡mention n¿”.

#### De-duplication

To demonstrate the effectiveness of the de-duplication technique, we
established two configurations as detailed in Table 6. The -De-duplication
denotes the pipeline operating without the de-duplication technique; +De-
duplicationReverse indicates the pipeline that removes mentions with the
highest probability, opposite to the original de-duplication technique.

Theoretically, PaDeLLM-NER should be the top-performing method, as its de-
duplication eliminates noisy mentions, enhancing precision. Following closely
is the -De-duplication, allows duplicate mentions to persist. +De-
duplicationReverse ranks lowest since it removes correct mentions and retains
incorrect ones, lowering recall and precision simultaneously. As shown in
Table 6, the results consistently align with our expectations, thereby
verifying the effectiveness of the de-duplication process. Moreover, the
difference among these variants is subtle, which can be attributed to the rare
cases where duplicate mentions exist. This further highlights the robustness
of proposed method.

##  5 Speedup Analysis

One concern noted is that batch inference does not speed up as much as
inference distributed across multiple GPUs. This observation is consistent
with our expectations and supported by Chen et al. (2023c) who found that
batch inference in LLMs tends to be slower than single sequence inference
under identical conditions, likely due to limitations in GPU memory bandwidth
Cai et al. (2024).

Transitioning from these performance considerations, it’s noteworthy that
PaDeLLM-NER is self-contained and can be seamlessly integrated with various
generative architectures, including well-established decoder-only models
Raffel et al. (2020a); Muennighoff et al. (2022); Touvron et al. (2023a, b);
Bai et al. (2023); Yang et al. (2023a) and recent innovations like RWKV Peng
et al. (2023), as well as multi-modal LLMs Feng et al. (2023); Liu et al.
(2023b, a) for tasks like Key Information Extraction tasks Huang et al.
(2019), all without needing architectural changes or additional data/modules.
Also, it could be incorporated with off-the-shelf LLMs such as ChatGPT Achiam
et al. (2023) and Claude-2***https://www.anthropic.com/news/claude-2 through
prompt engineering without the need for further training, an aspect we plan to
explore in future research.

##  6 Data Contamination Concerns

Since we are using LLMs as our foundational models, trained on extensive
datasets from various online sources Touvron et al. (2023b); Yang et al.
(2023a), there is a chance that the models may have encountered parts of our
evaluation sets during their pre-training phase, albeit unintentionally. This
could potentially affect our experimental results. However, the primary focus
of our experiments is the comparison of our proposed method with baseline
methods. Given that these methods employ the same LLM as the base model, data
contamination is unlikely to significantly impact the results.

##  7 Conclusion

In this work, we introduce Parallel Decoding in LLM for NER (PaDeLLM-NER), a
parallel decoding framework in LLMs for efficient NER. To achieve this, we
recast autoregressive prediction of all label-mention pairs of traditional NER
tasks into a two-step prediction: (1) predicting the number of mentions for a
specific label, and (2) identifying the $n^{th}$ mention for that label. This
recast allows the model to parallel decode all label-mention pairs in batches.
Extensive experimental results show that the proposed method can dramatically
reduces inference time, achieving inference speedup ranging from 1.76 to 10.22
times, without compromising prediction quality. Lastly, extensive ablation
studies are performed to clarify the design choices of PaDeLLM-NER.

##  8 Limitations

One clear disadvantage of PaDeLLM-NER is the multiplication of training
examples from one to $m*n$, where $m$ is the label count and $n$ the mention
count. Despite this, given that low latency is a major bottleneck in LLMs,
trading longer training for lower latency is justifiable. Also, given the
impressive generalization ability of LLMs, we believe that this method can be
smoothly adapted to few-shot scenarios requiring less computation resources,
which will be explored in future work.

Additionally, accurately counting the number of mentions remains a challenge
for LLMs as discussed in Appendix E. This issue could be alleviated by
implementing a specialized counting model dedicated to this task Liu and Low
(2023).

Finally, there are several instances of re-computation within the pipeline
that can be optimized. Specifically, input texts are encoded multiple times
throughout the process. During batch decoding, certain sequences may encounter
the “¡eos¿” token earlier, but due to the nature of batch inference, these
sequences continue to predict. We plan to improve this in the future by
implementing enhancements like KV cache reuse and batch inference with an
early quit mechanism, among other strategies.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * Ashok and Lipton (2023) Dhananjay Ashok and Zachary C Lipton. 2023.  Promptner: Prompting for named entity recognition.  _arXiv preprint arXiv:2305.15444_.
  * Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023.  Qwen technical report.  _arXiv preprint arXiv:2309.16609_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Cai et al. (2024) Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. 2024.  Medusa: Simple llm inference acceleration framework with multiple decoding heads.  _arXiv preprint arXiv:2401.10774_.
  * Chen et al. (2023a) Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023a.  Accelerating large language model decoding with speculative sampling.  _arXiv preprint arXiv:2302.01318_.
  * Chen et al. (2023b) Jiawei Chen, Yaojie Lu, Hongyu Lin, Jie Lou, Wei Jia, Dai Dai, Hua Wu, Boxi Cao, Xianpei Han, and Le Sun. 2023b.  Learning in-context learning for named entity recognition.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 13661–13675, Toronto, Canada. Association for Computational Linguistics.
  * Chen et al. (2023c) Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, and Arvind Krishnamurthy. 2023c.  Punica: Multi-tenant lora serving.  _arXiv preprint arXiv:2310.18547_.
  * Das et al. (2023) Sarkar Snigdha Sarathi Das, Haoran Zhang, Peng Shi, Wenpeng Yin, and Rui Zhang. 2023.  Unified low-resource sequence labeling by sample-aware dynamic sparse finetuning.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 6998–7010, Singapore. Association for Computational Linguistics.
  * Dettmers et al. (2022) Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022.  GPT3.int8(): 8-bit matrix multiplication for transformers at scale.  In _Advances in Neural Information Processing Systems_.
  * Ding et al. (2019) Ruixue Ding, Pengjun Xie, Xiaoyan Zhang, Wei Lu, Linlin Li, and Luo Si. 2019.  A neural multi-digraph model for chinese ner with gazetteers.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1462–1467.
  * Elbayad et al. (2019) Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. 2019.  Depth-adaptive transformer.  In _International Conference on Learning Representations_.
  * Feng et al. (2023) Hao Feng, Zijian Wang, Jingqun Tang, Jinghui Lu, Wengang Zhou, Houqiang Li, and Can Huang. 2023.  Unidoc: A universal large multimodal model for simultaneous text detection, recognition, spotting and understanding.  _arXiv preprint arXiv:2308.11592_.
  * Frantar et al. (2023) Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023.  OPTQ: Accurate quantization for generative pre-trained transformers.  In _The Eleventh International Conference on Learning Representations_.
  * Friedman et al. (2023) Dan Friedman, Alexander Wettig, and Danqi Chen. 2023.  Learning transformer programs.  _arXiv preprint arXiv:2306.01128_.
  * Gao et al. (2023) Fan Gao, Hang Jiang, Moritz Blum, Jinghui Lu, Yuang Jiang, and Irene Li. 2023.  Large language models on wikipedia-style survey generation: an evaluation in nlp concepts.  _arXiv preprint arXiv:2308.10410_.
  * Huang et al. (2019) Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and CV Jawahar. 2019.  Icdar2019 competition on scanned receipt ocr and information extraction.  In _2019 International Conference on Document Analysis and Recognition (ICDAR)_ , pages 1516–1520. IEEE.
  * Jie et al. (2019) Zhanming Jie, Pengjun Xie, Wei Lu, Ruixue Ding, and Linlin Li. 2019.  Better modeling of incomplete annotations for named entity recognition.  In _Proceedings of NAACL_.
  * Kirkpatrick (2010) Andy Kirkpatrick. 2010.  Researching english as a lingua franca in asia: The asian corpus of english (ace) project.  _Asian Englishes_ , 13(1):4–18.
  * Lan et al. (2023) Tian Lan, Deng Cai, Yan Wang, Heyan Huang, and Xian-Ling Mao. 2023.  Copy is all you need.  In _The Eleventh International Conference on Learning Representations_.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3045–3059.
  * Leviathan et al. (2023) Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023.  Fast inference from transformers via speculative decoding.  In _International Conference on Machine Learning_ , pages 19274–19286. PMLR.
  * Levow (2006) Gina-Anne Levow. 2006.  The third international Chinese language processing bakeoff: Word segmentation and named entity recognition.  In _Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing_ , pages 108–117, Sydney, Australia. Association for Computational Linguistics.
  * Li et al. (2022) Jingye Li, Hao Fei, Jiang Liu, Shengqiong Wu, Meishan Zhang, Chong Teng, Donghong Ji, and Fei Li. 2022.  Unified named entity recognition as word-word relation classification.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 10965–10973.
  * Liu et al. (2023a) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a.  Improved baselines with visual instruction tuning.
  * Liu et al. (2023b) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b.  Visual instruction tuning.  In _NeurIPS_.
  * Liu and Low (2023) Tiedong Liu and Bryan Kian Hsiang Low. 2023.  Goat: Fine-tuned llama outperforms gpt-4 on arithmetic tasks.  _arXiv preprint arXiv:2305.14201_.
  * Liu et al. (2021) Wei Liu, Xiyan Fu, Yue Zhang, and Wenming Xiao. 2021.  Lexicon enhanced Chinese sequence labeling using BERT adapter.  In _Proceedings of the ACL 2021 and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5847–5858, Online.
  * Liu et al. (2023c) Xuejing Liu, Wei Tang, Jinghui Lu, Rui Zhao, Zhaojun Guo, and Fei Tan. 2023c.  Deeply coupled cross-modal prompt learning.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 7957–7970, Toronto, Canada. Association for Computational Linguistics.
  * Liu et al. (2023d) Xuejing Liu, Wei Tang, Xinzhe Ni, Jinghui Lu, Rui Zhao, Zechao Li, and Fei Tan. 2023d.  What large language models bring to text-rich vqa?  _arXiv preprint arXiv:2311.07306_.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  SGDR: Stochastic gradient descent with warm restarts.  In _International Conference on Learning Representations_.
  * Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter. 2018.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Lu et al. (2023a) Jinghui Lu, Rui Zhao, Brian Mac Namee, and Fei Tan. 2023a.  Punifiedner: a prompting-based unified ner system for diverse datasets.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, pages 13327–13335.
  * Lu et al. (2023b) Jinghui Lu, Dongsheng Zhu, Weidong Han, Rui Zhao, Brian Mac Namee, and Fei Tan. 2023b.  What makes pre-trained language models better zero-shot learners? In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2288–2303, Toronto, Canada. Association for Computational Linguistics.
  * Lu et al. (2022) Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2022.  Unified structure generation for universal information extraction.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5755–5772.
  * Ma et al. (2020) Ruotian Ma, Minlong Peng, Qi Zhang, Zhongyu Wei, and Xuanjing Huang. 2020.  Simplify the usage of lexicon in Chinese NER.  In _Proceedings of the ACL 2020_ , pages 5951–5960, Online.
  * Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022.  Rethinking the role of demonstrations: What makes in-context learning work?  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11048–11064.
  * Muennighoff et al. (2022) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022.  Crosslingual generalization through multitask finetuning.  _arXiv preprint arXiv:2211.01786_.
  * Ning et al. (2023) Xuefei Ning, Zinan Lin, Zixuan Zhou, Huazhong Yang, and Yu Wang. 2023.  Skeleton-of-thought: Large language models can do parallel decoding.  _arXiv preprint arXiv:2307.15337_.
  * Ohta et al. (2002) Tomoko Ohta, Yuka Tateisi, Jin-Dong Kim, Hideki Mima, and Junichi Tsujii. 2002.  The genia corpus: An annotated research abstract corpus in molecular biology domain.  In _Proceedings of the human language technology conference_ , pages 73–77. Citeseer.
  * Paolini et al. (2020) Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, RISHITA ANUBHAI, Cicero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. 2020.  Structured prediction as translation between augmented natural languages.  In _International Conference on Learning Representations_.
  * Peng et al. (2023) Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, et al. 2023.  Rwkv: Reinventing rnns for the transformer era.  _arXiv preprint arXiv:2305.13048_.
  * Peng and Dredze (2015) Nanyun Peng and Mark Dredze. 2015.  Named entity recognition for chinese social media with jointly trained embeddings.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_.
  * Pradhan et al. (2013) Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013.  Towards robust linguistic analysis using OntoNotes.  In _Proceedings of the Seventeenth Conference on Computational Natural Language Learning_ , pages 143–152, Sofia, Bulgaria. Association for Computational Linguistics.
  * Raffel et al. (2020a) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020a.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Raffel et al. (2020b) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020b.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551.
  * Sainz et al. (2023) Oscar Sainz, Iker García-Ferrero, Rodrigo Agerri, Oier Lopez de Lacalle, German Rigau, and Eneko Agirre. 2023.  Gollie: Annotation guidelines improve zero-shot information-extraction.  _arXiv preprint arXiv:2310.03668_.
  * Santilli et al. (2023) Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Riccardo Marin, and Emanuele Rodola. 2023.  Accelerating transformer inference for translation via parallel decoding.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 12336–12355, Toronto, Canada. Association for Computational Linguistics.
  * Schuster et al. (2022) Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. 2022.  Confident adaptive language modeling.  _Advances in Neural Information Processing Systems_ , 35:17456–17472.
  * Tjong Kim Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the conll-2003 shared task.  In _Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003 -_.
  * Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wang et al. (2023a) Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2023a.  Towards understanding chain-of-thought prompting: An empirical study of what matters.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2717–2739, Toronto, Canada. Association for Computational Linguistics.
  * Wang et al. (2022a) Chenguang Wang, Xiao Liu, Zui Chen, Haoyun Hong, Jie Tang, and Dawn Song. 2022a.  DeepStruct: Pretraining of language models for structure prediction.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 803–823, Dublin, Ireland. Association for Computational Linguistics.
  * Wang et al. (2021) Hanrui Wang, Zhekai Zhang, and Song Han. 2021.  Spatten: Efficient sparse attention architecture with cascade token and head pruning.  In _2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA)_ , pages 97–110. IEEE.
  * Wang et al. (2023b) Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023b.  Gpt-ner: Named entity recognition via large language models.  _arXiv preprint arXiv:2304.10428_.
  * Wang et al. (2023c) Xiao Wang, Weikang Zhou, Can Zu, Han Xia, Tianze Chen, Yuansen Zhang, Rui Zheng, Junjie Ye, Qi Zhang, Tao Gui, et al. 2023c.  Instructuie: Multi-task instruction tuning for unified information extraction.  _arXiv preprint arXiv:2304.08085_.
  * Wang et al. (2022b) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022b.  Self-consistency improves chain of thought reasoning in language models.  In _The Eleventh International Conference on Learning Representations_.
  * Xiao et al. (2023) Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023.  Smoothquant: Accurate and efficient post-training quantization for large language models.  In _International Conference on Machine Learning_ , pages 38087–38099. PMLR.
  * Xie et al. (2023) Tingyu Xie, Qi Li, Jian Zhang, Yan Zhang, Zuozhu Liu, and Hongwei Wang. 2023.  Empirical study of zero-shot ner with chatgpt.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 7935–7956.
  * Xu et al. (2023) Derong Xu, Wei Chen, Wenjun Peng, Chao Zhang, Tong Xu, Xiangyu Zhao, Xian Wu, Yefeng Zheng, and Enhong Chen. 2023.  Large language models for generative information extraction: A survey.  _arXiv preprint arXiv:2312.17617_.
  * Yang et al. (2023a) Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023a.  Baichuan 2: Open large-scale language models.  _arXiv preprint arXiv:2309.10305_.
  * Yang et al. (2023b) Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023b.  Inference with reference: Lossless acceleration of large language models.  _arXiv preprint arXiv:2304.04487_.
  * Zhang and Lu (2023) Miao Zhang and Ling Lu. 2023.  A local information perception enhancement–based method for chinese ner.  _Applied Sciences_ , 13(17):9948.
  * Zhang et al. (2022a) Sheng Zhang, Hao Cheng, Jianfeng Gao, and Hoifung Poon. 2022a.  Optimizing bi-encoder for named entity recognition via contrastive learning.  In _The Eleventh International Conference on Learning Representations_.
  * Zhang et al. (2022b) Xin Zhang, Yong Jiang, Xiaobin Wang, Xuming Hu, Yueheng Sun, Pengjun Xie, and Meishan Zhang. 2022b.  Domain-specific ner via retrieving correlated samples.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 2398–2404.
  * Zhang et al. (2023) Xinpeng Zhang, Ming Tan, Jingfan Zhang, and Wei Zhu. 2023.  Nag-ner: a unified non-autoregressive generation framework for various ner tasks.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track)_ , pages 676–686.
  * Zhang and Yang (2018) Yue Zhang and Jie Yang. 2018.  Chinese NER using lattice LSTM.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1554–1564, Melbourne, Australia. Association for Computational Linguistics.
  * Zhou et al. (2023) Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023.  LIMA: Less is more for alignment.  In _Thirty-seventh Conference on Neural Information Processing Systems_.

Variant | CoNLL03 | ACE05 | GENIA | Mean
---|---|---|---|---
PaDeLLM-NER |  92.52 |  85.02 |  77.66 |  85.06
\+ Model scale up to 13B | 93.02 | 84.37 | 78.84 | 85.45

Table 7: Ablations on model scaling up. Figure 4: Percentage of different
error types.

| English Dataset | Chinese Dataset |
---|---|---|---
AutoReg | CoNLL03 | ACE05 | GENIA | Weibo | MSRA | Onto4 | Resume | Youku | Ecom | Mean
AutoRegAug | 33.85 |  37.10 | 60.50 |  45.02 | 27.42 | 35.90 |  30.39 |  18.21 | 31.50 |  35.54
AutoRegStruct |  28.36 | 49.95 |  49.03 | 62.45 |  18.97 |  25.53 | 53.02 | 18.56 |  22.51 | 36.48
Ours |  |  |  |  |  |  |  |  |  |
PaDeLLM-NER | 6.54 | 8.29 | 10.05 | 2.19 | 2.23 | 2.68 | 4.87 | 3.66 | 3.27 | 4.86

Table 8: Comparison of the number of generated tokens per sequence by PaDeLLM-
NER with baseline methods.

##  Appendix A Dataset Statistics

We evaluate our framework on 3 English and 6 Chinese flat/nested NER datasets.
In Table 9, we present the detailed statistics. Note that while the statistics
of the development set are reported, our training process does not involve the
development set.

For the MSRA dataset, we excluded four outlier instances from the test set due
to their excessively high number of mentions, significantly deviating from
typical examples. These outliers not only posed challenges for model inference
but also risked distorting the evaluation metrics, potentially leading to an
inaccurate assessment of the model’s performance on representative data.

Also, we perform label mapping to convert ground truth from special tokens to
Chinese words following Lu et al. (2023a). Further details are provided in
Table 10.

Dataset | Sentence |  | Mention
---|---|---|---
#All | #Train | #Dev | #Test |  | #All | #Train | #Dev | #Test
CoNLL2003 | 20,744 | 14,041 | 3,250 | 3,453 |  | 35,089 | 23,499 | 5,942 | 5,648
ACE2005 | 9,210 | 7,194 | 969 | 1,047 |  | 30,634 | 24,441 | 3,200 | 2,993
GENIA | 18,546 | 15,023 | 1,669 | 1,854 |  | 56,015 | 46,142 | 4,367 | 5,506
Weibo | 1,890 | 1,350 | 270 | 270 |  | 2,701 | 1,894 | 389 | 418
MSRA* | 50,725 | 44,364 | - | 4,361 |  | 80,214 | 74,703 | - | 5,511
OntoNotes 4.0 | 24,371 | 15,724 | 4,301 | 4,346 |  | 28,006 | 13,372 | 6,950 | 7,684
Resume | 4,759 | 3,819 | 463 | 477 |  | 16,565 | 13,438 | 1,497 | 1,630
Youku | 10,002 | 8,001 | 1,000 | 1,001 |  | 15,905 | 12,754 | 1,581 | 1,570
Ecommerce | 4,987 | 3,989 | 500 | 498 |  | 15,216 | 12,109 | 1,540 | 1,567
Table 9: Dataset Statistics. “#” denotes the amount. For MSRA, we remove four outlier examples in test set. Dataset | #Entity | Entity
---|---|---
Weibo | 8 |  {“PER.NAM(Specific Name)”:“名称特指”, “PER.NOM(Generic Name)”:“名称代称”, “GPE.NAM(Specific Geo-Political Entity)”:“行政区特指”, “GPE.NOM(Generic Geo-Political Entity)”:“行政区代称”, “LOC.NAM(Specific Location)”:“地点特指”, “LOC.NOM(Generic Location)”:“地点代称”, “ORG.NAM(Specific Organization)”:“组织特指”, “ORG.NOM(Generic Organization)”:“组织代称” }
MSRA | 3 |  {“LOC”:“地点’, “PER”:“名称”, “ORG”:“组织”}
OntoNotes 4.0 | 4 |  {“GPE”:“地缘”, “LOC”:“地点”, “PER”:“名称”, “ORG”:“组织”}
Resume | 8 |  {“NAME”:“名称”, “CONT(Nationality)”:“国籍”, “RACE”:“民族”, “TITLE”:“职位”, “EDU”:“学历”, “ORG”:“公司”, “PRO(Profession)”:“专业”, “LOC(Place of Birth)”:“籍贯”}
Youku | 3 |  {“TELEVISION”:“电视剧”, “PER(Celebrity)”:“明星”, “MISC”:“其他”}
Ecommerce | 2 |  {“HP(brand)”:“品牌”, “HC(commodity)”:“商品”}
Table 10: Entity tag of each dataset and the conversion from tag used in
dataset to corresponding Chinese natural language. For some tags that are hard
to understand, we provide their meaning in brackets. “#” denotes the amount of
entity types.

##  Appendix B Reformulation Examples

Two compete reformulated examples are presented in Table 11 for English and
Chinese, respectively.

Language | Input | Output
---|---|---
English |  text: But Fischler agreed to review his proposal after the EU ’s standing veterinary committee , mational animal health officials , questioned if such action was justified as there was only a slight risk to human health . entity type: PER ¡num¿ |  1 ¡mention 1¿Fischler
Chinese |  文本(text): 公报最后说，墨西哥政府认为，贩毒以及洗钱等与毒品有关的活动是威胁到国家主权和安全的一个全球性问题。(The communique concluded by stating that the Mexican government considers drug trafficking and related activities such as money laundering to be a global issue that threatens national sovereignty and security.) 指定NER标签(entity type): 地点(LOC) ¡数量¿(¡num¿) |  1 ¡第1文段¿(¡mention 1¿) 墨西哥(Mexican)
Table 11: Reformulated examples for English and Chinese dataset, respectively.
We provide translations to facilitate understanding. The examples come from
CoNLL2003 and MSRA dataset.

##  Appendix C Implementation Details

We train our model on all datasets for $4$ epochs, using a batch size of $128$
and a learning rate of $1e-5$, with the AdamW optimizer Loshchilov and Hutter
(2018) and a cosine scheduler Loshchilov and Hutter (2017). The maximum input
and output sequence lengths are set to 2048 and 512, respectively. Training is
conducted on 8 NVIDIA A100 GPUs. This configuration is applied across all
PaDeLLM-NER models, as well as two baseline models: AutoRegAug and
AutoRegStruct.

##  Appendix D Sequence Length Reduction

Results of average sequence length produced by different approaches are
presented in Table 8. Most notably, PaDeLLM-NER generates much shorter
sequences than the other models across all datasets. The lengths range from
6.54 on CoNLL20023 to 10.05 on GENIA for English datasets, and from 2.19 on
Weibo to 4.87 on Resume for Chinese datasets. The mean length for PaDeLLM-NER
is 4.86, which is significantly lower than the means of the other approaches:
35.54 for AutoRegAug and 36.48 for AutoRegStruct.

In summary, the result shows that PaDeLLM-NER produces much shorter generated
sequences compared to the other methods, which is around 13.19% to 13.67% of
the original length, respectively, indicating higher efficiency in its
inference.

##  Appendix E Error analysis

#### PaDeLLM-NER error analysis

For our error analysis, we utilize the ACE2005 dataset. We sample and manually
examine 50 erroneous examples for analysis. We seek to identify the root
causes of errors, which we have categorized into three types: (1) incorrect
mention count, referred to as Count Mismatch; (2) inaccuracies in the mention
corresponding to a specific index, termed Index Inaccuracy; and (3) errors in
the ground truth data, known as Ground Truth Errors.

The distribution of each error type is illustrated in Figure 4. It is
important to note that a significant portion of the errors stem from
inaccuracies in mention counts (i.e., Count Mismatch, about 56.8%),
underscoring the necessity for enhancements in the model’s counting
capabilities. Accurate mention counts are pivotal for the quality of
predictions. Overestimating the mention count often leads the model to either
repeat the last entity or, more problematically, fabricate an entity, thereby
escalating the rate of false positives. Conversely, underestimating the
mention count results in the model’s inability to identify some entities, thus
increasing the incidence of false negatives. Following closely is the Index
Inaccuracy error, indicating that the model sometimes struggles to accurately
pinpoint the correct mention for a given index, further emphasizing areas for
improvement.

Interestingly, our analysis reveal that a significant portion of the model’s
predictions, specifically 19.3%, are actually correct, challenging the
accuracy of the ground truth data. This observation suggests the presence of
inaccuracies within the ground truth, contributing to an elevated rate of
false positives. Prior research, as noted in studies by Min et al. (2022);
Wang et al. (2023a); Zhou et al. (2023), has demonstrated that LLMs
predominantly acquire their knowledge during the pre-training phase. These
models develop certain “core beliefs” that tend to align more closely with
human judgment. In this context, it appears that the models possess an
inherent capability to rectify errors in the ground truth data, demonstrating
their potential to improve data accuracy beyond initial human annotation.

##  Appendix F Model Scaling Up

As we increase the model size to 13B, Table 7 presents a mix of results. In
datasets like CoNLL2003 and GENIA, the model shows a significant improvement
in predictions. In contrast, the results on ACE2005 are slightly worse. Note
that the improvement in GENIA is substantial, at approximately 1.18%. Based on
these findings, it seems reasonable to suggest that continuously scaling up
the model size has the potential to maintain the performance that is at least
on par, or even superior, especially in specific industrial domains. However,
this hypothesis warrants further investigation, involving more families of
models Raffel et al. (2020a); Muennighoff et al. (2022); Touvron et al.
(2023a, b); Bai et al. (2023); Yang et al. (2023a) and a broader range of
datasets. We leave this exploration for future work.