T

Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

KBC-LM’23: Knowledge Base Construction from Pre-trained Language Models
workshop at ISWC 2023

<EMAIL_ADDRESS>url=https://hibaarnaout.com, ]

<EMAIL_ADDRESS>url=http://simonrazniewski.com/, ]

# Can large language models generate salient negative statements?

Hiba Arnaout  Max Planck Institute for Informatics, Germany  Simon Razniewski
Bosch Center for AI, Germany

(2023)

###### Abstract

We examine the ability of large language models (LLMs) to generate salient
(interesting) negative statements about real-world entities; an emerging
research topic of the last few years. We probe the LLMs using zero- and
$k$-shot unconstrained probes, and compare with traditional methods for
negation generation, i.e., pattern-based textual extractions and knowledge-
graph-based inferences, as well as crowdsourced gold statements. We measure
the correctness and salience of the generated lists about subjects from
different domains. Our evaluation shows that guided probes do in fact improve
the quality of generated negatives, compared to the zero-shot variant.
Nevertheless, using both prompts, LLMs still struggle with the notion of
factuality of negatives, frequently generating many ambiguous statements, or
statements with negative keywords but a positive meaning.

Model | Top Negative Statements
---|---
Text | didn’t make his high school team
~~doesn’t have social media~~
KG | isn’t a basketball coach
didn’t play as a power forward
ChatGPT 0-shot | didn’t invent basketball
didn’t only play basketball (positive)
ChatGPT k-shot | never played for a team outside the u.s.
didn’t play for the bulls exclusively (positive)
Alpaca 0-shot |  didn’t play for chicago until 84 (positive)
didn’t win a championship for the lakers
Alpaca k-shot | wasn’t the youngest player in the nba
didn’t win an oscar
Human | didn’t buy stakes in the chicago bulls
never coached the chicago bulls
Table 1: Negative statements about michael jordan (salient, somewhat salient,
nonsalient; ~~incorrect~~), using different methodologies: text-based
extractions, knowledge graph (KG) inferences, LLM generations, and human-
written statements.

##  1 Introduction

Motivation and Problem. Structured (knowledge graphs), and unstructured (text
corpora) information are the backbone of many AI applications, such as
question answering and chat bots. They mainly focus on storing positive
knowledge, and mostly contain little negative knowledge. The open-world
assumption, which advises to abstain from taking a stance on the truth of
absent information, compromises the usability of both forms of machine
knowledge. For instance, it is often the case that the NBA’s Basketball stars
take a coaching position after their retirement. Notably, this is not true for
michael jordan. Mining these surprising statements are useful to overcome
limitations of applications like question answering systems. For example,
querying Bing Chat111https://www.microsoft.com/en-us/edge/features/bing-chat
whether michael jordan invested in his team, the chicago bulls, returns an
irrelevant answer about his achievements with the team. In fact, it is an
interesting piece of information, that, even though he has a business-oriented
mind, he did not monetarily invest in the bulls, but in other sports
franchise, including an investment in the not so well-known team the charlotte
hornets.

State of the Art. A new research area has emerged in the last few years,
suggesting the importance of the explicit materialization of important
negative statements about real-world subjects [1]. Several methodologies have
been proposed [2, 3, 4, 5, 6]. The goal is to compile lists of statements
(biographic summaries) about subjects, where the statements are truly
negative, but also salient, unexpected, or normally mistaken as true
positives. To compile these lists, different data sources and methodologies
have been explored. In [2, 3], using web-scale knowledge graphs, candidate
salient negatives are derived from existing positive statements about highly
related entities. The computation relies on the local closed-world assumption,
an assumption of completeness over identified relevant subgraphs, coupled with
ranking metrics such as relative frequencies. Similarly, [4] explores graph
embeddings to generate candidate negative statements, which are then scored
using a fine-tuned language model (LM), by descending order of negativity.
Textual sources have been explored in [5], where commonsense negative
statements are extracted, by mining query logs, using pre-defined patterns.
[6] makes use of the edit history of large collaborative encyclopedias, namely
Wikipedia, by looking at sentences edited, where only an entity or a number
are changed. The old version of the sentence is then considered an interesting
negative statement.

LLMs for Negative Statements Generation.  Recently, LMs have been examined
about their ability to store factual knowledge about general topics [7, 8].
With LMs such as BERT [9], this was done via masked probing, e.g., “Paris is
the the capital of [MASK]” generates france as the top prediction. With large
LMs (LLMs), such as GPT-3 [10], autoregressive generation from textual prompts
is the standard, e.g., “Complete the following. Paris is..”, and receive the
completion the capital of France. A few papers focused on the ability of these
models to store and understand negative knowledge [11, 2, 12]. In [11], using
masked probing, authors found that LMs, such as BERT, struggle to understand
negation, predicting fly for the probe “Birds cannot [MASK]”. In [2], methods
to infer negative statements from knowledge graphs and text have been compared
on a more specific negation task, namely generating salient negative
commonsense statements. Results of these models are compared to ones using
GPT-3. Even though performing better than BERT-like models [11], GPT-3 was not
able to beat the SOTA model (inferences from KGs), neither on the true
negativity of statements, nor their salience. More recently, [12] studies
advanced LLMs, such as ChatGPT [13], on their ability to store negative
knowledge in a constrained text generation and question answering tasks. The
finding are contradictions in the LLM’s belief, when comparing results of both
tasks. For instance, LLMs generate the sentence “Lions live in the ocean”, but
answer “No” when asked “Do lions live in the ocean?”. [12] is an important
step towards examining LLMs’ understanding of the falseness of statements,
however, it has four main differences from our study: (i) our prompts are not
constrained to commonsense knowledge; (ii) not constrained to puzzles around a
set of words, but allowed to generate arbitrary subject-relevant statements;
(iii) our comparison includes SOTA baselines from KG and text, not just LLMs;
(iv) our study evaluates also the salience of outputs, not just their
correctness.

We summarize our contributions as follows.

  * •

We design constraint-free prompts for LLM-based negation generation, where we
only instantiate the input subject.

  * •

We examine LLMs’ understanding of salient factual negation, finding that, even
though they struggle with the notion of true negativity (-18% in correctness
compared to SOTA model), on truly negative statements, the guided few-shot
ChatGPT variant ranks first among models in salience.

  * •

We study both encyclopedic and commonsense domains, finding that it is more
challenging for LLMs to generate longer lists of salient commonsense
negatives. For instance, the zero-shot ChatGPT variant shows a decrease of 22%
in correctness@5 (compared to @1) for commonsense subjects. No decrease is
observed for encyclopedic subjects.

  * •

We compare the LLM-generated negative statements to existing SOTA methods,
from text [5] and knowledge graphs [3].

  * •

We measure the quality of the negative statements over two aspects, the
correctness (true negativity) and salience (interestingness).

The data generated can be downloaded at: https://www.mpi-
inf.mpg.de/fileadmin/inf/d5/research/negation_in_KBs/data.csv.

##  2 Probe Construction

Given a subject, we probe the LLM to generate a list of salient negative
statements about it.

Zero-shot Probe.  In this probe, we test the performance of the LLM without
providing any samples in our instructions.

Write a list of [n] salient factual negated statement about [SUBJECT].

The goal is to inspect the model’s interpretation of the notion of salient
negation without any prior examples nor definitions.

Guided Few-shot Probe.  In this probe, we guide the model with both
definitions and examples (for in-context learning).

A salient factual negated statement about an entity means that the statement
doesn’t hold in reality. Moreover, the negated statement is either surprising,
unexpected, or useful to the reader. For example:
[EXAMPLE1]
[...]
Given this definition and examples, write a list of [n] salient factual
negated statement about [SUBJECT].

In the following sample, we show a $4$-shot probe with 2 salient and 2
nonsalient samples about different types of subjects, and request 3 salient
negative statements about lebanon (LLM=ChatGPT).

A salient factual negated statement about an entity means that the statement
doesn’t hold in reality. Moreover, the negated statement is either surprising,
unexpected, or useful to the reader. For example:
penguins can’t fly.
tom cruise never won an oscar.

On the other hand, the following examples are factual negated statements that
are not salient:

penguins can’t run for presidency.
tom cruise never won the nba best player award.

Given this definition and examples, write a list of 3 salient factual negated
statement about lebanon.

Answer:

  1. 1.

is not a desert country.

  2. 2.

is not an oil-rich country.

  3. 3.

is not a landlocked country.

In Section 3, we experiment with different number of samples and different
salient:nonsalient ratio (see Appendix D).

Model | cor@1 | cor@3 | cor@5 | sal@1 | sal@3 | sal@5
---|---|---|---|---|---|---
| overall
Text Extractions | 0.38 | 0.30 | 0.33 | 0.63 | 0.69 | 0.68
KG Inferences | 0.94 | 0.76 | 0.75 | 0.88 | 0.84 | 0.83
ChatGPT 0-shot | 0.71 | 0.65 | 0.60 | 0.73 | 0.73 | 0.71
ChatGPT k-shot | 0.76 | 0.69 | 0.66 | 0.89 | 0.76 | 0.75
Alpaca 0-shot | 0.34 | 0.32 | 0.36 | 0.62 | 0.71 | 0.65
Alpaca k-shot | 0.50 | 0.47 | 0.47 | 0.66 | 0.55 | 0.56
Human | 0.77 | 0.71 | 0.69 | 0.73 | 0.70 | 0.70
| encyclopedic subjects
Text Extractions | 0.32 | 0.26 | 0.29 | 0.86 | 0.91 | 0.88
KG Inferences | 0.88 | 0.87 | 0.86 | 0.91 | 0.86 | 0.83
ChatGPT 0-shot | 0.71 | 0.76 | 0.71 | 0.65 | 0.65 | 0.62
ChatGPT k-shot | 0.76 | 0.73 | 0.74 | 0.89 | 0.74 | 0.72
Alpaca 0-shot | 0.32 | 0.33 | 0.38 | 0.63 | 0.70 | 0.64
Alpaca k-shot | 0.52 | 0.45 | 0.48 | 0.69 | 0.59 | 0.58
Human | 0.78 | 0.70 | 0.69 | 0.69 | 0.64 | 0.65
| commonsense subjects
Text Extractions | 0.47 | 0.36 | 0.39 | 0.44 | 0.47 | 0.48
KG Inferences | 1.0 | 0.65 | 0.64 | 0.83 | 0.81 | 0.83
ChatGPT 0-shot | 0.72 | 0.55 | 0.50 | 0.81 | 0.84 | 0.83
ChatGPT k-shot | 0.75 | 0.65 | 0.58 | 0.89 | 0.79 | 0.78
Alpaca 0-shot | 0.36 | 0.31 | 0.34 | 0.61 | 0.72 | 0.67
Alpaca k-shot | 0.48 | 0.48 | 0.46 | 0.63 | 0.51 | 0.55
Human | 0.76 | 0.73 | 0.69 | 0.78 | 0.75 | 0.75
Table 2: Results on correctness and salience of top negative statements (best
performance, second best).

##  3 Evaluation

Data.  We consider 50 subjects, 25 encyclopedic entities such as elon musk,
and 25 commonsense concepts, such as jogging (Full list in Appendix A). Our
intuition behind these choices is diversity: (i) in types, e.g., activities,
occupations, people; and (ii) in popularity, e.g., tom cruise (a famous
hollywood actor) and peri gilpin (a less known tv actor).

Methods.  To compile lists of negative statements about these subjects, we
consider:

  * •

Text Extractions: The pattern-based method [5] relies on a handful of manually
crafted patterns, in the form of why-questions, to extract interesting
negative statements from rich query logs, e.g., “why doesn’t amazon..” with
the completion “accept paypal”. We instantiate the query-log API with Google
and Bing, merge the results, and rank by frequency.

  * •

KG Inferences: The peer-based negation inference methodology [3] relies on a
given KG to identify highly related entities to the input entity (called
peers). Positive statements about these peers are used to infer candidate
negatives, which are finally ranked using statistical metrics, such as
relative frequency, e.g., “unlike similar physicists, such as max planck and
albert einstein, stephen hawking never won the nobel prize in physics”. We
instantiate the KGs to Wikidata [14] and Ascent [15], for
encyclopedic/commonsense subjects, respectively.

  * •

ChatGPT 0-shot: The zero-shot probe introduced in Section 2 is submitted to
ChatGPT [13] (May 2023 version).

  * •

ChatGPT k-shot: The few-shot probe in Section 2, with $k$=3
(salient:nonsalient 3:0), is submitted to ChatGPT.

  * •

Alpaca 0-shot: The zero-shot probe introduced in Section 2 is submitted to
Alpaca-13B, a model fine-tuned from LLaMA on instruction-following
demonstrations by Stanford [16].

  * •

Alpaca k-shot: The few-shot probe from Section 2, with $k$=3
(salient:nonsalient 3:0), is submitted to Alpaca-13B.
To ensure reproducibility, the randomness (temperature) for all LLMs variants
is set to 0.

  * •

Human 222We are aware of the risk that workers might use LLMs to generate
these statements. In the absence of reliable detection tools on this newly
emerging problem, we rely on our personal judgement as well as string
matchings to discard untrustworthy answers. In particular, any response that
matches the exact wording of one of the responses of the LLM baselines, or any
near-duplicates in human-generations, were rejected.: We ask MTurkers to write
lists of salient negative statements about a given subject. We show them
examples of what a salient negative statement looks like. We collect, for each
subject, two lists of statements from two workers. The performance is later
measured as the average of the two.

Model | Correct | Incorrect | Ambiguous | Positive Meaning
---|---|---|---|---
Text Extractions | 0.33 | 0.26 | 0.41 | 0
KG Inferences | 0.75 | 0.13 | 0.12 | 0
ChatGPT 0-shot | 0.60 | 0.10 | 0.19 | 0.11
ChatGPT k-shot | 0.66 | 0.17 | 0.10 | 0.07
Alpaca 0-shot | 0.36 | 0.42 | 0.13 | 0.09
Alpaca k-shot | 0.47 | 0.38 | 0.04 | 0.10
Human | 0.69 | 0.05 | 0.12 | 0.14
Sample Statement | rabbits can’t vomit | the beatles didn’t tour | avocado isn’t bad |  lebanon isn’t devoid
of historical sites |  |  |  |
Table 3: Detailed look at the factuality and true negativity of generated
statements.

Metrics. For the returned statements, we measure:

  * •

Correctness: The true negativity (is it actually false?) and factuality of a
statement (is it a judgeable statement?), e.g., not an opinion. We allow the
labels: correct, incorrect, ambiguous, or positive meaning. Samples are shown
in Table 3.

  * •

Salience: The unexpectedness, informativeness, or interestingness of a
statement. We allow: salient (1), somehow salient (0.5), and nonsalient (0).

Results are annotated on their salience by 2 domain-experts 333Experts on the
topic of salient negative knowledge at web-scale., with inter-annotator
agreement = 60%. Correctness, the more straight forward metric of the two, was
annotated by 1 of the domain-experts.

True Factuality and Negativity of Statements.  Results for correctness are
shown in Table 2, and investigated further in Table 3. The KG inferences model
ranks first on correctness overall. This is due to the factuality of KG
statements. KG triples, especially encyclopedic ones, are expressed using
precise and well-defined relations, such as award received. Moreover, they
have been curated using manual and automated techniques, and hence, their
truthfulness is easy to verify. Moreover, both variants of ChatGPT’s probes
perform significantly better than variants of Alpaca on correctness in both
domains, with an out-performance of up to 36% in correctness@1. We also notice
that, for both Alpaca and ChatGPT, their few-shot probes perform better than
the zero-shot probes, with an improvement of 16% for Alpaca and 5% for
ChatGPT. Finally, we find that many of the generated statements by humans and
LLMs were actually statements with negative keywords but a positive meaning,
such as lebanon isn’t devoid of historical sites, with up to 14% of generated
statements for the former and 11% for the latter. More samples are in Appendix
C.

Salience of Truly Negative Statements.  Results for salience are shown in
Table 2. This metric is only computed over (previously annotated) correct
statements. The best performances are shared between the KG inferences model
and ChatGPT’s few-shot variant. Though not performing comparably well overall,
the text extractions model ranks first on salience of encyclopedic subjects @3
and 5. This is especially apparent for prominent entities, which are
frequently queried using famous search engines. Again, ChatGPT’s variants
significantly outperforms Alpaca’s on the notion of salience, with up to 23%
improvement in salience@1, maintaining the same level of quality for both
types of subjects. Sample results from all models are shown in Table 1 and
Appendix E. An experiment on the quality of generated negatives over two
popularity levels, namely prominent and long tail subjects, is in Appendix B.

Effect of $k$ Value on LLM’s Few-shot Probe.  We examine the LLM using
different numbers of samples, for in-context learning. We consider a subset of
5 entities (3 encyclopedic and 2 commonsense), and assess the performance of
the few-shot ChatGPT using different values of $k$, with different
salient:nonsalient ratios. Results are in Table 4. Adding a small but equal
number of salient and nonsalient samples (3:3) improves the correctness by 8%,
compared to only adding salient samples (3:0), however, at the expense of
their salience, which drops by by 14%. Adding only nonsalient samples (0:3)
compromises both metrics. Finally, adding a larger but equal number of salient
and nonsalient samples (10:10) does not result in any improvements.

k | sal:nonsal | Correctness | Salience
---|---|---|---
3 | 3:0 | 0.72 | 0.54
3 | 0:3 | 0.52 | 0.30
6 | 3:3 | 0.80 | 0.40
20 | 10:10 | 0.52 | 0.34
Table 4: Results given different values for the in-context learning parameters
(best performance, second best).

##  4 Take-home Lessons & Open Issues

In this paper, we perform a systematic evaluation of LLMs’ ability to generate
salient negative statements. We assess them against existing method and
crowdsourced statements. We find that LLMs’ few-shot probes show promising
results in salience@1. Moreover, we find that ChatGPT outperforms Alpaca on
this task, in both correctness and salience. One of the remaining limitations,
however, is the ability of LLMs to recognize truly negative factual
statements, as opposed to ambiguous, or seemingly negative statements with
positive meaning. We hope that this study, as well as the following
observations, give insights to future researchers on this topic.

Prompt Engineering.  There is a wide consensus that LLMs are very powerful
when you ask them for information in the right manner. In our task, we notice
that the wording, especially of the zero-shot probe, changes the results
dramatically. For instance, using the expressions negative statements, negated
statements, and negation statements returns completely different responses.
For instance, the probe with the word negated (alone without salient factual)
returns obviously true statements with negative keywords added to them, e.g.,
“stephen hawking was not a physicist”. The probe with the word negative does
not return any results, but an apology from the AI about not being able to
give bad statements about individuals. On this and other tasks, designing
intuitive prompts and studying the ability of LLMs to understand them is the
most important part of the process [17].

The Notion of Salient Negation.  Assessing the truthfulness of statements is
one thing, but assessing the salience of negatives is more challenging.
Salience is a subjective metric. For instance, for a Basketball fan, the fact
that jordan did not star in the film space jam 2 (the first was built around
him), is a big deal. For others, the salience is not obvious. In addition of
the expertise of the reader, their nature is also important. In other words,
are these negations generated for a human-reader, or to equip machines with
better negative knowledge? For instance, what might not appear salient to a
human, can be important to improve the reasoning skills of a chat bot. In this
study, we assume that the reader is a human, who usually has a higher standard
for what is interesting than a machine. Generally, designing experiments
should take into consideration downstream applications and information about
the end-user.

Maintenance.  Ideally, models must always keep track of real-world changes
which affect the truthfulness of statements, coverage of emerging entities,
etc. This is relatively easy in the collaborative knowledge graphs, which are
updated on a daily basis. For LLMs, the process of re-training is much more
expensive. e.g., in May 2023, ChatGPT still generates the statement brendan
fraser has never won an oscar, which is no longer true, due to his win in 2023
(the training of the model has been completed in September 2021).

## References

  * Razniewski et al. [2023] S. Razniewski, H. Arnaout, S. Ghosh, F. Suchanek,  Completeness, recall, and negation in open-world knowledge bases: A survey,  arXiv (2023).
  * Arnaout et al. [2022] H. Arnaout, S. Razniewski, G. Weikum, J. Z. Pan,  UnCommonSense: Informative negative knowledge about everyday concepts,  in: CIKM, 2022.
  * Arnaout et al. [2020] H. Arnaout, S. Razniewski, G. Weikum,  Enriching knowledge bases with interesting negative statements,  in: AKBC, 2020.
  * Safavi et al. [2021] T. Safavi, J. Zhu, D. Koutra,  NegatER: Unsupervised Discovery of Negatives in Commonsense Knowledge Bases,  in: EMNLP, 2021.
  * Romero et al. [2019] J. Romero, S. Razniewski, K. Pal, J. Z. Pan, A. Sakhadeo, G. Weikum,  Commonsense properties from query logs and question answering forums,  in: CIKM, 2019.
  * Karagiannis et al. [2019] G. Karagiannis, I. Trummer, S. Jo, S. Khandelwal, X. Wang, C. Yu,  Mining an "anti-knowledge base" from Wikipedia updates with applications to fact checking and beyond,  PVLDB (2019).
  * Lee et al. [2020] N. Lee, B. Z. Li, S. Wang, W.-t. Yih, H. Ma, M. Khabsa,  Language models as fact checkers?,  in: ACL, FEVER workshop, 2020.
  * Petroni et al. [2019] F. Petroni, T. Rocktäschel, S. Riedel, P. Lewis, A. Bakhtin, Y. Wu, A. Miller,  Language models as knowledge bases?,  in: EMNLP, 2019.
  * Devlin et al. [2019] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova,  BERT: Pre-training of deep bidirectional transformers for language understanding,  in: NAACL, 2019.
  * Radford et al. [2019] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever,  Language Models are Unsupervised Multitask Learners,  OpenAI technical report (2019).
  * Kassner and Schütze [2020] N. Kassner, H. Schütze,  Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly,  in: ACL, 2020.
  * Chen et al. [2023] J. Chen, W. Shi, Z. Fu, S. Cheng, L. Li, Y. Xiao,  Say what you mean! large language models speak too positively about negative commonsense knowledge,  arXiv (2023).
  * OpenAI [2022] OpenAI, Introducing chatgpt, https://openai.com/blog/chatgpt, 2022.
  * Vrandečić and Krötzsch [2014] D. Vrandečić, M. Krötzsch,  Wikidata: a free collaborative knowledge base,  CACM (2014).
  * Nguyen et al. [2022] T. Nguyen, S. Razniewski, J. Romero, G. Weikum,  Refined commonsense knowledge from large-scale web contents,  TKDE (2022).
  * Taori et al. [2023] R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, T. B. Hashimoto, Alpaca: A strong, replicable instruction-following model, https://crfm.stanford.edu/2023/03/13/alpaca.html, 2023.
  * Jang et al. [2023] J. Jang, S. Ye, M. Seo,  Can large language models truly understand prompts? a case study with negated prompts,  in: Proceedings of The 1st Transfer Learning for Natural Language Processing Workshop, 2023.

##  Appendix A Encyclopedic and Commonsense Subjects

We consider 50 subjects of different domains, namely commonsense and and of
different popularity, namely prominent and long tail (see Table 5).

|  Encyclopedic |  Commonsense
---|---|---
Prominent |  stephen hawking, michael jordan, lebanon, michelle obama, microsoft, china, amazon, albert einstein, the beatles, elon musk, angela merkel, taxi driver, taj mahal, white house, eat pray love, tom cruise, brendan fraser, the godfather, my cousin vinny, mercedes-benz group, gmc, linkedin |  elephant, soup, lawyer, acne, mother, gorilla, pancake, newspaper, jaguar, avocado, garlic, chef, salad, rabbit, jogging, cufflink, strudel, librarian, armchair
Long tail |  peri gilpin, caramel, ubisoft |  tabbouleh, breadfruit, kitchenette, hockey stick, basketball court, coffee table
Table 5: Subjects considered in our experiments.

##  Appendix B Prominent and Long Tail Subjects

We recompute the quality of negatives (@5) over two levels of subject-
popularity, namely prominent and long tail. Figure 1 indicates a significant
decrease in both salience and correctness for long tail subjects, for the
text-based method; dropping to only 1% on salience. Using query logs as the
corpus, users query prominent/trendy subjects much more frequently than long
tail ones. We find the human-written statements for both popularity-levels
comparable, with a slight advantage for prominent subjects. Similarly, the KG
inferences model shows comparable results with a slight advantage of prominent
subjects in correctness, and of long tail subjects in salience. Finally, we
find an unexpected improvement, for all LLM variants, of long tail subjects
over prominent ones, in both metrics. One interpretation could be the large
amount of noisy web sources (main data source for training LLMs), about famous
entities. For example, tabbouleh (long tail) is a specific instance of salad
(prominent). While negatives about the former are more clear-cut, e.g.,
tabbouleh isn’t made with rice but bulgur, negatives about the latter seem
more unfocused, e.g., salad isn’t always a healthy choice.

Figure 1: Prominent subjects (left-side), long tail subjects (right-side)
(best performance, second best)

##  Appendix C Negative Statement with Positive Meaning

As shown in Table 3, many of the LLM-generated and crowdsourced statements are
in fact positive. Some of the recurring expressions which convey a positive
meaning using negative keywords:

Expression: not exclusively (15 statements)
Amazon did not exclusively focus on selling its own products.

Expression: not without (2)
Example: Strudel is not tasty without sugar.

Expression: not just (9)
Example: Acne is not just a teenage problem.

Expression: not only (20)
Example: Librarians do not only work in public libraries.

Expression: not limited to (5)
Example: Coffee tables are not limited to indoor use.

Expression: not solely (7)
Example: GMC does not solely operate in the United States.

Expression: not all (10)
Example: Not all librarians are women.

##  Appendix D $k$-shot In-context Learning Probe

In this probe $k$=6 (3:3); LLM=ChatGPT.

A salient factual negated statement about an entity means that the statement
doesn’t hold in reality. Moreover, the negated statement is either surprising,
unexpected, or useful to the reader. For example:
penguins can’t fly.
istanbul isn’t the capital of turkey.
tom cruise never won an oscar.

On the other hand, the following examples are factual negated statements that
are not salient:

penguins can’t run for presidency.
istanbul isn’t the capital of france.
tom cruise never won the nba best player award.

Given this definition and examples, write a list of 3 salient factual negated
statement about microsoft.

Answer:

  1. 1.

is not primarily a dating platform.

  2. 2.

does not charge users a fee to create an account.

  3. 3.

does not allow users to post anonymous content.

##  Appendix E Sample Results

The following tables show the top results about linkedin, chef, and angela
merkel, respectively:

Model | Top Negative Statements (linkedin)
---|---
Text | isn’t working
isn’t loading
KG | isn’t headquartered in san francisco
isn’t a software company
ChatGPT 0-shot | isn’t designed for sharing personal content
doesn’t permit users to buy followers
ChatGPT k-shot | isn’t used for online dating
doesn’t allow users to post pictures of their pets
Alpaca 0-shot | doesn’t have a user-friendly interface
doesn’t provide any value to its users
Alpaca k-shot | isn’t a social media platform
doesn’t own the content posted on its platform
Human | doesn’t have a billion members
wasn’t founded by mark zuckerberg

Model | Top Negative Statements (chef)
---|---
Text | doesn’t wear hat
doesn’t eat their own food
KG | doesn’t take orders
doesn’t bring drinks
ChatGPT 0-shot | didn’t use any garlic
didn’t win any cooking competitions
ChatGPT k-shot | doesn’t just cook food
not all have formal culinary training
Alpaca 0-shot | don’t need to have an understanding of nutrition
don’t need to have good knife skills
Alpaca k-shot | don’t need to be certified
don’t usually work with raw ingredients
Human | doesn’t wash the dishes
doesn’t always wear the chef’s hat

Model | Top Negative Statements (angela merkel)
---|---
Text | didn’t listen to donald trump
doesn’t deserve to be honoured by germany
KG | isn’t on twitter
isn’t a lawyer
ChatGPT 0-shot | isn’t a native german speaker
didn’t originally pursue a career in politics
ChatGPT k-shot | has never been married
is not a member of the SPD
Alpaca 0-shot | isn’t a member of the CDU
isn’t a scientist
Alpaca k-shot | isn’t the first female chancellor of germany
isn’t from east germany
Human | didn’t grow up in a wealthy family
isn’t a member of the SPD


###  S-IV.2 Perturbative input correlations

We have shown that the point $P_{0}=(\zeta_{0}=0,\eta_{0}=\Theta_{c}\rho)$
where the R and N phase meet is a point of non-analyticity for the principal
eigenvalue regarded as a function of the parameters, and thus belongs to a
phase boundary. Moreover, this phase boundary cannot stop there, because it is
a boundary between two phases that have different symmetries – one that
displays orientation selectivity and one that does not. How is this phase
boundary continued for $\zeta>0$? Will it curve up or down in the
$(\zeta,\eta)$ space?

Since we possess the exact solution for $\zeta=0,\eta>\Theta_{c}\rho$,
perturbation theory is an ideal tool to address this question. We will build a
perturbation theory in the small parameter $\zeta/\eta$. As we saw in Sec.
S-IV.1, our starting point for perturbation theory is a highly degenerate set
of eigenfunctions, mainly due to the degeneracy in the wavenumber. But since
the full operator for $\zeta>0$ commutes with cortical translations, different
translational eigenstates are not coupled by the perturbation, and
nondegenerate perturbation theory with respect to wavenumbers may be applied.

The theory will prove the following three facts:

(1) the $\omega$ degeneracy is removed by an infinitesimal $\zeta>0$ for any
$\eta$, and this happens in such a way that $\omega=0$ is always the principal
eigenstate;

(2) the phase boundary starting at the point $P_{0}=(0,\Theta_{c}\rho)$ has a
flat slope at that point in the $\zeta/\eta$ plane;

(3) the $xy$ degeneracy of the p-wave eigenfunctions survives at finite
$\zeta$.

###  S-IV.3 Optimal wavenumber for oriented eigenfunctions: transverse
orientations

We have mentioned that the perturbation does not couple degenerate
wavenumbers. The same is true with the additional degeneracy in the
orientation of selectivity, and it is possible to study the two 2p
eigenfunctions separately because the full operator $\hat{L}^{p}$ does not
couple them for any value of the parameters. Indeed, we have it by symmetry
that $\langle\Psi^{x}|\hat{L}^{p}|\Psi^{y}\rangle=0$ for any $\zeta$ and
$\omega$. This means that we can study the effect of a small but finite
$\zeta$ separately on the two eigenfunctions (applying nondegenerate
perturbation theory).

We begin with the y-oriented wave (transverse orientation). While $\Psi^{y}$
is an eigenfunction of $\hat{L}$ only for $\zeta=0$, it can be checked that
its generalization

$\chi^{y}_{0,1}(\bm{r})=\frac{r_{y}}{\sqrt{\pi}\sigma^{2}}\exp\left(-\frac{r^{2}}{2\sigma^{2}}\right)e^{-i\frac{\eta^{2}}{\mu^{2}}\bm{\omega
r}},$ (S102)

is an exact eigenfunction of the full operator $\hat{L}^{p}$ over the whole
phase diagram. Indeed, it is an exact eigenfunction of $\hat{L}$ and, being
orthogonal to the constraint ket $|a_{\omega}\rangle$, it belongs to the null
space of the constraint operators $\hat{S}$ and $\hat{T}$:

$\langle\chi^{y}_{0,1}|\hat{S}|\chi^{y}_{0,1}\rangle=\langle\chi^{y}_{0,1}|\hat{T}|\chi^{y}_{0,1}\rangle=0.$
(S103)

The corresponding eigenvalue of $\hat{L}^{p}$ is given, for every point in
$(\zeta,\eta)$ space, by

$\displaystyle\Lambda^{y}=\frac{2\pi\mu^{2}}{\beta^{2}}\exp\left(-\frac{\omega^{2}}{2\Omega^{2}}\right),$
(S104)

with $\beta$ and $\Omega$ defined according to Eq. (S25) and (S11)
respectively.

For $\zeta=0$, as we knew, this eigenvalue is independent on the wavenumber.
However, for any $\zeta>0$, Eq. (S104) describes an eigenvalue that decreases
monotonically with the wavenumber, hence the degeneracy is removed. We can
conclude that, in the limit of small $\zeta$, the principal y-oriented
eigenfunction is uniform over the cortex, i.e. translation symmetry is not
broken.

###  S-IV.4 Optimal wavenumber for oriented eigenfunctions: longitudinal
orientations

We now turn to considering the x-oriented function $\Psi^{x}$. It can be
checked that $\Psi^{x}$ is orthogonal to the constraint state
$|a_{\omega}\rangle$, which entails

$\langle\Psi^{x}|\hat{S}|\Psi^{x}\rangle=\langle\Psi^{x}|\hat{T}|\Psi^{x}\rangle=0,$
(S105)

This holds true for any value of $\zeta$. However $\Psi^{x}$ is an eigenstate
only for $\zeta=0$ and, differently from the case of $\Psi^{y}$ seen above, it
is not straightforward to build a generalization of $\Psi^{x}$ that will be an
eigenstate of $\hat{L}^{p}$ at any point in parameter space.

Therefore, we will restrict here our attention to sufficiently small non-zero
values of $\zeta$ and will build a perturbation theory in the parameter
$\epsilon=\zeta^{2}/\eta^{2}$. Hence we write the operator $\hat{L}^{p}$ as
$\hat{L}^{p}=\hat{L}^{p}(\epsilon=0)+\hat{\Delta}+O(\epsilon^{2})$, where
$\hat{\Delta}$ includes the first order in $\epsilon$, and we will treat
$\hat{\Delta}$ as a perturbation. In the shift operator
$\hat{\Delta}=\Delta\hat{L}+\Delta\hat{S}+\Delta\hat{T}+\Delta\hat{T}^{\dagger}$,
because of Eq. (S105), we only have to compute the $\hat{L}$-term.

We begin by expanding to the first order in $\epsilon$ Eq. (S13), which yields

$\Delta
L(\bm{r},\bm{s};\omega)=\epsilon\left[-\frac{\omega^{2}\eta^{2}}{2}+i\omega(r_{x}-s_{x})+\frac{(\bm{r}-\bm{s})^{2}}{2\eta^{2}}\right]e^{-i\omega(r_{x}-s_{x})-\frac{r^{2}+s^{2}}{4\rho^{2}}-\frac{(\bm{r-s})^{2}}{2\eta^{2}}}.$
(S106)

We only keep the terms that have a nonvanishing expectation value in
$\Psi^{x}$; in particular, we neglect terms that change sign if we swap the
two variables $r_{x}$ and $s_{x}$, because the integral would be zero. In
addition, we may ignore terms whose expectation value in $\Psi^{x}$ (i.e.
whose contribution to $\langle\Psi^{x}|\Delta\hat{L}|\Psi^{x}\rangle$) will
bear no dependence on the wavenumber. After some algebra this leaves a single
first-order term in Eq. (S106) that obeys all these requirements, namely:

$\displaystyle\Delta
L(\bm{r},\bm{s};\omega)\sim-\frac{\omega^{2}\zeta^{2}}{2}e^{-i\omega(r_{x}-s_{x})-\frac{r^{2}+s^{2}}{4\rho^{2}}-\frac{(\bm{r-s})^{2}}{2\eta^{2}}}.$
(S107)

The corresponding expectation value is

$\Delta=\langle\Psi^{x}|\Delta\hat{L}|\Psi^{x}\rangle=-\frac{\omega^{2}\zeta^{2}}{2}\Lambda_{2p}(\zeta=0),$
(S108)

a negative shift in the eigenvalue that is minimized by setting $\omega=0$.

We have thus proven that, for sufficiently small $\zeta$ and given $\eta$, the
principal eigenstate is always cortically uniform ($\omega=0$), as long as the
principal eigenstate for the given $\eta$ and $\zeta=0$ is an R-phase. This
entails that in the limit $\zeta\rightarrow 0$, the principal eigenstate of
the system has a zero wavenumber for any $\eta>\Theta_{c}\rho$. Hence, the
slope of the phase boundary at $(0,\Theta_{c}\rho)$ cannot be positive.

While the degeneracy in wavenumber has been removed by first-order
perturbation theory, the degeneracy between the $x$ and $y$ orientations has
not been removed, as seen by comparing Eq. (S104) and Eq. (S108) to the second
order in $\omega$ and using $\Omega\sim 1/\zeta$.

###  S-IV.5 Optimal wavenumber for non-oriented eigenfunctions

The s-wave (that is, non-oriented) eigenfunctions of $\hat{L}^{p}$ are also
degenerate in the cortical wavenumber for $\zeta=0$. To see which wavenumber
effectively prevails, we must build a perturbation theory in
$\epsilon=\zeta^{2}/\eta^{2}$ starting from the (unknown) principal eigenstate
of the zero-wavenumber operator, which we call $|s\rangle$ because of its
being an s-mode.

From the discussion of Sec. S-IV (see Eq. S97), we know that the the principal
eigenstate at $\zeta=0$, when an s-wave, must have the form
$\langle\omega,\bm{r}|s\rangle=\psi_{s}(r)e^{-i\omega r_{x}}$. We can thus use
Eq. (S16) to write the level shift as

$\Delta\hat{L}^{p}(\omega)=\langle s|\left(1-|a_{\omega}\rangle\langle
a_{\omega}|\right)\Delta\hat{L}_{\omega}\left(1-|a_{\omega}\rangle\langle
a_{\omega}|\right)|s\rangle$ (S109)

where the matrix elements of $\Delta\hat{L}_{\omega}$ have the form given in
Eq. (S106).

We notice now that factors of the type $e^{-i\omega r_{x}}$ will cancel in the
integrands of all scalar products that appear in Eq. (S109). As a consequence,
the third term in the square brackets of Eq. (S106) may be ignored, as it adds
no dependence on the cortical wavenumber.

The first term in the square brackets of Eq. (S106), on the other hand, yields
the level shift

$\Delta\hat{L}_{e}^{(1)}=-\frac{\omega^{2}\zeta^{2}}{2}\langle
s|\hat{L}^{p}(\zeta=0)|s\rangle=-\frac{\omega^{2}\zeta^{2}}{2}\int\psi_{s}(\bm{r})L^{c}(\bm{r},\bm{s})\psi_{s}(\bm{s})\bm{dr}=-\frac{\omega^{2}\zeta^{2}\Lambda_{s}}{2}$
(S110)

where to write the last equality we have used the fact that $\psi_{s}$ is, by
definition, an eigenfunction of $\hat{L}^{c}$ with a positive eigenvalue
$\Lambda_{s}$. The resulting shift is a monotonically decreasing function of
the wavenumber.

The only remaining term is the second one in the square brackets of Eq.
(S106), namely

$\Delta
L^{(2)}(\bm{r},\bm{s};\omega)=i\epsilon\omega(r_{x}-s_{x})e^{-i\omega(r_{x}-s_{x})-\frac{r^{2}+s^{2}}{4\rho^{2}}-\frac{(\bm{r-s})^{2}}{2\eta^{2}}},$
(S111)

which also yields a level shift of the form

$\Delta\hat{L}_{e}^{(2)}=\langle s|\Delta\hat{L}^{(2)}|s\rangle+|\langle
s|a_{\omega}\rangle|^{2}\langle
a_{\omega}|\Delta\hat{L}_{\omega}|a_{\omega}\rangle-2\ \Re\left[\langle
s|a_{\omega}\rangle\langle a_{\omega}|\Delta\hat{L}^{(2)}|s\rangle\right].$
(S112)

The ensuing integrals are quickly estimated by symmetry considerations. The
factors $e^{-i\omega(r_{x}-s_{x})}$ in Eq. (S111) cancel everywhere in Eq.
(S112). While only the real part of these imaginary exponentials would
contribute, the resulting matrix element is effectively antisymmetric in the
swapping of the $\bm{r}$ and $\bm{s}$ coordinates. This makes the first two
terms in Eq. (S112) vanish by symmetry; since the final square bracket is
purely imaginary, the third term is also zero.

Hence the full eigenvalue shift is given by Eq. (S110) and decreases
monotonically as a function of the wavenumber. We conclude that the wavenumber
degeneracy at $\zeta=0$ is removed even by an infinitesimal range of
presynaptic correlations, and the uniform cortical mode is favored for any
$\eta$.

Building upon this, we conclude that, to the lowest order in $\zeta/\eta$, the
principal eigenvalues of the s- and p-modes are unchanged from those at
$\zeta=0$, and therefore the phase boundary starting from
$P_{0}=(0,\Theta_{c}\rho)$ has a flat slope at that point (Fig. S7).

##  S-V The triple point

We pointed out the existence of a point $P$ on the $\zeta=0$ axis where the
$N$-phase transitions into the $R$-phase, and showed that this phase boundary
continues parallel to the $\zeta$-axis for perturbatively small values of
$\zeta$. This must be matched with what was shown about the long-range limit,
the existence of a linear phase boundary between the R and T phase. These
boundary lines cannot terminate, but can only continue into each other, the
reason being that beyond the termination point of a phase boundary two
different symmetries would have to merge. The simplest diagram adhering to
this requirement is one where the R-region extending above the TR boundary for
long ranges connects, at short ranges, to the R-region that extends above the
NR boundary. The missing stretch of phase boundary is sketched as a dashed red
curve in Fig. S7.

The immediate consequence of this scenario is that an extra boundary, located
at values of $\eta$ below the lower limit of the R-phase, must separate the
short-range N-region from the T-region that begins for longer ranges (orange
dashed curve of Fig. S7). In principle one could expect this second boundary
to start from any point along the lower contour of the R-phase, i.e. at an
arbitrary value of $\zeta$. However, it must be remembered the discontinuity
between the R and N eigenfunctions is only necessary for $\zeta=0$, where
projective fields from LGN are decoupled from each other and there is
degeneracy in the wave number. For $\zeta>0$, a suitable path through the
T-phase can always bridge the N and R eigenfunctions continuously. Variational
reasoning is sufficient to infer that, where such transition exists, it is
advantageous over a sharp transition.

It follows that we expect the T region to taper all the way to point $P_{0}$,
which consequently is a triple point of the system.

Figure S7: Patching together of phase boundaries. For uncorrelated inputs
(Sec. S-IV), the $\eta$-axis contains a transition point where rotation
symmetry is broken (red dot). The perturbation theory for short-range input
correlations (Sec. S-IV.2) shows that this transition point is continued by a
flat phase boundary. The asymptotic rank reduction used for the long-range
limit (Sec. S-III), revealed an RT boundary far away from the origin. We also
know from Sec. S-II.3 that the N phase is forbidden outside a quarter-circle
containing the red dot on its contour, and the R phase is forbidden inside it.
This leads to predicting an N-T boundary (orange dashed line) contained within
the quarter-circle, and an R-T boundary stretching from the red dot into the
long-range regime. The red dot is a triple point of the system.


# Sound Event Detection in Urban Audio With Single and Multi-Rate PCEN

###### Abstract

Recent literature has demonstrated that the use of per-channel energy
normalization (PCEN), has significant performance improvements over
traditional log-scaled mel-frequency spectrograms in acoustic sound event
detection (SED) in a multi-class setting with overlapping events. However, the
configuration of PCEN’s parameters is sensitive to the recording environment,
the characteristics of the class of events of interest, and the presence of
multiple overlapping events [1]. This leads to improvements on a class-by-
class basis, but poor cross-class performance. In this article, we experiment
using PCEN spectrograms as an alternative method for SED in urban audio using
the UrbanSED dataset, demonstrating per-class improvements based on parameter
configuration. Furthermore, we address cross-class performance with PCEN using
a novel method, Multi-Rate PCEN (MRPCEN). We demonstrate cross-class SED
performance with MRPCEN, demonstrating improvements to cross-class performance
compared to traditional single-rate PCEN.

Index Terms—  Acoustic noise, acoustic sensors, acoustic signal detection,
signal classification, spectrogram.

##  1 Introduction

Noise suppression is a critical step in acoustic signal detection,
particularly so in the case of practical sound event detection (SED) in field
recordings. Popular approaches to this task use convolutional operators,
mimicking the methods used in computer vision [2]. When using audio as an
input to these methods, the images are typically time-frequency
representations (spectrograms). Traditionally, log-scaling is the primary
approach for noise reduction, and is the standard approach when spectrograms
are the feature of interest. For single-source audio in clean acoustic
environments, this is sufficient for SED.

However, real world environments typically don’t have clean, separated events
of a single class; this is particularly true in the case of urban audio. Field
recordings often have multiple sound sources of varying acoustic qualities,
leading to varying cross-class performance. Furthermore, the introduction of
auditory deformations can lead to rapid performance degradation [3].

Former research has proposed using per-channel energy normalization (PCEN) as
a time-frequency representation to mitigate the effects of background noise,
demonstrating its use as an input to convolutional methods in SED. PCEN has
proven beneficial in various single-source tasks, including bioacoustic event
detection [1], keyword spotting [3], and vocal detection in music [4]. The
per-channel background suppression characteristics make PCEN an attractive
choice in SED in urban environments, as background noise in urban environments
tends to be Brownian, rather than Gaussian, an assumption made when log-
scaling spectrograms [5].

One constraint of PCEN is its dependence on parameter configurations in
relation to the specific acoustic properties of the sound event of interest
[6]. This makes PCEN poorly suited for multi-class classification, as a given
PCEN parameter configuration will likely suit only one particular class of
sound events while performing poorly across other classes. In this article, we
demonstrate the effectiveness of PCEN for SED in urban environments.
Furthermore, we propose a new approach to acoustic event detection that
combines the foreground separation characteristic of PCEN while preserving
multi-class performance: Multi-Rate PCEN (MRPCEN). By computing a multi-
layered PCEN spectrogram at different parameter configurations, we gain the
advantages of PCEN, particularly robustness to varying acoustic conditions and
auditory deformations without the performance loss associated with cross-class
performance.

##  2 Audio Representation

While log-scaling mel-spectrograms is a simple and computationally efficient
method of range compression, it has limited effects in certain conditions,
particularly those in which background noise is non-Gaussian noise. PCEN has
been used as a pre-processing method for time-frequency representations that
reduces the effects of noise on convolutional neural networks by Gaussianizing
the distribution of magnitudes across mel-frequency spectrogram coefficients
[3].

Fig. 1: A comparison of standard SED audio featurizations compared to Multi-
Rate PCEN. Note the unevenly distributed noise in the log-mel spectrogram,
with more noise in the lower end of the spectrum (due to Brown noise found in
urban environments), which is removed in the PCEN spectrogram.

###  2.1 PCEN

PCEN is a sequence of audio processing steps on a spectrogram
$\textbf{E}(t,f)$ using adaptive gain control scaled by the response an
autogregressive filter $\phi_{T}$, followed by dynamic range compression. It
takes the form:

$\textbf{PCEN}(t,f)=\left(\frac{\textbf{E}(t,f)}{(\epsilon+(\mathbf{E}*\phi_{T})(t,f))^{\alpha}}+\delta\right)^{r}-\delta^{r}$
(1)

The gain $\alpha$ scales the smoothing of the spectrogram. $\delta$ and $r$
are the bias and power of the dynamic range compression that PCEN provides. We
can see that the filter response $\mathbf{M}(t,f)=(\mathbf{E}*\phi_{T})(t,f)$
scales our PCEN magnitude by:

$\mathbf{M}(t,f)=(\mathbf{E}*\phi_{T})(t,f)=s\mathbf{E}(t,f)+(1-s)\mathbf{M}(t-\tau,f)$
(2)

where $0<s<1$ is the weight of 1st order autogregressive filter (AR(1)), and
$\tau$ is the discretized step defined by the hop size of the input
spectrogram ($\epsilon$ is an offset term). The resulting filter is a low-pass
filter of 0dB gain and a cuttoff frequency of
$\omega_{c}=\frac{2\pi\tau}{T}=\arccos{(1-\frac{s^{2}}{2(1-s)})}$ at 3 dB, and
sidelobe falloff of 10 dB per decade near $\omega_{c}$ [6].

We consider this filter response $\mathbf{M}(t,f)$ an approximation to the
relative magnitude of stationary background noise at each frequency band $f$.
The effect of scaling our output PCEN spectrogram by the reciprocal of this
response is amplifying/magnifying the response of foreground events and
suppressing background events on a per-channel basis. In regimes where the
background noise isn’t Gaussian, such as the case with Brownian noise in urban
environments, this decorrelates noise across different frequency bands.

A critical tuning parameter of this smoothing of the autoregressive filter is
the rate parameter $T$, which defines the cutoff frequency by
$\omega_{c}=\frac{2\pi\tau}{T}$. Setting $T$ too low will reduce the noise
reduction effects of PCEN, but setting it too high will suppress the sound
event of interest, especially if the sound event of interest is stationary.
Prior practical recommendations for the rate parameter are defined by the
stationarity of the sound, the frequency range, and the chirp rate. For single
sources in a consistent acoustic environment, tuning the rate parameter $T$ to
these specifications proves sufficient for audio detection and classification
tasks [6].

###  2.2 Multi-rate PCEN

In a field setting, with several audio classes with varying acoustic
properties and variable recording conditions, a singular ideal value for $T$
is much harder to identify. For example, in the case of urban audio, the
combination of short, fast decay sounds (such as gunshots or dog barks)
compared to longer ambient sounds (e.g. sirens, air conditioners) lead to
widely varying preferred values of $T$. To capture information across varying
regimes of our rate parameter, $T$, we take inspiration from 3-channel RGB
images in computer vision. These three separate but correlated feature maps
can be passed to a convolutional neural network (CNN) which can use this
multi-frequency image to make predictions that a black-and-white image could
not.

We replicate this multi-regime approach, but instead of frequency responses,
we vary our rate parameter $T$ in each layer, as shown in figure 1. Each $i$th
layer of the image has a differing level of gain control applied based on the
cutoff threshold from $T_{i}$. By using multiple logarithmically-scaled values
of $T_{i}$, we can produce a multi-layered image that captures information of
sounds at varying decay lengths. This ensures that sound events that may be
suppressed using one value of $T_{i}$ will be preserved in another. The
resulting multi-channel image can be used as an input to a CNN much like a
multi-channel color image is used in machine vision. By incorporating
information at varying degrees of gain control, our model preserves the
robustness of PCEN without degrading multi-class performance.

##  3 Experimental Design

###  3.1 UrbanSED

We used the UrbanSED dataset to evaluate the performance on this method for
the task of sound event detection[5]. UrbanSED contains 10,000 synthetic
soundscapes of 10 distinct sound classes, with each class having approximately
1000 instances per class, drawn from the UrbanSound8K dataset. Each soundscape
contains 1-9 time/class labeled foreground events with additive background
Brownian noise. The dataset is pre-split in a 6-2-2 training, validation, and
evaluation subsets. Because UrbanSED is synthetic, we can ensure no spurious
unlabeled audio is included in the soundscapes, making it a standard benchmark
for state-of-the-art SED models.

###  3.2 Data augmentation

Because one of the primary predicted benefits of PCEN is improved robustness
to audio deformations, we augmented our dataset with several reverberant
duplicates of UrbanSED.

In this implementation, reverb is modeled as a convolution of a source signal
and an impulse response of a given acoustic environment. Augmenting a given
audio clip with reverb is done by computing the convolution of the impulse and
the audio file. This effectively computes the response of the audio clip in
the acoustic environment associated with the impulse. We used 6 distinct
reverb responses, three impulse responses recorded in different acoustic
environments, and three synthetic impulse responses of white noise with an
exponential decay envelope $e^{-t/\tau_{c}}$.

The three real impulse responses were recorded in a bedroom111“My Bedroom”,
https://freesound.org/people/Uzbazur/sounds/382907/, an
alleyway222“alley.wav”,
https://freesound.org/people/NoiseCollector/sounds/126804/, and a
tunnel333“tunnel_2013”,
https://freesound.org/people/recordinghopkins/sounds/175358/, each with
increasing decay times. The three synthetic impulse responses had decay time
constants $\tau_{c}$ of $0.1$s, $0.3$s, and $0.5$s. In addition to reverb
augmentation, we also duplicated our dataset by pitch shifting each sample by
{$\pm 1,\pm 2$} semitones. We applied pitch shifts and convolutional to reverb
to our dataset using using MuDA, a library for musical data augmentation [7].

###  3.3 Audio featurization

Training audio was processed using librosa 0.7.2 [8] which generated a PCEN
spectrogram for each audio sample and rate parameter $T$. Each spectrogram had
the following parameters: sampling rate $44.1$kHz, window size $1024$ samples,
hop length $512$ samples, and $128$ mel-frequency bands. Our resulting frame
rate was $44.1$kHz$/512\approx 86$Hz. At a 10 second length sample length,
each spectrogram had a horizontal length of 862 samples per band.

Our rate parameters were 10 logarithmically spaced values, ranging from
$2^{0}=1$ to $2^{9}=512$, corresponding to averaging over windows ranging from
$10$ms to $6$s. The adaptive gain control bias was set to $\epsilon=10^{-6}$,
the gain was set to $\alpha=0.98$, the dynamic range compression bias was set
to $\delta=2$, and the compression power $r=0.5$ for all rate configurations;
these parameters were chosen based on prior work in audio classification [1].
Each audio sample had 10 PCEN spectrograms, one per rate parameter. Individual
layers/subsets were selected in training and evaluation. Following the
channels-last conventions of Keras, the resulting PCEN spectrograms have the
shape of (128,862,10), and the resulting log-mel spectrograms had a shape
(128,862,1).

###  3.4 Network architecture

The network architecture is inspired by the $L^{3}$ audio sub-network [9] for
discriminative audio-video correspondence embeddings. This architecture has
demonstrated success in its use for classification and predictions at a fine
time-resolution with mel-spectrogram inputs[2]. We follow the implementation
of this architecture found in [2], with the input layer adjusted to accept
multi-layer images to ensure that all methods were being evaluated on near-
identical architecture. Additional parameters in the input layer may lead to
lower performance due to additional depth requirements, but this is assumed to
be negligible for this application.

Models were built and trained using Keras 2.3.1 [10], with Tensorflow 2.2.0
[11]. The model was trained using the Adam optimizer [12], using UrbanSED’s
pre-folded training and validation sets. The loss function was binary cross-
entropy on a per-class per-frame bases. The validation metric was accuracy. If
no improvement in the accuracy was seen in 10 epochs, the learning rate would
be reduced. Early stopping was implemented if no improvement was seen after 30
epochs. Evaluation was handled via the sed_eval package [13], which computed
segment-based classification metrics, including precision, recall, and error
rate. We used the F1-score, the harmonic mean of precision and recall, as our
main metric of effectiveness. These metrics were computer both per-class and
overall. For reproducability, the implementation and experimental framework,
including data preprocessing, model training, and evaluation, is publicly
available on github444https://github.com/ChrisIck/pcen-t-varying.

###  3.5 Data preparation

We trained and evaluated models across multiple datasets to test robustness
and stability to audio deformation. We primarily used the dry dataset as our
baseline dataset, which was UrbanSED with no reverb augmentation, to ensure we
were achieving state of the art performance seen in [2]. The realreverb set is
built from the reverb-augmented audio using the 3 real recorded impulse
responses, and is the primary set we evaluate, as this is closest to real-
world conditions. The simreverb set, using the remaining 3 synthetic impulse
responses, demonstrated the strongest separation between PCEN and log-mel
models, but the results don’t generalize as well as when using real impulse
responses. All models were trained on data augmented by $\\{\pm 1,\pm 2\\}$
pitch-shifts, but were validated and evaluated on non-shifted data.

###  3.6 Rate parameters

We trained and evaluated models on fixed sets of rate parameters. We
experimented with single rate parameter models and varying sets of multi-
layered models, each with rate constants in $T_{k}=2^{k}$ for
$k\in\\{0,1,...,9\\}$. We tested a total of 44 unique rate parameter
configurations for our PCEN models. This included single-rate parameter models
and $n$-layer models including rate parameters from $T_{i}$ to $T_{i+n-1}$. As
a baseline, we also computed a model using a traditional log-scaled mel-
spectrogram, as seen in previous literature for this application and dataset
[2].

###  3.7 Training and evaluation

Each set of models was trained and evaluated on both datasets to see how
stable each model was to reverberation conditions that were both contained in
the respective training dataset, as well as in conditions distinct from those
in the training set. Evaluation was done on bootstrapped subsamples of each
evalution set, sampling 100 evaluation examples with replacement, 100 times
per model and dataset. Evaluations were completed both in overall micro-
averaged performance metrics across classes, as well as on a per-class basis.

##  4 Results

Fig. 2: F1-score of single-rate PCEN models at various $T$ values (blue dots),
compared to 8-layer MRPCEN (dashed green) and log-mel (dotted red) models
evaluated on the 100 bootstrapped samples of the realreverb dataset.

###  4.1 Per-class performance

We demonstrate PCEN’s robustness to noise by comparing performance of models
trained and evaluated on the realreverb dataset. We see in figure 2 that
single rate PCEN demonstrates performance at or above log-scaled spectrograms,
depending on its choice of rate parameter $T$ and the class of the target
event. Certain classes, like the jackhammer and gunshot classes, prefer higher
$T$ values in the neighborhood of $T=2^{7}$ where mid-low frequency is
filtered out with the adaptive gain aspect of PCEN. Similarly, for higher and
full-band frequency stationary events, such as air conditioners and sirens,
lower values of $T$ in the neighborhood of $T\in[2^{1},2^{3}]$ perform better,
as the low-frequency noise doesn’t interfere with the relevant frequency
bands. However, it is crucial to note that due to the diversity of sonic
characteristics of these different classes, there does not exist a single
ideal value of $T$ that produces the strongest performance overall.

###  4.2 Cross-class performance

MRPCEN successfully outperforms most single-rate models in cross-class
performance, as seen in figure 2. The plot shows per-class performance of each
model evaluated on the realreverb dataset, compared to a log-mel based model
and an 8-layer MRPCEN. In comparison, MRPCEN model preserves information at
multiple rates, it consistently performs at or above the majority of the
single rate models. Furthermore, MRPCEN performs at or above the level of the
standard log-mel spectrogram with the exception of the “car_horn” and
“jackhammer” classes, providing it with the highest overall performance metric
on this dataset (compared to the strongest performing single-rate PCEN model
$T=2^{1}$) in table 1.

Model |  Overall F1 (realreverb) |  Overall F1 (simreverb) |  Overall F1 (All Data)
---|---|---|---
Logmel |  0.334 |  0.175 |  0.274
PCEN |  0.345 |  0.167 |  0.268
MRPCEN |  0.356 |  0.204 |  0.285
Table 1: Overall Micro-averaged F1 Scores on the realreverb and simreverb
datasets

##  5 Conclusion

The results here show PCEN as a viable alternative to log-mel spectrograms,
showing equivalent or improved performance depending on rate parameter choice,
which in turn depends on the acoustic characteristics of the target sound
event and acoustic environment. We can also see that MRPCEN provides cross-
class performance improvements over single-rate PCEN models. MRPCEN permits
simultaneous prediction of differing classes with distinct acoustic
characteristics in a single model. In a field setting, this will lead to less
per-class knowledge and tuning expertise needed to effectively deploy a model
that can perform well in multi-class applications.

##  6 Acknowledgements

This work was supported in part through by NSF award 1955357, and in part by
the NYU IT High Performance Computing resources, services, and staff
expertise.

## References

  * [1] Vincent Lostanlen, Kaitlin Palmer, Elly Knight, Christopher Clark, Holger Klinck, Andrew Farnsworth, Tina Wong, Jason Cramer, and Juan Bello,  “Long-distance detection of bioacoustic events with per-channel energy normalization,”  Proceedings of the Detection and Classification of Acoustic Scenes and Events 2019 Workshop (DCASE2019), 2019.
  * [2] B. McFee, J. Salamon, and J. P. Bello,  “Adaptive pooling operators for weakly labeled sound event detection,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 26, no. 11, pp. 2180–2193, 2018.
  * [3] Yuxuan Wang, Pascal Getreuer, Thad Hughes, Richard F. Lyon, and Rif A. Saurous,  “Trainable frontend for robust and far-field keyword spotting,”  CoRR, vol. abs/1607.05666, 2016.
  * [4] Jan Schlüter and Bernhard Lehner,  “Zero-Mean Convolutions for Level-Invariant Singing Voice Detection,”  in Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR 2018), Paris, France, 2018.
  * [5] J. Salamon, D. MacConnell, M. Cartwright, P. Li, and J. P. Bello,  “Scaper: A library for soundscape synthesis and augmentation,”  in 2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), 2017, pp. 344–348.
  * [6] V. Lostanlen, J. Salamon, M. Cartwright, B. McFee, A. Farnsworth, S. Kelling, and J. P. Bello,  “Per-channel energy normalization: Why and how,”  IEEE Signal Processing Letters, vol. 26, no. 1, pp. 39–43, 2019\.
  * [7] Brian McFee, Eric J. Humphrey, and Juan P. Bello,  “A software framework for musical data augmentation,”  in 16th International Society for Music Information Retrieval Conference, 2015, ISMIR.
  * [8] Brian McFee, Colin Raffel, Dawen Liang, Daniel P.W. Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto,  “librosa: Audio and Music Signal Analysis in Python,”  in Proceedings of the 14th Python in Science Conference, Kathryn Huff and James Bergstra, Eds., 2015, pp. 18 – 24\.
  * [9] Relja Arandjelovic and Andrew Zisserman,  “Look, listen and learn,”  CoRR, vol. abs/1705.08168, 2017.
  * [10] François Chollet et al.,  “Keras,”, 2015.
  * [11] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng,  “TensorFlow: Large-scale machine learning on heterogeneous systems,” 2015,  Software available from tensorflow.org.
  * [12] Diederik P. Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,” 2014,  cite arxiv:1412.6980Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
  * [13] A. Mesaros, T. Heittola, and T. Virtanen,  “Metrics for polyphonic sound event detection,”  June 2016.


# Surface lattice resonance lasers with epitaxial InP gain medium

Anna Fischer∗ Blackett Laboratory, Department of Physics, Imperial College
London, London, UK  IBM Research Europe - Zürich, Säumerstrasse 4, Rüschlikon,
8803, Switzerland  Toby Severs Millard∗ Blackett Laboratory, Department of
Physics, Imperial College London, London, UK  National Physical Laboratory,
Teddington TW11 0LW, United Kingdom  Xiaofei Xiao  Blackett Laboratory,
Department of Physics, Imperial College London, London, UK  T.V. Raziman
Blackett Laboratory, Department of Physics, Imperial College London, London,
UK  Department of Mathematics, Imperial College London, London, UK  Jakub
Dranczewski  Blackett Laboratory, Department of Physics, Imperial College
London, London, UK  IBM Research Europe - Zürich, Säumerstrasse 4, Rüschlikon,
8803, Switzerland  Ross C. Schofield  Blackett Laboratory, Department of
Physics, Imperial College London, London, UK  Heinz Schmid  IBM Research
Europe - Zürich, Säumerstrasse 4, Rüschlikon, 8803, Switzerland  Kirsten
Moselund  Paul Scherrer Institut, Forschungsstrasse 111, Villigen, 5232,
Switzerland  EPFL, Lausanne, 1015, Switzerland  Riccardo Sapienza  Blackett
Laboratory, Department of Physics, Imperial College London, London, UK  Rupert
F. Oulton  Blackett Laboratory, Department of Physics, Imperial College
London, London, UK

###### Abstract

∗ These authors contributed equally to the work.

Surface lattice resonance (SLR) lasers, where gain is supplied by a thin film
active material and the feedback comes from multiple scattering by plasmonic
nanoparticles, have shown both low threshold lasing and tunability of the
angular and spectral emission. However, typically used materials such as
organic dyes and QD films suffer from photo-degradation which hampers
practical applications. Here, we demonstrate photo-stable single-mode lasing
of SLR modes sustained in an epitaxial solid-state InP slab waveguide. The
nanoparticle array is weakly coupled to the optical modes, which decreases the
scattering losses and hence the experimental lasing threshold is as low as 90
$\mu$J/cm2. The nanoparticle periodicity defines the lasing wavelength and
enables tuneable emission wavelengths over a 70 nm spectral range. Combining
plasmonic nanoparticles with an epitaxial solid-state gain medium paves the
way for large-area on-chip integrated SLR lasers for applications including
optical communication, optical computing, sensing, and LiDAR.

## Introduction

On-chip lasers in combination with integrated photonic circuits are promising
for the next generation of optical communication systems [1, 2], chemical and
biological sensors [3], optical computation [4, 5, 6], and LiDAR systems [7].
While Si based photonic integrated circuits benefit from low-cost, large-scale
production while offering high speeds, band-width and efficiency, they are
limited by the integration of suitable gain media [2, 8].

Surface lattice resonance (SLR) lasers are of rising interest, as they show
large-area, directional, and controllable mode emission [9], as well as broad
post-fabrication control in the form of optical [10] and magnetic [11]
switching, mechanical mode tuning [12] and thermal mode control [13]. SLRs are
formed by the hybridisation of the collective mode of a nanoparticle array and
a diffractive surface wave, forming a cavity [9]. Similar to coupled resonator
optical waveguides [14], they can be engineered to achieve high resonant
quality factors. In-plane SLR arrays, where nanoparticles are polarised in the
sample plane, have been carefully developed to reach quality factors above
$2\times 10^{3}$ that rival those of bound-states in the continuum (BICs)
[15].

Most SLR lasers have so far relied on organic dye molecules as gain media
which are difficult to integrate with solid-state photonic integrated circuits
[16, 17, 18]. Alternatively to dyes, semiconductor quantum dots (QDs) have
been successfully used to achieve low threshold lasing [19, 20, 21]. However,
QDs [19, 20, 21] are subject to photo-degradation, being more susceptible to
oxidation due to their large surface area. This is a well known issue for
nanolaser devices, with organic dye molecules [16, 17, 18] and perovskites
[22, 23] degrading similarly [24]. A solid-state SLR laser with lanthanide-
based upconverting nanoparticle gain medium has been demonstrated with
continuous-wave pumping, although, the nanoparticle film limits integration
with Si platforms [25].

Here, we introduce the first SLR laser with solid-state epitaxial
semiconductor gain medium that can be incorporated defect-free on-chip through
wafer-bonding of InP to SiO2 on Si [26]. We place the nanoparticle outside of
the gain medium waveguide forming a plasmonic-photonic hybrid SLR mode [14].
The cylindrical plasmonic resonators are suspended in SiO2 and coupled to a
slab waveguide of SiO2/InP/SiO2. This has the benefit of optically accessing
the InP gain medium without complicated fabrication, and reduces plasmonic
absorption losses. We demonstrate low threshold, single-mode emission and
spectral control of devices across large lateral areas, and further show that
InP is robust to photo-degradation under prolonged pumping.

## Results

Figure 1: Single-mode lasing from the Au nanoparticle array on InP (a)
Graphical illustration of the sample showing the SiO2 substrate (dark purple),
InP layer (150 nm, light purple), SiO2 layer (110 nm, transparent dark purple)
with embeded Au nanoparticle array (gold). (b) SEM image of the Au
nanoparticle array on InP with nanoparticle diameters of 60 nm and a period of
308 nm. The inset shows the same device at a higher magnification. (c) The
photoluminescence and lasing spectra from the array in (b) at different pump
fluences, indicating single-mode lasing. (d) Integrated intensity versus pump
fluence (LL-curve) in logarithmic scale (left axis, blue) showing a
characteristic S-shape with a rate equation based fit (black line). The right
axis shows the narrowing of the FWHM upon lasing (orange).

The SLR laser is composed of a slab waveguide of 150 nm thick InP on a Si/SiO2
substrate with a square 100x100 $\mu$m array of cylindrical Au nanoparticles,
suspended in the second layer of 110 nm thick SiO2, 50 nm above the InP.
Figure 1a shows a graphical representation of a typical SLR laser device
measured, with a 2D cross-section shown in the inset. An SEM image of a
nanoparticle array with 60 nm diameter and a period of 310 nm is shown in
Figure 1b.

Normally incident laser light at 633 nm excites the InP significantly above
the 918 nm room temperature bandgap [27] and photoluminescence (PL) is
subsequently emitted. A portion of the PL that is emitted in the sample plane
is contained within a waveguide mode through coupling to the SLR mode created
by the nanoparticle array. In turn, the gain medium experiences population
inversion, for the wavelength of light trapped in said mode, and begins to
lase. The 50 nm SiO2 spacing layer between the nanoparticles and InP acts to
control the degree of coupling, allowing for optimisation of the laser
process. Laser light is generated in the SLR mode, which couples to the far
field via scattering from the nanoparticle array. From this qualitative
description of the device mechanism, it is intuitive that the properties of
the lasing mode can be tuned via appropriate tailoring of the InP waveguide
gap, the SiO2 spacing layer and SLR array dimensions. A detailed description
of the fabrication can be found in the Methods section.

Figure 2: Array period defines lasing peak wavelength. (a) Peak wavelength
versus Au nanoparticle array period for different nanoparticle diameters shows
how the lasing mode red-shifts for increased periods. The plotted wavelengths
are the peak wavelengths of devices just above threshold. The black lines are
obtained from analytical theory with an InP thickness of 165 nm and a carrier-
induced refractive increase of 10%. The gray shaded area gives a range of
possible analytical values for InP thicknesses between 150-165 nm, and
refractive index changes of 8-13 %. (b) Lasing threshold versus period for
arrays of nanoparticles with different diameters, showing lowest threshold for
75 and 60 nm diameter particles.

On exciting the device from figure 1b with increased fluence, the emission
spectra develop a sharp single peak, as shown in figure 1c. Simultaneously,
the broad PL emission increases linearly with fluence and red shifts due to
localised Varshni heating [28] of InP. The transition to lasing is
characterised by the appearance of the single sharp peak within the PL band,
with an amplitude that increases non-linearly with pump fluence. The threshold
of this lasing peak is clearly shown in the logarithmic LL curve of figure 1d;
highlighted by a higher order dependence of the integrated intensity (blue) on
fluence and a clear drop in the FWHM (orange) of the highest intensity peak.
The grey dashed line and box mark the lasing threshold of 94.99 $\pm$ 0.82 uJ
cm-2 pulse-1—the lowest threshold seen across the devices—extracted by fitting
rate equations that account for both spontaneous and stimulated emission [29].
It is important to note here that the intensity has been integrated over a 60
nm window around the spectral position of the laser peak; a necessary filter
to observe lasing against the broad PL emission. The observed threshold range
is comparable to the lowest reported SLR laser threshold of 19 $\mu$J cm-2
pulse-1 coming from perovskite QDs [23]. Other gain media with comparable
arrays show higher thresholds, with organic dyes ranging between 300 - 1500
$\mu$J cm-2 pulse-1 [10, 16], core-shell QDs between 160 - 1000 [21, 19], and
lanthanide-based nanoparticles show thresholds of 400 $\mu$J cm-2 pulse-1
under pulsed-excitation [25].

We estimate the quality factor from the mode linewidth of this device to be
around 800 comparing well with other plasmonic in-plane SLR cavities [15].
This may under-represent the actual performance of the cavity, as wavelength
chirping from the optical modulation of the semiconductor carrier density
introduces artificial broadening. Additionally, the PL emission of InP
exhibits a blue shift due to a significant increase in carrier density under
high pump fluence, further contributing to broadening [30, 31, 32]. Despite
the PL shift, the SLR keeps the lasing peak position well-anchored with only a
<2 nm blue shift; comparable to the 1.1 nm FWHM.

Figure 3: Threshold and mode dependence on pump beam radius and photo-
stability of InP gain medium to prolonged pumping.  (a) Lasing threshold
versus pump radius for a device with 60 nm diameter nanoparticles and a period
of 308 nm. The threshold converges to a minimum value of 94.69 $\pm$ 2.22
$\mu$J cm-2 pulse-1 with increased pump radius. (b) Above-threshold spectra
for the same device as in (a) showing single mode emission for pump radii
larger than the characteristic cavity size and multimode emission below. (c)
Prolonged pumping over two hours (7x108 pulses) showing lasing intensity to be
photo-stable within 6.1 % (7.1 %) for pump diameter of 36.5 µm (46.4 µm).

Here, the hybrid SLR and slab waveguide mode defines the lasing wavelength,
where it is dependent on the array period, nanoparticle diameter, and the InP
thickness.

The broad PL region of InP allows the lasing mode to be tuned over a spectral
range of $\sim$70 nm, with the laser emission red-shifting with increasing
diameter and period; as shown by the peak wavelengths just above threshold in
figure 2a. As the period increases to 290 nm the SLR mode red-shifts beyond
the InP PL region and another mode appears to lase at shorter wavelengths. We
can distinguish three modes across our devices; two more prominent modes and a
third mode in-between these (Figure 2a). This middle mode is most prominent at
periods of $\sim$290 nm. For these periods, the wavelengths of the two main
modes are at either edge of the InP gain region, which is centered around 900
nm. As the main modes experience less gain, the lossier third mode can reach
threshold due to lower gain competition. At higher pump powers, the middle
mode also reaches threshold for other periods (SI figure 1).

To theoretically describe the two main modes, we analytically calculate the
grating condition for coupling to the TE and TM modes of the system assuming
normal emission of the modes, as is typical for SLR lasers with weak coupling
[16] (see Supporting Information SII).This analytical method was validated by
comparison with finite element method (FEM) simulations. To obtain agreement
between the theoretical prediction and the experimental observation of the
mode wavelengths, we tuned the values for the thickness and the refractive
index of the InP waveguide around the expected values. Increasing both
properties by 10% gives a good overlap with the experimental results (black
lines in figure 2a). A 10% mismatch in waveguide thickness can likely occur
during the MOCVD growth of the InP layer. The increase in refractive index can
be explained due to band filling and plasma dispersion effects typical for
semiconductors at high carrier densities [30]. These carrier-induced changes
in refractive index are reinforced by the very short pump pulses and hence
high peak powers used in our experiments.

We use finite-difference time-domain simulations to consider the bandstructure
of a typical device with increasing $k_{\parallel}$ and observe an
intersection of the TE and TM modes that occurs at a higher angle (see
Supporting Information SIII). This crossing leads to the formation of a third
mode with expected oblique emission. Tuneable directional emission has
previously been enabled through lattice engineering, which moves high-symmetry
point modes into the gain region of the active material [19]. In our work, the
presence of distinct TE and TM modes due to the hybrid SLR and waveguide mode
leads to additional crossings in-between symmetry points and hence closer
spaced cavity modes.

The scattering and absorption by the plasmonic nanoparticles are strongly
dependent on their size, leading to lower lasing thresholds for devices with
60 and 75 nm diameter cylindrical nanoparticles, compared to 100 and 125 nm.
The reflectance peaks of nanoparticles with diameters of 100 and 125 nm are
centred at $\sim$600 and $\sim$640 nm, respectively. The spectral overlap with
the 633 nm pump laser reduces the amount of light reaching the InP, increasing
the lasing threshold. Given the broadband nature of plasmonic resonances,
there is also increased absorption at the the operating wavelengths of the
devices. This reduces the quality factor of the cavity and increases the
lasing threshold. Simultaneously, larger particles scatter light more
strongly, allowing light to escape the cavity more easily. While the devices
with 75 nm and 60 nm nanoparticles show similar low thresholds for the TE SLR
mode, the 60 nm nanoparticle devices show no lasing in the TM SLR mode for
periods below 290 nm. In this range, resonant modes and amplified spontaneous
emission (ASE) were observed, but the devices did not reach a laser threshold.
This is due to the in-plane polarising field of the TE mode coupling better to
the nanoparticles than the TM mode. One should note that the threshold cannot
be reduced indefinitely by reducing particle size. Minimising absorption but
maintaining sufficient scattering are both important. Besides the threshold
dependence on nanoparticle diameter, there is also a dependence on period. For
devices with periods around 290 nm, the thresholds show a small increase. This
increase is likely due to gain competition between the modes as periods in
this region support multimode operation (Figure 2a,b). For periods $>$295 nm
the threshold decreases due to reduced absorption in InP as the lasing mode
red-shifts [30].

Theoretically, SLR laser arrays are considered to be infinite in size in order
to simplify the system analysis. However, experiments are limited by the pump
beam radius used to excite the array. By increasing the beam radius the
characteristic properties of the laser converge to values that would be
achieved in an infinite system. Figure 3 a and b demonstrate this convergence
for lasing threshold and number of lasing modes, respectively. Within the
errors of the experimental data, the device can be said to have converged to a
threshold value of 94.69 $\pm$ 2.22 $\mu$J cm-2 pulse-1 at radii $\geq$ 29.85
$\mu$m, thereby defining the minimum effective cavity diameter of the laser.
For this estimation the larger error bar on the 24.88 $\mu$m measurement has
been ignored as an outlier. This effective cavity approximation is a measure
of the coherence length of the particle-waveguide interaction of the SLR modes
and shows a weak interaction over $\sim$100 nanoparticles.

Spectra highlighting the development of the lasing mode with increased pump
radius, figure 3b, shows reasonable agreement with our estimate for the
effective cavity size. At smaller pump radii, where lasing is seen, there is
typically a second and third peak competing for gain in the system. As the
pump radius increases from 20 $\mu$m these extra peaks diminish and the system
becomes single mode from 26.54 $\mu$m. Further increasing the pump spot radius
results in no change to the spectral shape of emission. For smaller pump spot
radii, the inset of figure 3b shows an observable blue shift and the emergence
of multiple modes. These features, alongside increased spacing between these
multiple modes with decreasing pump spot radius, are consistent with the
presence of lateral confinement linked to pump spot size [14]. The origin of
the lateral confinement is likely to be carrier density-induced nonlinear
refraction, where the centre of the pump beam has a higher refractive index
than surrounding un-pumped regions. The potential created by the refractive
index gradient leads to the quantisation of the cavity and multiple equally
spaced modes

A critical limitation of previously demonstrated SLR lasers is their
susceptibility to photo-bleaching under prolonged operation. A recent study on
the photo-stability of liquid crystal gain media characterised multiple
materials including those commonly used for SLR lasers, such as organic
fluorescent dyes, perovskites, CdSe/ZnS quantum dots and CdSe/CdS quantum rods
[24]. Of these, the largest photo-stable half-life observed was 3.8x109 pulses
at 0.403 GW/cm2 peak power density for CdSe/CdS quantum rods. Epitaxial bulk
semiconductors are not expected to suffer the same degradation, as is
indicated by the photo-stability plot in figure 3 (c). Here, a 200 fs laser
was used to deliver 7x108 pulses over two hours to the 60 nm diameter 308
period device. The measurement was taken twice at 36.5 $\mu$m and 46.4 $\mu$m
pump laser radii and at 0.516 and 0.472 GW/cm2 peak power densities,
respectively. The SLR lasing intensity is normalised to the pump power density
at each measurement to remove the effect of input laser power fluctuations.
The resulting two data sets show no signs of reduced performance and have only
a 6.2% (36.5 $\mu$m) and 7.1% (46.4 $\mu$m) standard deviation from their
respective means. In comparison, the emission intensity of the CdSe/CdS device
mentioned above, has already dropped by $~{}40\%$ for this number of pulses.
This highlights the extremely stable lasing performance for our InP waveguide-
coupled SLR laser.

## Conclusion

In conclusion, we have demonstrated an SLR laser with an epitaxial solid-state
active material of InP and a novel design that delocalises the cavity from
plasmonic nanoparticles. This device structure is theoretically optimised for
efficient coupling and reduced absorption losses from the plasmonic
nanoparticles. The SLR lasers show large-area and single-mode emission with
low thresholds down to 94 $\mu$J cm-2 pulse-1. The emission wavelength is
tuneable over a broad wavelength range of 70 nm through variation of the
nanoparticle array period. We theoretically describe the different TE and TM
modes of the system both through analytical calculations and numerical
simulations. We obtained an effective cavity radius of 30 $\mu$m by varying
the optical pump beam size indicating the cavity coherence length. When
pumping the device with a beam smaller than the effective cavity, lasing
requires higher thresholds leading to a refractive index gradient resulting in
mode quantisation. The semiconductor gain material is not affected by
photobleaching, showing stable emission intensity up to 7x108 excitation
pulses. The plasmonic-photonic hybrid SLR forms the foundation for further
lattice designs and advanced band-edge engineering without relying on thin
films of gain material. Our devices pave the way for on-chip SLR lasing
without degradation for applications in optical communication, optical
computing and LiDAR.

## Methods

### Optical measurements

The SLR lasers are characterised in a micro photoluminescence setup. They are
pumped with a 633 nm, 200 fs-pulsed laser at 100 kHz. The pump beam excites
the sample from the top and is focused through a 20x objective. The emitted
light is collected in reflection through the same objective. The emission
spectra are measured in a Princeton Instruments spectrometer. We used a
digital micromirror device (DMD) to pump the devices with variable pump
diameters for the data in Figure 3.

### Sample fabrication

InP bonding and SiO2 spacing layer. The active InP layer is integrated on a Si
wafer through direct wafer bonding [26]. The layer is grown on a sacrificial,
lattice-matched III/V wafer, and then bonded to an Si wafer through an
adhesion layer of oxide (which becomes the BOX) and annealing. The sacrificial
wafer can then be removed by wet etching. The 50 nm SiO2 layer separating InP
from the gold particles is deposited with Plasma-Enhanced Chemical Vapor
Deposition.

Au nanoparticle fabrication. Two layers of resist, polymethyl methacrylate
(PMMA) A4 495K and A4 950K, are spun onto the SiO2 spacing layer
consecutively, both at 4000 RPM for 1 minute and each layer baked on a
hotplate for 18 minutes at 180oC. A further layer of e-spacer was spun at 2000
RPM for 1 minute with a 30 s bake at 90oC. The resist is exposed to a desired
pattern in an electron beam lithography system (EBL, Raith E-line plus). The
PMMA is developed in a 3:1 Isopropyl alcohol (IPA) to Methyl isobutyl ketone
(MIBK) solution, followed by a 30 rinse in IPA to halt development. Residual
PMMA is removed via plasma ashing at 40% power for 6 s. A 1.5 nm Cr adhesion
layer is thermally evaporated followed by a 50 nm layer of Au (Angstrom
Engineering Amod). The PMMA and unwanted Cr/Au is removed in the lift-off
process, where the sample is left in Acetone for 24 hours. Finally, 60 nm of
SiO2 is sputtered (Angstrom Engineering Amod) onto the Au and spacing layer
SiO2 to encapsulate the Au and create the superstrate. The full set of 36
devices measured is comprised of arrays with periods = 269 $\pm$ 3, 273 $\pm$
2, 279 $\pm$ 3, 284 $\pm$ 3, 288 $\pm$ 4, 293 $\pm$ 6, 299 $\pm$ 4, 304 $\pm$
6 and 308 $\pm$ 6; each with nanoparticle diameters = 58 $\pm$ 8, 74 $\pm$ 6,
99 $\pm$ 6 and 126 $\pm$ 5.

### Theory

The lasing modes were analytically calculated using the grating equation, and
validated through comparison with numerical simulations using FEM. When
assuming normal emission, which is most likely for simple SLRs, and for weak
coupling, a simple formula for the mode wavelength can be deduced: $\pm
n_{\mathrm{eff}}k_{0}=2\pi/P$, where $n_{\mathrm{eff}}$ denotes the the
effective refractive index for excited mode in the slab waveguide, $k_{0}$
denotes the wavenumber of the free space light, and $P$ denotes the period
length. The effective refractive index $n_{\mathrm{eff}}$ is numerically
calculated using a finite-difference frequency-domain method. To place the
theoretically predicted mode wavelengths in a range similar to the experiment,
the thickness of the InP layer and the refractive index of the InP were
varied. Moreover, we performed finite-difference time-domain simulations with
ANSYS Lumerical, to obtain the dispersion of the modes of the system and
explain additional emission modes. More details can be found in the supporting
information.

### Acknowledgments

AF, JD, HS, and KM acknowledge support from the EU ITN EID project CORAL (GA
no. 859841). T.S.M. acknowledges the support of the UK government Department
for Science, Innovation and Technology through the UK National Quantum
Technologies Programme. This work is supported by the UK Engineering and
Physical Sciences Research Council (EPSRC), through grants: EP/S000755/1,
EP/M013812/1 and EP/T027258. We thank the Cleanroom Operations Team of the
Binnig and Rohrer Nanotechnology Center (BRNC) for help with sample
fabrication.

### Author Contributions

T.S.M. and J.D. fabricated the devices. A.F. and T.S.M. took optical
measurements. X.X. and T.V.R. developed the theory and carried out
simulations. T.S.M. and A.F. wrote the paper with contributions from all co-
authors. H.S., K.M., R.F.O. and R.S. conceived and supervised the project.
T.S.M. and A.F. contributed equally to this work.

## References

  * Wang  _et al._ [2017] Z. Wang, A. Abbasi, U. Dave, A. De Groote, S. Kumari, B. Kunert, C. Merckling, M. Pantouvaki, Y. Shi, B. Tian, K. Van Gasse, J. Verbist, R. Wang, W. Xie, J. Zhang, Y. Zhu, J. Bauwelinck, X. Yin, Z. Hens, J. Van Campenhout, B. Kuyken, R. Baets, G. Morthier, D. Van Thourhout, and G. Roelkens, Novel Light Source Integration Approaches for Silicon Photonics, Laser & Photonics Reviews 11, 1700063 (2017).
  * Li  _et al._ [2022] N. Li, G. Chen, D. K. T. Ng, L. W. Lim, J. Xue, C. P. Ho, Y. H. Fu, and L. Y. T. Lee, Integrated Lasers on Silicon at Communication Wavelength: A Progress Review, Advanced Optical Materials 10, 2201008 (2022).
  * Altug  _et al._ [2022] H. Altug, S.-H. Oh, S. A. Maier, and J. Homola, Advances and applications of nanophotonic biosensors, Nature Nanotechnology 17, 5–16 (2022).
  * Feldmann  _et al._ [2021] J. Feldmann, N. Youngblood, M. Karpov, H. Gehring, X. Li, M. Stappers, M. Le Gallo, X. Fu, A. Lukashchuk, A. S. Raja, J. Liu, C. D. Wright, A. Sebastian, T. J. Kippenberg, W. H. P. Pernice, and H. Bhaskaran, Parallel convolutional processing using an integrated photonic tensor core, Nature 589, 52 (2021).
  * Skalli  _et al._ [2022] A. Skalli, J. Robertson, D. Owen-Newns, M. Hejda, X. Porte, S. Reitzenstein, A. Hurtado, and D. Brunner, Photonic neuromorphic computing using vertical cavity semiconductor lasers, Optical Materials Express 12, 2395 (2022).
  * Sludds  _et al._ [2022] A. Sludds, S. Bandyopadhyay, Z. Chen, Z. Zhong, J. Cochrane, L. Bernstein, D. Bunandar, P. B. Dixon, S. A. Hamilton, M. Streshinsky, A. Novack, T. Baehr-Jones, M. Hochberg, M. Ghobadi, R. Hamerly, and D. Englund, Delocalized photonic deep learning on the internet’s edge, Science 378, 270–276 (2022).
  * Zhou  _et al._ [2023] Z. Zhou, X. Ou, Y. Fang, E. Alkhazraji, R. Xu, Y. Wan, and J. E. Bowers, Prospects and applications of on-chip lasers, eLight 3, 1 (2023).
  * Yang  _et al._ [2023] J. Yang, M. Tang, S. Chen, and H. Liu, From past to future: on-chip laser sources for photonic integrated circuits, Light: Science & Applications 12, 10.1038/s41377-022-01006-0 (2023).
  * Kravets  _et al._ [2018] V. G. Kravets, A. V. Kabashin, W. L. Barnes, and A. N. Grigorenko, Plasmonic surface lattice resonances: A review of properties and applications, Chemical Reviews 118, 5912–5951 (2018).
  * Taskinen  _et al._ [2020] J. M. Taskinen, A. J. Moilanen, H. Rekola, K. Kuntze, A. Priimagi, P. Törmä, and T. K. Hakala, All-optical emission control and lasing in plasmonic lattices, ACS Photonics 7, 2850–2858 (2020).
  * Freire-Fernández  _et al._ [2022] F. Freire-Fernández, J. Cuerda, K. S. Daskalakis, S. Perumbilavil, J.-P. Martikainen, K. Arjas, P. Törmä, and S. van Dijken, Magnetic on–off switching of a plasmonic laser, Nature Photonics 16, 27 (2022).
  * Gupta  _et al._ [2019] V. Gupta, P. T. Probst, F. R. Goßler, A. M. Steiner, J. Schubert, Y. Brasse, T. A. F. Konig, and A. Fery, Mechanotunable surface lattice resonances in the visible optical range by soft lithography templates and directed self-assembly, ACS applied materials & interfaces 11, 28189–28196 (2019).
  * Kelavuori  _et al._ [2022] J. Kelavuori, V. Vanyukov, T. Stolt, P. Karvinen, H. Rekola, T. K. Hakala, and M. J. Huttunen, Thermal control of plasmonic surface lattice resonances, Nano Letters 22, 3879–3883 (2022).
  * Wood  _et al._ [2015] J. J. Wood, L. Lafone, J. M. Hamm, O. Hess, and R. F. Oulton, Plasmonic crows for tunable dispersion and high quality cavity modes, Scientific Reports 5, 17724 (2015).
  * Bin-Alam  _et al._ [2021] M. S. Bin-Alam, O. Reshef, Y. Mamchur, M. Z. Alam, G. Carlow, J. Upham, B. T. Sullivan, J.-M. Ménard, M. J. Huttunen, R. W. Boyd, and K. Dolgaleva, Ultra-high-q resonances in plasmonic metasurfaces, Nature Communications 12, 974 (2021).
  * Schokker and Koenderink [2014] A. H. Schokker and A. F. Koenderink, Lasing at the band edges of plasmonic lattices, Physical Review B 90, 155452 (2014).
  * Wang  _et al._ [2018] D. Wang, M. R. Bourgeois, W.-K. Lee, R. Li, D. Trivedi, M. P. Knudson, W. Wang, G. C. Schatz, and T. W. Odom, Stretchable nanolasing from hybrid quadrupole plasmons, Nano Letters 18, 4549–4555 (2018).
  * Pourjamal  _et al._ [2019] S. Pourjamal, T. K. Hakala, M. Nečada, F. Freire-Fernández, M. Kataja, H. Rekola, J.-P. Martikainen, P. Törmä, and S. van Dijken, Lasing in ni nanodisk arrays, ACS nano 13, 5686–5692 (2019).
  * Guan  _et al._ [2020a] J. Guan, L. K. Sagar, R. Li, D. Wang, G. Bappi, N. E. Watkins, M. R. Bourgeois, L. Levina, F. Fan, S. Hoogland, O. Voznyy, J. Martins de Pina, R. D. Schaller, G. C. Schatz, E. H. Sargent, and T. W. Odom, Engineering directionality in quantum dot shell lasing using plasmonic lattices, Nano Letters 20, 1468 (2020a).
  * Guan  _et al._ [2020b] J. Guan, L. K. Sagar, R. Li, D. Wang, G. Bappi, W. Wang, N. Watkins, M. R. Bourgeois, L. Levina, F. Fan, S. Hoogland, O. Voznyy, J. M. de Pina, R. D. Schaller, G. C. Schatz, E. H. Sargent, and T. W. Odom, Quantum dot-plasmon lasing with controlled polarization patterns, ACS Nano 14, 3426 (2020b).
  * Winkler  _et al._ [2020] J. M. Winkler, M. J. Ruckriegel, H. Rojo, R. C. Keitel, E. De Leo, F. T. Rabouw, and D. J. Norris, Dual-wavelength lasing in quantum-dot plasmonic lattice lasers, ACS Nano 14, 5223 (2020).
  * Huang  _et al._ [2021] Z. Huang, C. Yin, Y. Hong, H. Li, K. Hong, T. S. Kao, M. Shih, and T. Lu, Hybrid plasmonic surface lattice resonance perovskite lasers on silver nanoparticle arrays, Advanced Optical Materials 9, 2100299 (2021).
  * Xing  _et al._ [2022] D. Xing, C.-C. Lin, Y.-L. Ho, Y.-C. Lee, M.-H. Chen, B.-W. Lin, C.-W. Chen, and J.-J. Delaunay, Ligand engineering and recrystallization of perovskite quantum-dot thin film for low-threshold plasmonic lattice laser, Small 18, 2204070 (2022).
  * Vellaichamy  _et al._ [2023] M. Vellaichamy, M. Škarabot, and I. Muševič, Optical gain and photo-bleaching of organic dyes, quantum dots, perovskite nanoplatelets and nanodiamonds, Liquid Crystals 0, 1 (2023).
  * Fernandez-Bravo  _et al._ [2019] A. Fernandez-Bravo, D. Wang, E. S. Barnard, A. Teitelboim, C. Tajon, J. Guan, G. C. Schatz, B. E. Cohen, E. M. Chan, P. J. Schuck, and T. W. Odom, Ultralow-threshold, continuous-wave upconverting lasing from subwavelength plasmons, Nature Materials 18, 1172–1176 (2019).
  * Caimi  _et al._ [2021] D. Caimi, P. Tiwari, M. Sousa, K. E. Moselund, and C. B. Zota, Heterogeneous Integration of III–V Materials by Direct Wafer Bonding for High-Performance Electronics and Optoelectronics, IEEE Transactions on Electron Devices 68, 3149 (2021).
  * Pavesi  _et al._ [1991] L. Pavesi, F. Piazza, A. Rudra, J. F. Carlin, and M. Ilegems, Temperature dependence of the inp band gap from a photoluminescence study, Phys. Rev. B 44, 9052 (1991).
  * Varshni [1967] Y. Varshni, Temperature dependence of the energy gap in semiconductors, Physica 34, 149 (1967).
  * Schofield  _et al._ [2023] R. C. Schofield, M. Fu, E. Clarke, I. Farrer, A. Trapalis, H. S. Dhar, R. Mukherjee, J. Heffernan, F. Mintert, R. A. Nyman, and R. F. Oulton, Bose-einstein condensation of light in a semiconductor quantum well microcavity (2023).
  * Bennett  _et al._ [1990] B. R. Bennett, R. A. Soref, and J. A. Del Alamo, Carrier-induced change in refractive index of inp, gaas, and ingaasp, IEEE Journal of Quantum Electronics 26, 113 (1990).
  * Tiwari  _et al._ [2021] P. Tiwari, P. Wen, D. Caimi, S. Mauthe, N. V. Triviño, M. Sousa, and K. E. Moselund, Scaling of metal-clad inp nanodisk lasers: optical performance and thermal effects, Optics Express 29, 3915 (2021).
  * Tiwari  _et al._ [2022] P. Tiwari, A. Fischer, M. Scherrer, D. Caimi, H. Schmid, and K. E. Moselund, Single-mode emission in inp microdisks on si using au antenna, ACS Photonics 9, 1218 (2022).