# Long Dialog Summarization: An Analysis

1Ankan Mullick 2Ayan Kumar Bhowmick 3 Raghav R 4 Ravi Kokku
2 Prasenjit Dey 1Pawan Goyal 1Niloy Ganguly
1Department of Computer Science and Engineering, IIT Kharagpur
2Merlyn Mind Inc. 3 Carnegie Mellon University, USA
<EMAIL_ADDRESS>{2ayan, 2prasenjit<EMAIL_ADDRESS>
<EMAIL_ADDRESS>{1pawang<EMAIL_ADDRESS>
###### Abstract

Dialog summarization has become increasingly important in managing and
comprehending large-scale conversations across various domains. This task
presents unique challenges in capturing the key points, context, and nuances
of multi-turn long conversations for summarization. It is worth noting that
the summarization techniques may vary based on specific requirements such as
in a shopping-chatbot scenario, the dialog summary helps to learn user
preferences, whereas in the case of a customer call center, the summary may
involve the problem attributes that a user specified, and the final resolution
provided. This work emphasizes the significance of creating coherent and
contextually rich summaries for effective communication in various
applications. We explore current state-of-the-art approaches for long dialog
summarization in different domains and benchmark metrics based evaluations
show that one single model does not perform well across various areas for
distinct summarization tasks.

##  1 Introduction

Dialog summarization has emerged as a crucial aspect of managing and
understanding large-scale conversations in various contexts such as online
tutoring Jain et al. (2023), customer service Liu et al. (2019), patient
consultations Abacha et al. (2023); Joshi et al. (2020), and casual chatbot
interactions. While the automatic summarization of text remains inherently
challenging due to the complexity of determining and preserving the most
relevant content, this paper aims to devise goal-oriented summarization and
how current state of the art language models can be utilised effectively to
achieve superior performance in the domain of large-scale dialog
summarization.

Summarization will vary according to the requirements - it can be a context
driven summary or an object oriented summary as in Fig 1. In the former case,
in a dialogue session about technical smartphone issues, a context-driven
summary would capture the sequential troubleshooting steps and relevant device
details, providing a concise overview of the problem-solving process. In the
object driven summary, in a travel planning conversation, an objective-driven
summary would prioritize the user’s goal of obtaining the best itinerary
within specified criteria. The summary would highlight suitable destinations,
recommended accommodations, transportation options, tailored to the user’s
preferences and constraints. Thus, the same summarization approach may not
lead to the best summaries which will be very useful according to the user’s
need and leads us to achieve the goal using rubric-driven summarization with
the use of custom-trained language models.

Figure 1: Different Summarization Process

The cornerstone of our approach involves specifying context-driven rubrics in
a natural language format, which can guide the fine-tuning of large language
models (LLMs) for better goal-oriented summarization performance. As these
rubrics address the nuances of different dialog settings and user groups, the
fine-tuned models are expected to generate summaries that exhibit a deeper
understanding of the conversation, capturing critical information in a
coherent and contextually relevant manner. In essence, this paper delves into
the foundational objective of enriching the dialog summarization task and
create LLM driven solutions that cater to diverse contexts and objectives. By
exploring this direction, our work surpasses traditional summarization
techniques and offers a compelling groundwork for anyone seeking a deeper
comprehension of the intricacies of dialog summarization and its potential
applications.

Earlier researchers explore dialog summarization in different directions. Vig
et al. (2022); Zhang et al. (2022); Zhu et al. (2020) provide different
approaches for conversation summarization tasks. Xiong et al. (2022); Pagnoni
et al. (2022) and FactPegasus Wan and Bansal (2022) study pretraining pipeline
for summarizations. Different state-of-the-art models like LongFormer Beltagy
et al. (2020), BART Lewis et al. (2019), T5 Raffel et al. (2020), Flan T5
Chung et al. (2022) are useful to extract summarized information from dialogs.
Leuski et al. (2003) present interactive summarization with user control.
There exists several benchmarked conversational datasets with summary - QmSum
Zhong et al. (2021), Multiwoz Budzianowski et al. (2018); Eric et al. (2020),
SParC Yu et al. (2019), SumScreen (Tv series summary) Chen et al. (2021),
SQuALITY Pang et al. (2021) etc. Some researchers focus on developing
evaluation criteria to detect the usefulness of different summarization
approaches. ROUGE Lin (2004), BLEU Papineni et al. (2002), Meteor Banerjee and
Lavie (2005), BERTScoreZhang et al. (2019) are state of the art metrics to
assess dialog summaries.

Researchers make impressive advancements on a wide range of downstream
applications for summarizations. Since, one approach is not working best
across different tasks and different domains, the real-world implications of
such advancements are largely unexplored. In this paper, we study utilities of
different summarization approaches for extracting compressed contexts from
domain specific conversations. Our through extensive experiments on two
benchmarked datasets show that different approaches can be useful in different
contexts.

##  2 Dataset

We explore two different datasets for the evaluations: 1) QmSum Zhong et al.
(2021): It is a multi-domain query-based meeting summarization dataset
consisting of 1,808 queries and 232 long meeting transcripts, with different
topic of discussions such as software product, academics, and committee
meetings. 2) SumScreen Chen et al. (2021): This summarization dataset
comprises 26.9K pairs of TV series transcripts (‘Forever Dreaming’ and
‘TvMegaSite’) and human annotated recaps. These TV shows are of different
genres like drama, romance, family, etc. Due to the larger size of the
datasets, we randomly sample 15% of the ‘Forever Dreaming’ data from train,
dev and test set for our experiment purposes.

##  3 Summarization Approaches

We evaluate various state-of-the-art summarization algorithms along with
different settings111Code/Data is available : https://shorturl.at/tyzD8
corresponding to the two datasets discussed above.

###  3.1 Algorithms

LongFormer Beltagy et al. (2020): The Longformer (LF) is a transformer-based
model designed to handle long documents efficiently using an attention pattern
that effectively combines local and global information, enabling to handle
long inputs.

T5 Raffel et al. (2020): “Text-to-Text Transfer Transformer” (T5) model is a
transfer learning technique can be used to generate summary. We use T5-base
model (770 million parameters).

Flan T5 Chung et al. (2022): Flan T5 (FT5) is a instruction fine-tuned
approach that shows finetuning can improve performance across a range of
models, prompting setups, and evaluation tasks.

BART Lewis et al. (2019): BART is a denoising autoencoder for pretraining
sequence-to-sequence models. Pretraining which can generate summary.

Chat-GPT Aydın and Karaarslan (2023); Wang et al. (2023): We apply ChatGPT
(V3.5) to generate summary from dialogs. We apply two different versions of
ChatGPT - (P1): where we feed summary and dialog together as text (P2): where
we feed the summary as prompt and dialog as text. In (P2), we specify the
summary explicitely.

###  3.2 Input Settings

Since, the conversations exceed the maximum token limit of the models, we
follow various settings:

A) Direct: Apply the model directly on the whole dataset. In this setting, the
models automatically truncate the input to the maximum token limit. It is
shown for the case of LongFormer (LF).

B) Chunk and Summarize: In this technique, we tokenize the conversation with
chunks of maximum token limit and feed individual segments to the models in
the following two ways:

i) Feed each conversation segment independently to generate individual
summaries of each segment and finally merge the summaries to get the overall
summary. In Table 1 these methods are shown as method-name with the chunk size
(e.g., LF-8192).

ii) Generate the summary of the first segment and then feed the obtained
summary to the models along with the next conversation segment to generate an
updated summary at each step. Continuing with this approach, finally, we get
the overall summary from the last conversation segment. This approach is used
to preserve the context of a dialog. This is shown as “type-2” in the Table 1.

C) Extract then summarize: In this setting, initially we retrieve sentences
from the dialogs (train set) so that the total token size of extracted
sentences are within the maximum token limit of the models and then feed it to
the summarization frameworks to generate the final summary. The sentences are
extracted using the approach similar to Mukherjee et al. (2022). We apply
SentenceBERT Reimers and Gurevych (2019) to generate the embeddings of the
sentences of dialogues and summaries and then calculate the pair-wise cosine
similarity. Thereafter, we set a minimum similarity threshold to extract
certain dilaog sentences of high similarity ($>$30%) with the summary
sentences. Then, we feed the extracted dialogs to the models to generate
summaries (In Table 1 these are named as Ex-model like ‘Ex-T5’, ‘Ex-FT5’
etc.).

##  4 Experimental Results

In this section, we evaluate the quality of the final summary generated for a
given dialog. For this purpose, we rely on two different approaches, one based
on comparing our generated summary using different methods described in Sec. 3
while the other is based on comparing the coverage of intents and entities in
the original dialog with that of the generated summary.

###  4.1 Ground truth comparison

Here we compare the ground truth summary (reference summary) with the
generated summary obtained from a given model in terms of how closely the
generated summary string matches the ground truth summary string. We measure
the extent of similarity between the two summaries with the help of the
following standard evaluation metrics: 1) BLEU score: BLEU (Bilingual
Evaluation Understudy) metric Papineni et al. (2002) measures the precision of
n-gram overlap between the generated summary and the reference summary. 2)
ROUGE-Score): ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a
set of evaluation metrics widely used in summarization tasks Lin (2004). We
use ROUGE-1 (focuses on the unigram or single-word overlap), ROUGE-2 measures
the bigram (two-word sequence) and ROUGE-L (considering the longest common
subsequence (LCS)). 3) METEOR: METEOR (Metric for Evaluation of Translation
with Explicit ORdering) Banerjee and Lavie (2005) evaluates the generated
summaries that takes into account the harmonic mean of unigram precision and
recall, along with a synonymy matching and sentence structure similarity
component. 4) BERTScore Zhang et al. (2019): It is a robust technique which
computes token similarity using contextual embeddings. It is an useful metric
to evaluate summarization models. 5) Intent-Entity coverage (IEC): To further
evaluate the quality of generated summaries in capturing the key information
from the original dialog conversation, we propose measuring Intent based
entity overlap. These metrics enable a better understanding of how well the
generated summaries preserve the primary objectives and capture salient
aspects from the conversations. We use _ChatGPT_ Aydın and Karaarslan (2023);
Wang et al. (2023) in order to extract the set of entities from a given text
(dialog conversation or summary), by providing suitable prompts to _ChatGPT_.
The overlap in entities is computed as follows: To compute the entity Overlap
based on _ChatGPT_ , we extract the set of entities from both the original
dialog conversation and the generated summary using prompts provided to
ChatGPT. We then compute the overlap between the two entity sets using
_Jaccard similarity_ which measures the proportion of common intents in both
sets to the combined unique intents.

Method QmSum SumScreen (FD) Bl R-1 R-2 R-L Mt BS IEC Bl R-1 R-2 R-L Mt BS IEC
LF 6.09 26.81 7.11 24.40 24.76 58.96 30.81 0.32 15.39 1.42 12.52 3.63 47.20
26.38 LF-8192 3.58 23.13 5.72 21.14 18.29 56.57 29.71 0.27 14.88 1.83 11.74
3.33 43.95 11.94 LF-4096 1.74 22.85 6.01 20.88 15.45 55.93 26.15 0.20 12.73
1.50 12.74 3.55 43.82 13.71 LF-type2 0.50 10.87 1.50 7.01 2.4 25.95 23.83 0.18
14.69 1.05 13.06 3.84 42.91 12.05 Ex-LF 2.49 20.84 3.75 18.37 19.08 50.34
27.15 0.06 17.14 1.45 15.62 3.16 44.79 23.65 T5-4096 2.36 25.00 7.13 23.27
11.55 59.85 32.05 0.35 18.57 1.48 17.62 3.82 44.99 31.87 T5-2048 2.18 24.80
6.27 22.99 11.61 55.42 29.78 0.50 19.14 1.61 18.01 5.76 47.46 29.36 T5-type2
2.62 21.20 3.88 19.80 19.35 55.67 27.42 1.15 16.84 1.22 15.19 5.07 47.33 25.86
Ex-T5 7.40 24.11 7.24 22.15 22.65 54.55 23.71 0.99 17.25 1.41 15.88 13.29
45.41 21.67 FT5-2048 1.76 24.00 5.62 22.55 9.04 55.36 18.35 0.47 16.86 1.41
15.54 5.51 46.61 10.24 FT5-type2 2.62 21.20 3.88 19.80 19.35 57.02 21.12 0.62
16.05 1.47 14.22 4.56 47.33 17.20 Ex-FT5 6.84 23.88 7.29 22.41 22.23 54.51
28.13 0.39 14.66 1.36 13.54 5.24 45.51 27.98 BART-2048 0.90 24.71 5.48 21.06
13.30 47.45 25.74 0.10 19.35 2.02 17.28 8.68 43.88 23.63 BART-type2 1.94 20.49
5.03 19.42 18.31 38.20 17.32 0.39 17.03 1.37 15.54 4.29 44.92 13.84 Ex-BART
3.10 17.29 3.54 15.94 17.36 47.97 21.35 0.44 22.86 1.64 22.86 12.64 45.65
18.75 ChatGPT 0.63 15.74 2.51 13.89 9.61 54.62 25.78 0.40 25.04 4.34 18.71
18.72 57.88 28.65 ChatGPT P1 1.11 17.72 1.67 15.48 20.78 49.69 27.91 0.35
24.82 2.77 21.83 18.74 57.10 30.34 ChatGPT P2 1.00 17.31 2.40 16.97 22.01
50.98 29.12 2.19 22.97 3.11 20.17 20.32 57.33 31.67

Table 1: Results (in %) in terms of BLEU (Bl), Rouge (R-1, R-2 and R-L),
Meteor (Mt), BERTScore (BS) and Intent-Entity Coverage (IEC) on QmSum and
SumScreen (‘Forever Dreaming’) datasets

###  4.2 Results and Discussion

We present our experimental results in terms of the metrics defined above in
Tables 1. From Table 1, we observe a diverse range of metric values across the
different summary generation models, indicating the varying performance
achieved by Longformer, T5, Flan-T5, BART, and ChatGPT in the challenging task
of summarizing long dialog conversation. We apply segmentation in the diloags
to be fed to the models with different token length as input according to
token limits. Different token input lengths are - 8192 for LongFormer (LF);
4096 for for LongFormer (LF) and T5; 2048 for T5, Flan-T5 (FT-5) and BART.
Although metric-wise performance varies, some trends are noticeable. For
instance, BART demonstrates the lowest BLEU score of 0.90, whereas Longformer
attains the highest BLEU score (6.09), ROUGE-1 (26.81) and ROUGE-L (24.40)
values. Conversely, T5 registers the lowest values for ROUGE-1 (21.20) and
ROUGE-L (19.80) but displays the highest ROUGE-2 score, along with Longformer,
at 7.13 and 7.11, respectively. Flan-T5, on the other hand, achieves the
lowest ROUGE-2 score of 3.88. In terms of the METEOR metric, Longformer holds
the highest score at 24.26, while Flan-T5 records the lowest score of 9.04.
For a single approach results also varies for two different datasets. For
QmSum, the important conversational sentences which form the summary are
mostly located in first and middle segment (77.39% dialog sentences) of the
conversations where as for SumScreen-FD data, 70.1% of the important
conversational sentences which form the summary are from first and middle
segment of the dialog. That may explain why LongFormer (LF) without token
limit, works well for QmSum but not for SumScreen-FD. ChatGPT often takes into
account the last segment of the dataset, which may explain why chatGPT
produces better outcomes for SumScreen-FD than QmSum.

These contrasting results suggest that each model exhibits specific strengths
and weaknesses depending on the aspects of summarization being evaluated. The
overall low values for these metrics illustrate the inherent complexity and
difficulty of the long dialog summarization task. A possible intuition for
such observations is that the ability of the Longformer model to handle long-
range contextual information helps it better capture essential content and
overall structure. In contrast, the capability of T5 in maintaining local
coherence and structure might contribute to its higher ROUGE-2 score, although
its lower ROUGE-1 and ROUGE-L scores indicate room for improvement in content
coverage. The lower scores of Flan-T5 may be attributed to limitations in its
architecture or training data used for the task, indicating a need for further
refinement. The varied performance of these models highlights the importance
of choosing the most suitable architecture or ensemble strategy based on the
desired outcome and evaluation metric for dialog summarization tasks.

Compared to models like Longformer, T5, Flan-T5, and BART, ChatGPT may perform
better in maintaining contextual relevance and coherence in dialog
summarization due to its conversational AI focus. In fact, when we explicitly
provide the summary as a promot (P2), ChatGPT performs better in most of the
cases. However, it may struggle to capture complete content from long
dialogues, as it tends to emphasize recent information. This tendency, caused
by limited context window and recency bias, results in summaries focusing on
the latter parts of conversations, potentially missing crucial ideas from
earlier sections.

##  5 Conclusion

In this paper, we explore how different approaches perform across various long
dialog datasets for summarization task. In conclusion, the need for context
and objective-driven summarization is evident, as relying on a single approach
may not yield the best results. This highlights the significance of rubric-
driven summarization techniques and the utilization of custom-trained language
models.

## Limitations

Our datasets are not multilingual and multimodal. So, we need to explore how
state of the art approaches can be utilized in multlingual and multimodal
scenarios - which we aim to do as a part of future work.

## Ethics Statement

Our work does not reveal any personal sensitive information and we use
publicly available benchmarked datasets and models in different contexts.

## References

  * Abacha et al. (2023) Asma Ben Abacha, Wen-wai Yim, Griffin Adams, Neal Snider, and Meliha Yetisgen-Yildiz. 2023.  Overview of the mediqa-chat 2023 shared tasks on the summarization & generation of doctor-patient conversations.  In _Proceedings of the 5th Clinical Natural Language Processing Workshop_ , pages 503–513.
  * Aydın and Karaarslan (2023) Ömer Aydın and Enis Karaarslan. 2023.  Is chatgpt leading generative ai? what is beyond expectations?  _What is beyond expectations_.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In _Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_ , pages 65–72.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_.
  * Budzianowski et al. (2018) Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018.  Multiwoz–a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling.  _arXiv preprint arXiv:1810.00278_.
  * Chen et al. (2021) Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2021.  Summscreen: A dataset for abstractive screenplay summarization.  _arXiv preprint arXiv:2104.07091_.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Eric et al. (2020) Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, Adarsh Kumar, Anuj Goyal, Peter Ku, and Dilek Hakkani-Tur. 2020.  Multiwoz 2.1: A consolidated multi-domain dialogue dataset with state corrections and state tracking baselines.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 422–428.
  * Jain et al. (2023) Raghav Jain, Tulika Saha, Jhagrut Lalwani, and Sriparna Saha. 2023.  Can you summarize my learnings? towards perspective-based educational dialogue summarization.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 3158–3173.
  * Joshi et al. (2020) Anirudh Joshi, Namit Katariya, Xavier Amatriain, and Anitha Kannan. 2020.  Dr. summarize: Global summarization of medical dialogue by exploiting local structures.  _arXiv preprint arXiv:2009.08666_.
  * Leuski et al. (2003) Anton Leuski, Chin-Yew Lin, and Eduard Hovy. 2003.  ineats: interactive multi-document summarization.  In _The Companion Volume to the Proceedings of 41st Annual Meeting of the Association for Computational Linguistics_ , pages 125–128.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81.
  * Liu et al. (2019) Chunyi Liu, Peng Wang, Jiang Xu, Zang Li, and Jieping Ye. 2019.  Automatic dialogue summary generation for customer service.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 1957–1965.
  * Mukherjee et al. (2022) Rajdeep Mukherjee, Abhinav Bohra, Akash Banerjee, Soumya Sharma, Manjunath Hegde, Afreen Shaikh, Shivani Shrivastava, Koustuv Dasgupta, Niloy Ganguly, Saptarshi Ghosh, et al. 2022.  Ectsum: A new benchmark dataset for bullet point summarization of long earnings call transcripts.  _arXiv preprint arXiv:2210.12467_.
  * Pagnoni et al. (2022) Artidoro Pagnoni, Alexander R Fabbri, Wojciech Kryściński, and Chien-Sheng Wu. 2022.  Socratic pretraining: Question-driven pretraining for controllable summarization.  _arXiv preprint arXiv:2212.10449_.
  * Pang et al. (2021) Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, et al. 2021.  Quality: Question answering with long input texts, yes!  _arXiv preprint arXiv:2112.08608_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992.
  * Vig et al. (2022) Jesse Vig, Alexander Richard Fabbri, Wojciech Kryściński, Chien-Sheng Wu, and Wenhao Liu. 2022.  Exploring neural models for query-focused summarization.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 1455–1468.
  * Wan and Bansal (2022) David Wan and Mohit Bansal. 2022.  Factpegasus: Factuality-aware pre-training and fine-tuning for abstractive summarization.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1010–1028.
  * Wang et al. (2023) Fei-Yue Wang, Qinghai Miao, Xuan Li, Xingxia Wang, and Yilun Lin. 2023.  What does chatgpt say: The dao from algorithmic intelligence to linguistic intelligence.  _IEEE/CAA Journal of Automatica Sinica_ , 10(3):575–579.
  * Xiong et al. (2022) Wenhan Xiong, Anchit Gupta, Shubham Toshniwal, Yashar Mehdad, and Wen-tau Yih. 2022.  Adapting pretrained text-to-text models for long text sequences.  _arXiv preprint arXiv:2209.10052_.
  * Yu et al. (2019) Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Li, Bo Pang, Tao Chen, et al. 2019.  Sparc: Cross-domain semantic parsing in context.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019.  Bertscore: Evaluating text generation with bert.  _arXiv preprint arXiv:1904.09675_.
  * Zhang et al. (2022) Yusen Zhang, Ansong Ni, Ziming Mao, Chen Henry Wu, Chenguang Zhu, Budhaditya Deb, Ahmed Awadallah, Dragomir Radev, and Rui Zhang. 2022.  Summn: A multi-stage summarization framework for long input dialogues and documents: A multi-stage summarization framework for long input dialogues and documents.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1592–1604.
  * Zhong et al. (2021) Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. 2021.  Qmsum: A new benchmark for query-based multi-domain meeting summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5905–5921.
  * Zhu et al. (2020) Chenguang Zhu, Ruochen Xu, Michael Zeng, and Xuedong Huang. 2020.  A hierarchical network for abstractive meeting summarization with cross-domain pretraining.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 194–203.


Learning a Compressive Sensing Matrix with Structural Constraints via Maximum Mean Discrepancy Optimization

Michael Koller and
Wolfgang Utschick
The authors are with Professur für Methoden der Signalverarbeitung, Technische Universität München,
80333 München, Germany, e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>
Koller et al.

We introduce a learning-based algorithm to obtain a measurement matrix for compressive sensing related recovery problems.
The focus lies on matrices with a constant modulus constraint
which typically represent a network of analog phase shifters in hybrid precoding/combining architectures.
We interpret a matrix with restricted isometry property as a mapping of points from a high- to a low-dimensional hypersphere.
We argue that points on the low-dimensional hypersphere, namely, in the range of the matrix,
should be uniformly distributed to increase robustness against measurement noise.
This notion is formalized in an optimization problem which uses one of the maximum mean discrepancy metrics in the objective function.
Recent success of such metrics in neural network related topics motivate a solution of the problem based on machine learning.
Numerical experiments show better performance than random measurement matrices that are generally employed in compressive sensing contexts.
Further, we adapt a method from the literature to the constant modulus constraint.
This method can also compete with random matrices and it is shown to harmonize well with the proposed learning-based approach
if it is used as an initialization.
Lastly, we describe how other structural matrix constraints, e.g., a Toeplitz constraint, can be taken into account, too.

compressive sensing,
machine learning,
maximum mean discrepancy,
restricted isometry property,
sparse channel estimation

§ INTRODUCTION

In the context of possible technologies for up-and-coming cellular systems,
millimeter wave systems with a high number of antennas have been
proposed <cit.>.
One way of controlling hardware costs and power consumption is to connect the antennas to a smaller number of receiver chains,
which is done via an analog mixing network <cit.>.
For data transmission, digital precoding is combined with configuring the analog mixing network (analog precoding).

In order to design both of these precoding stages (hybrid precoding),
state information of the high-dimensional channel is necessary.
However, the presence of an analog mixing network makes channel estimation challenging
because the observation vector has a smaller dimension (number of receiver chains) than the channel vector (number of antennas) which is to be estimated.
To facilitate channel estimation in this situation, channel models can play an important role.
In particular, sparsity-related structural knowledge about channels in millimeter wave systems has been exploited,
e.g., <cit.>.

This naturally leads to the application of cs theory in channel estimation contexts
where the analog mixing network then plays the role of the measurement or sensing matrix.
Among the best known conditions suitable measurement matrices should fulfill is the rip (e.g., <cit.>).
Since deterministic constructions of measurement matrices with rip are not known,
the cs theory uses random matrices which enjoy favorable properties with high probability <cit.>.

One challenge with analog mixing networks is that they cannot realize any random matrix.
Rather, the hardware is constrained to provide matrix entries with constant modulus, i.e.,
only the phase of the entries can be adapted.
Additionally, it can be argued that such phase shifter matrices might only be able to provide a finite number of different phases (quantized phases, e.g., <cit.>),
thus making the realization of random matrices that are dictated by the cs theory more difficult.

In this paper, we propose a learning-based approach to find a deterministic measurement matrix for channel estimation.
The main idea is to interpret a matrix with rip as a matrix which (approximately) maps points from a high-dimensional hypersphere to a low-dimensional hypersphere.
Moreover, we suggest to choose the matrix such that in addition the points in its range (on the low-dimensional hypersphere) are uniformly distributed
in order to combat the influence of measurement noise, which typically exists in communications systems.

We use one of the mmd metrics <cit.> as learning cost function to formalize the idea.
mmd-based cost functions have had considerable success in machine learning, e.g., for generative networks <cit.>.
In numerical experiments, the matrix obtained from the proposed algorithm is shown to compete with or outperform the approach based on random matrices.
It may be argued that such a deterministic matrix might be easier to realize via an analog mixing network than random matrices.

Furthermore, we adapt an algorithm from <cit.> to the phase shifter constant modulus constraint
and demonstrate its good performance in numerical experiments as well.
In addition, this algorithm is shown to harmonize well with the learning-based approach of the current paper
by providing an initialization of the learning algorithm.

The rest of this paper is organized as follows.
<Ref> introduces the signal model as well as cs background.
The main idea including a brief introduction to mmd is outlined in <Ref>.
An overview of prior work and the modification of <cit.> is presented in <Ref>.
Finally, numerical experiments are conducted in <Ref> which is followed by a conclusion.

We write $ \exp(\mbX), \sin(\mbX), \cos(\mbX) $ for element-wise application of the exponential, sine, and cosine functions to a matrix $ \mbX $.
Transpose and conjugate transpose of vectors $ \mbx \in \C^n $ are denoted as $ \mbx^\tp $ and $ \mbx^\herm $, respectively.
We use $ \langle \cdot, \cdot \rangle $ for the Euclidean inner product and $ \| \cdot \| $ for its corresponding norm.
Lastly, the complex normal probability distribution with mean $ \mbzero_m \in \C^m $ and noise covariance $ \sigma^2 \mbI_m \in \C^{m\times m} $
is written as $ \mc{CN}(\mbzero_m, \sigma^2 \mbI_m) $.

§ PRELIMINARIES

In what follows, the signal model is

\begin{equation}\label{eq:yAhn}
\mby = \mbA \mbh + \mbn
\end{equation}

where $ \mbA \in \C^{m\times n} $ is called measurement matrix (with $ m < n $), $ \mbh \in \C^n $ is called channel,
$ \mby \in \C^m $ is a measurement or an observation, and $ \mbn \sim \mc{CN}(\mbzero_m, \sigma^2 \mbI_m) $ corresponds to measurement noise.
The matrix $ \mbA $ represents an analog mixing network with phase shifters
so that the matrix entries $ [\mbA]_{k,l} $ have the form

\begin{equation}\label{eq:elements_of_A}
[\mbA]_{k,l} = \frac{1}{\sqrt{m}} \exp(\imag \phi_{k,l})
\end{equation}

with a phase $ \phi_{k,l} \in \R $.
All entries have a constant modulus constraint, $ |[\mbA]_{k,l}| = 1/\sqrt{m} $,
which is why we refer to such a matrix as constant modulus matrix.

This paper provides two methods to determine a constant modulus measurement matrix $ \mbA $
for recovering $ \mbh $ from (<ref>).
The main contribution is a novel learning-based algorithm.
Additionally, we adapt a method from the literature to the constant modulus constraint.
This method computes a different matrix $ \mbA $ for every snr,
whereas the learning-based algorithm yields a single matrix which can be employed for all snr.
We define the snr as

\begin{equation}\label{eq:snr}
\mr{SNR} = \frac{\| \mbA \mbh \|^2}{\expec[\|\mbn\|^2]} = \frac{\| \mbA \mbh \|^2}{m \sigma^2},
\end{equation}

where the noise variance $ \sigma^2 $ is adjusted for every pair $ (\mbA, \mbh) $ to achieve the desired snr.

Since the goal is to estimate the $ n $-dimensional channel from an $ m $-dimensional observation with $ m < n $,
the inverse problem (<ref>) generally has no unique solution.
However, it is well known from cs theory that a unique solution can exist if additional structural information about the channels is available.
One channel model that was studied in this context (see, e.g., <cit.>) can be written as

\begin{equation}\label{eq:channel_model}
\mbh = \sum_{k=1}^p s_k \mba(\theta_k)
\end{equation}

which corresponds to a sum of $ p $ paths which arrive from angles $ \theta_k \in [0, 2\pi] $ with path gains $ s_k \in \C $ at the base station.
For a uniform linear array at the base station, the steering vector $ \mba(\theta_k) \in \C^n $ has the form

\begin{equation}
\mba(\theta_k) = \frac{1}{\sqrt{n}}
\begin{bmatrix}
\exp(\imag 0 \cdot \theta_k) & \dots & \exp(\imag (n-1) \cdot \theta_k)
\end{bmatrix}^\tp.
\end{equation}

We briefly highlight relevant parts of cs theory in the next section
where we then explain how cs is used with this channel model.

§.§ Compressive Sensing

cs studies the inverse problem (<ref>) for general (no constant modulus constraint) matrices $ \mbB \in \C^{m\times n} $.
In the case where $ \mbh = \mbs \in \C^n $ is a $ p $-sparse vector ($ p $ nonzero entries),
it can be recovered if $ \mbB $ has a sufficient rip <cit.>.
A matrix $ \mbB \in \C^{m\times n} $ is said to have the rip if the restricted isometry constant $ \delta \geq 0 $ is small in

\begin{equation}\label{eq:rip}
(1-\delta) \| \mbs \|^2 \leq \| \mbB \mbs \|^2 \leq (1+\delta) \| \mbs \|^2
\end{equation}

for all $ p $-sparse vectors $ \mbs \in \C^n $ <cit.>.

More generally, it may be assumed that the signals of interest, $ \mbh $,
allow for a sparse representation $ \mbh = \mbPsi \mbs $ in some basis $ \mbPsi \in \C^{n\times n} $
such that the rip condition is formulated where $ \mbs $ is replaced with $ \mbPsi \mbs $.
Even more generally, $ \mbh $ may lie in an (infinite) union of subspaces or some abstract subset $ \mc{H} \subset \C^n $
(see, e.g., <cit.>)
and one defines a generalized notion of rip as

\begin{equation}\label{eq:rip_general}
(1-\delta) \| \mbh \|^2 \leq \| \mbB \mbh \|^2 \leq (1+\delta) \| \mbh \|^2
\end{equation}

for all $ \mbh \in \mc{H} $.

Orthogonal Matching Pursuit (OMP) <cit.>

[1]
matrix $ \mbC $, observation $ \mby $, sparsity $ p $
$ S^{(0)} \leftarrow \emptyset $, $ \mbs^{(0)} \leftarrow \mbzero $
$ i = 1 $ to $ p $
$ j^* \leftarrow \argmax_{j\in\{1, \dots, n\}}\{[\mbC^\herm (\mby - \mbC \mbs^{(i-1)})]_j\} $
$ S^{(i)} \leftarrow S^{(i-1)} \cup \{ j^* \} $
$ \mbs^{(i)} \leftarrow \argmin_{\tilde{\mbs}}\{ \| \mby - \mbC \tilde{\mbs} \|,\, \mr{support}(\tilde{\mbs}) \subset S^{(i)} \} $
$ \mbs^{(p)} \quad $ ($ p $-sparse vector)

One way of translating the model (<ref>) into a cs setting is to introduce a so-called dictionary

\begin{equation}\label{eq:dictionary_grid}
\mbPsi_L =
\begin{bmatrix}
\mba(\hat{\theta}_1) & \dots & \mba(\hat{\theta}_L)
\end{bmatrix} \in \C^{n\times L}
\end{equation}

which collects $ L $ steering vectors corresponding to a grid of
$ L $ equidistantly sampled angles $ \hat{\theta}_\ell $ in $ [0, 2\pi] $.
Then, one approximates $ \mbh \approx \mbPsi_L \mbs $ with a $ p $-sparse vector $ \mbs \in \C^L $ which contains the path gains.
Channel estimation can now be performed by writing

\begin{equation}\label{eq:ytilde}
\tilde{\mby} = (\mbA \mbPsi_L) \mbs + \mbn
\end{equation}

and by using for example omp <cit.> to recover an estimate $ \hat{\mbs} $ of $ \mbs $
so that the channel is estimated as $ \hat{\mbh} = \mbPsi_L \hat{\mbs} $.
omp is described in <Ref> and would be used with $ \mbC = \mbA \mbPsi_L $, see, e.g., <cit.>.
Note that (<ref>) is just an auxiliary approximation of (<ref>)
because $ \mbPsi_L \mbs $ is an approximation of $ \mbh $.
In particular, channels of the form (<ref>) do not have an exact representation
in terms of $ \mbPsi_L $ whenever $ \theta_k \notin \{ \hat{\theta}_1, \dots, \hat{\theta}_L \}$
for some $ k = 1, \dots, p $.

To obtain a matrix with rip, one usually resorts to drawing random matrices which can fulfill the property with high probability <cit.>.
A typical example is a Gaussian random matrix $ \mbB \in \C^{m\times n} $
where the entries are drawn independently with $ [\mbB]_{k,l} \sim \mc{CN}(0, \frac{1}{\sqrt{m}}) $.
These, however, cannot be realized via an analog mixing network.
More generally, sub-Gaussian random matrices have been shown to fulfill the rip <cit.>.
Sub-Gaussian random variables include bounded random variables <cit.>.
Thus, it makes sense to consider random constant modulus matrices $ \mbA $
which are randomly drawn with independent entries of the form

\begin{equation}\label{eq:random_Steinhaus}
\frac{1}{\sqrt{m}} \exp(\imag \phi) \quad\text{with}\quad \phi \sim \mc{U}([0, 2\pi])
\end{equation}

where $ \phi $ is uniformly distributed in the interval $ [0, 2\pi] $.

§ LEARNING A MEASUREMENT MATRIX

To motivate the main idea of this paper, we first rewrite the rip condition (<ref>) in the form

\begin{equation}\label{eq:rip_divided}
(1-\delta) \leq \left\| \mbB \frac{\mbh}{\|\mbh\|} \right\|^2 \leq (1+\delta)
\end{equation}

for all $ \mbh \in \mc{H} $.
Recovery algorithms need $ \delta \geq 0 $ to be small enough.
In the best case, we have $ \delta = 0 $ and (<ref>) reads

\begin{equation}\label{eq:rip_hypersphere}
\left\| \mbB \frac{\mbh}{\|\mbh\|} \right\|^2 = 1.
\end{equation}

Here, $ \htilde := \frac{\mbh}{\|\mbh\|} $ is a vector on the (high-dimensional) hypersphere in $ \C^n $
and according to (<ref>) the matrix $ \mbB $ has the property
that the vector $ \mbB \htilde $ lies on the (low-dimensional) hypersphere in $ \C^m $.
In the general case, in order for $ \mbdelta $ to be as small as possible,
(<ref>) requires the norm $ \| \mbB \htilde \| $ to be as close to one as possible.
In other words, we want to find a matrix which brings $ \mbB \htilde $ as close to the
hypersphere in $ \C^m $ as possible.

Let $ \mbh, \mbh' \in \mc{H} $ be two different vectors
and let $ \mby = \mbB \mbh $ and $ \mby' = \mbB \mbh' $ be the corresponding noiseless observations.
The rip formalizes the notion that the distance between $ \mby $ and $ \mby' $
should be proportional to the distance between $ \mbh $ and $ \mbh' $
such that two vectors $ \mbh, \mbh' $ which are far apart from each other
do not lead to the same observation even if the observations are noisy.
As a consequence, $ \mbh $ and $ \mbh' $ can still be uniquely recovered if the noise is small.
In this sense, the rip guarantees some robustness against measurement noise.

In the spirit of the above description,
it seems sensible in general to try to find a matrix $ \mbB $
which places two different observations far apart from each other,
so that even in the presence of noise, they are distinguishable.
When we work with the normalized vectors $ \htilde $,
this means that we now have two goals:
On the one hand, $ \| \mbB \htilde \| $ should be close to one in order to achieve a rip,
on the other hand, two different $ \mbB \htilde $ and $ \mbB \htilde' $ should be far apart
from each other.
The latter goal can also be formulated as follows:
We want to find a matrix $ \mbB $ which distributes $ \mbB \htilde $ isotropically around the origin.

To cast this idea into an optimization problem, let $ \mbu $ be a random variable which is uniformly distributed on the hypersphere in $ \C^m $.
That is, we have $ \| \mbu \| = 1 $ and $ \mbu $ is isotropically distributed around the origin.
Further, let all elements in $ \mc{H} $ be realizations of a random variable $ \mbh $
and consider again the normalized $ \htilde $.
Then, if $ d $ denotes a (for now generic) distance between the distributions of $ \mbB \htilde $ and $ \mbu $,
the goal is to choose $ \mbB $ such that this distance is minimized.
Together with the original goal to find a suitable constant modulus matrix $ \mbA $, we arrive at the abstract optimization problem:

\begin{equation}\label{eq:abstract_problem}
\min_{\mbA\in\C^{m\times n}, |[\mbA]_{k,l}| = \frac{1}{\sqrt{m}}} d(\mbA \htilde, \mbu).
\end{equation}

We make this notion more precise in <Ref>.
To this end, <Ref> first reviews the concept of mmd
which can be used to define a distance between probability distributions.

The optimization problem (<ref>) encodes three goals.
In order for $ \mbA \htilde $ to be uniformly distributed on the hypersphere,
the norm $ \| \mbA \htilde \| $ needs to be as close to one as possible
since the norm of $ \mbu $ always equals one.
This corresponds to finding a matrix which has the rip
with $ \delta $ in (<ref>) as small as possible.
At the same time, $ \mbA \htilde $ needs to be isotropically distributed around the origin
because this is a property of $ \mbu $.
This corresponds to placing noiseless observations far apart from each other.
Therefore, problem (<ref>) simultaneously aims for a rip condition as well as for robustness against noise.
Finally, due to the constraints, we obtain a constant modulus matrix.

§.§ Maximum Mean Discrepancy (MMD)

To state problem (<ref>) more precisely, a distance between the probability distributions of $ \mbA \htilde $ and $ \mbu $ is necessary.
Many distances, for example the Kullback-Leibler divergence, can only be evaluated
if closed-form expressions of the distributions are available or if the distributions are estimated beforehand.
In the general case, we have access to samples of the channel $ \mbh $, which are, e.g., obtained from a simulation environment or a measurement campaign.
Moreover, the distribution of $ \mbh $, and thus in particular also of $ \mbA \htilde $, is not known.
It is therefore more convenient to work with a distance which can be evaluated based on samples from the involved distributions,
i.e., samples of $ \mbA \htilde $ and $ \mbu $.
In the following, we introduce one of the mmd distances which has this property.

Let $ (\mc{X}, d) $ be a metric space and let $ \mbx $ and $ \mby $ be two random variables with respective probability measures $ p $ and $ q $.
mmd provides the possibility to define a metric on the space of probability measures on $ \mc{X} $.
For a given set $ \mc{F} $ of functions $ f: \mc{X} \to \R $,
mmd is defined as <cit.>

\begin{equation}\label{eq:mmd}
\mmd_{\mc{F}}(p, q) = \sup_{f\in\mc{F}}( \expec_{\mbx\sim p}[f(\mbx)] - \expec_{\mby\sim q}[f(\mby)] ).
\end{equation}

Depending on how $ \mc{F} $ is chosen, mmd can be a metric
and then we have $ \mmd_{\mc{F}}(p, q) = 0 $ if and only if $ p = q $, and $ \mmd_{\mc{F}}(p, q) > 0 $ otherwise.
An overview of suitable function sets $ \mc{F} $ can, e.g., be found in <cit.>.

Even if mmd is a metric, the evaluation of (<ref>), i.e., solving the optimization problem, might still be difficult.
One particularly interesting set $ \mc{F} $ in this regard is the unit ball in a rkhs.
A brief introduction into rkhs can be found in <cit.>.
Every rkhs is a Hilbert space which is associated with a kernel $ k: \mc{X} \times \mc{X} \to \R $.
And, conversely, every kernel defines an associated rkhs.
Intuitively speaking, a kernel $ k $ can represent all functions which are an element of the rkhs.
For this reason, the optimization w.r.t. the whole set $ \mc{F} $ in (<ref>) can be expressed by means of the kernel only
and the computation of the supremum is avoided.

Among the best known kernels is the Gaussian one:

\begin{equation}\label{eq:kernel_gauss}
k_\sigma: \R^{2m} \times \R^{2m} \to \R, \: (\mbx, \mby) \mapsto k_\sigma(\mbx, \mby) = e^{-\frac{1}{2 \sigma^2} \| \mbx - \mby \|^2}
\end{equation}

with a parameter $ \sigma > 0 $.
Using the Gaussian kernel, (<ref>) can be expressed as <cit.>

\begin{multline}\label{eq:mmd_kernel}
\mmd_k^2(p, q) = \expec_{\mbx, \mbx'}[k(\mbx, \mbx')] \\- 2 \expec_{\mbx, \mby}[k(\mbx, \mby)] + \expec_{\mby, \mby'}[k(\mby, \mby')]
\end{multline}

and it is a metric.
Here, $ \mbx $ and $ \mbx' $ are independent random variables with distribution $ p $,
and $ \mby $ and $ \mby' $ are independent random variables with distribution $ q $.
The Gaussian kernel is a typical choice when $ \mmd_k $ is used, as, e.g., <cit.> demonstrate.
One reason is that the associated rkhs enjoys the property of being universal which makes (<ref>) a metric <cit.>.

If only samples $ \{ \mbx_i \}_{i=1}^{M} $ and $ \{ \mby_j \}_{j=1}^N $ of $ \mbx \sim p $ and $ \mby \sim q $, respectively, are given,
(<ref>) allows for a biased estimate <cit.>:

\begin{multline}\label{eq:mmd_empirical}
\mmd_k^2(\{ \mbx_i \}_{i=1}^{M}, \{ \mby_j \}_{j=1}^N) = \sum_{i=1}^{M} \sum_{j=1}^{M} \frac{k(\mbx_i, \mbx_j)}{M^2}
\\- 2 \sum_{i=1}^{M} \sum_{j=1}^N \frac{k(\mbx_i, \mby_j)}{MN} + \sum_{i=1}^N \sum_{j=1}^N \frac{k(\mby_i, \mby_j)}{N^2}.
\end{multline}

Throughout the paper, we employ this estimate (<ref>) which converges in probability to (<ref>)
at a rate of $ \mathcal{O}((M+N)^{-\frac{1}{2}}) $, see <cit.>.
A detailed analysis of the metric can be found in <cit.>.

Since the kernel (<ref>) is differentiable, so is $ \mmd_k $.
This, together with the fact that only samples from the involved distributions are required,
makes (<ref>) interesting for learning applications.
For example, <cit.> uses $ \mmd_k $ as neural network cost function in the context of data generation.
There, a set of true images is given and the goal is to find a function which takes random noise as argument
and yields artificial new images as function values.
The function is chosen to be a neural network and the goal is to minimize $ \mmd_k $
between its output and the true images,
so that the neural network is able to generate photo-realistic images
which resemble the true ones.
The minimization problem is solved via stochastic gradient descent with impressive results.

§.§ Learning

With the help of mmd, we express (<ref>) as a learning problem.
To this end, we want to replace the notion of a generic distance $ d $ in (<ref>) between $ \mbA \htilde $ and $ \mbu $ with $ \mmd_k $.

For this purpose, we represent $ \mbA \htilde $ and $ \mbu $ in terms of real quantities because it also simplifies the implementation.

\begin{equation}
\stack: \C^m \to \R^{2m},\:\mbz \mapsto \stack(\mbz) = [\Re(\mbz)^\tp, \Im(\mbz)^\tp]^\tp
\end{equation}

denote stacking real and imaginary parts of a complex vector $ \mbz $ into a real vector $ \stack(\mbz) $.
Further, let $ \{ \htilde_i \}_{i=1}^{\Ttrain} $ and $ \{ \mbu_i \}_{i=1}^{\Ttrain} $ be training samples.
Here, $ \{ \htilde_i \}_{i=1}^{\Ttrain} $ is a sample of normalized channel data points
and $ \{ \mbu_i \}_{i=1}^{\Ttrain} $ a set of points drawn from a uniform distribution on the hypersphere in $ \C^m $.
The channel data points may either stem from a measurement campaign or may be artificially generated according to a channel model.
A random vector $ \mbu $ with uniform distribution on the hypersphere in $ \C^m $ is obtained via normalization of an isotropic Gaussian random vector, e.g.,
$ \mbu = \frac{\mbv}{\|\mbv\|} $ with $ \mbv \sim \mc{C}\mc{N}(\mbzero_m, \mbI_m) $, which means that sampling is simple.
We then express (<ref>) as follows:

\begin{equation}\label{eq:complex_problem}
\min_{\substack{\mbA\in\C^{m\times n}\\ |[\mbA]_{k,l}| = \frac{1}{\sqrt{m}}}}
\mmd_k^2( \{ \stack(\mbA \htilde_i) \}_{i=1}^{\Ttrain}, \{ \stack(\mbu_i) \}_{i=1}^{\Ttrain}).
\end{equation}

Next, we address the constant modulus constraint $ |[\mbA]_{k,l}| = 1/\sqrt{m} $
and express the optimization problem in terms of real optimization variables instead of $ \mbA \in \C^{m\times n} $.
Note that since every entry of $ \mbA \in \C^{m\times n} $ has to have the form $ [\mbA]_{k,l} = \frac{1}{\sqrt{m}} \exp(\imag \phi_{k,l}) $ with a phase $ \phi_{k,l} \in \R $,
the constant modulus matrix $ \mbA $ is completely parameterized by $ mn $ real parameters.
We collect these parameters in a matrix $ \mbPhi \in \R^{m\times n} $
such that we can write $ \mbA = \frac{1}{\sqrt{m}}\exp(\imag \mbPhi) $ where the exponential function is meant to be applied element-wise
and where the constraint $ |[\mbA]_{k,l}| = \frac{1}{\sqrt{m}} $ is already incorporated.
If also $ \sin $ and $ \cos $ are applied element-wise,
we can write

\begin{equation}
\mbA = \Re(\mbA) + \imag \Im(\mbA) = \frac{1}{\sqrt{m}}\cos(\mbPhi) + \imag \frac{1}{\sqrt{m}}\sin(\mbPhi)
\end{equation}

so that we get from $ \mbA \htilde = (\Re(\mbA) + \imag \Im(\mbA))(\Re(\htilde) + \imag \Im(\htilde)) $:

\begin{equation}\label{eq:stack_Ah}
\stack(\mbA \htilde)
\begin{bmatrix}
\Re(\mbA \htilde) \\
\Im(\mbA \htilde)
\end{bmatrix}
\begin{bmatrix}
\frac{\cos(\mbPhi)}{\sqrt{m}} & - \frac{\sin(\mbPhi)}{\sqrt{m}} \\
\frac{\sin(\mbPhi)}{\sqrt{m}} & \frac{\cos(\mbPhi)}{\sqrt{m}}
\end{bmatrix}
\begin{bmatrix}
\Re(\htilde) \\ \Im(\htilde)
\end{bmatrix}.
\end{equation}

<Ref> reveals that $ \stack(\mbA \htilde) $ only depends on $ \mbA \in \C^{m\times n} $ via the real matrix $ \mbPhi \in \R^{m\times n} $.
Let us express this as $ \stack(\mbA \htilde) = \stack(\mbA(\mbPhi) \htilde) $ so that we can rewrite (<ref>) as a real optimization problem:

\begin{equation}\label{eq:real_problem}
\min_{\mbPhi\in\R^{m\times n}}
\mmd_k^2( \{ \stack(\mbA(\mbPhi) \htilde_i) \}_{i=1}^{\Ttrain}, \{ \stack(\mbu_i) \}_{i=1}^{\Ttrain}).
\end{equation}

In the literature, such optimization problems are solved via stochastic gradient descent,
e.g., <cit.>.
Since gradient descent can be employed and seems to yield good results,
mmd-based problems are considered to be easy to optimize <cit.>.
Accordingly, we also use stochastic gradient descent to solve problem (<ref>).
The details are summarized in <Ref>.
General convergence guarantees are part of ongoing research.
The standard procedure in machine learning is
to employ some form of grid or random search over various hyperparameters of the
stochastic gradient descent algorithm in combination with random initializations
of the optimization parameters.
A validation data set is then used to choose the best solution.
Possible hyperparameter values for <Ref> are described in <Ref>.

In every iteration of <Ref>, $ \mmd_k^2 $ and its gradient need to be calculated.
With backpropagation, evaluating the gradient has the same order of computation time
as the forward pass <cit.>.
To compute $ \mmd_k^2 $ for a batch of $ T $ samples,
the kernel is evaluated $ \mathcal{O}(T^2) $ times, see (<ref>),
with vectors of dimension $ 2m $ (stacked real and imaginary parts).
Since $ \stack(\mbA \htilde) $ can be computed in $ \mathcal{O}(mn) $ time,
the overall computation time is $ \mathcal{O}(mnT + 2mT^2) $.
As an alternative, <cit.> could be used,
where a linear (in $ T $) version of mmd is introduced
such that the computation time would be $ \mathcal{O}(mnT) $.

Learning $ \mbA = \mbA(\mbPhi) $

[1]
training data $ \{ (\htilde_i, \mbu_i) \}_{i=1}^{\Ttrain} $
randomly initialize $ \mbPhi \in [0, 2\pi]^{m\times n} $
termination criterion not met
draw a batch of $ T $ samples uniformly from the training data: $ \{ (\htilde_t, \mbu_t ) \}_{t=1}^T \subset \{ (\htilde_i, \mbu_i) \}_{i=1}^{\Ttrain} $
compute the stochastic gradient

\begin{equation*}
\mbg_{\mbPhi} = \frac{\partial}{\partial \mbPhi} \mmd_k^2( \{ \stack(\mbA(\mbPhi) \htilde_t) \}_{t=1}^T, \{ \stack(\mbu_t) \}_{t=1}^T )
\end{equation*}

update $ \mbPhi $ with a gradient algorithm using $ \mbg_{\mbPhi} $

§ RELATED WORK

This section reviews prior work related to learning a measurement matrix and describes a modification of one of the presented methods
which allows us to employ it in the context of the signal model described in <Ref>.

§.§ Prior Work

The authors of <cit.> consider the real-valued signal model $ \mby = \mbB \mbh + \mbn $
where $ \mbB \in \R^{m\times n} $ is a general real matrix.
Their Uncertain Component Analysis determines the matrix as

\begin{equation}
\mbB = \argmax_{\tilde{\mbB} \in \R^{m\times n}, \tilde{\mbB} \tilde{\mbB}^\tp = \mbI} \prod_{i} \mr{Pr}(\mbh_i \mid \mby_i; \tilde{\mbB})
\end{equation}

with given data points $ \mbh_i $ and $ \mby_i =\tilde{\mbB} \mbh_i $ and
where $ \mr{Pr}(\mbh_i \mid \mby_i; \tilde{\mbB}) $ is the posterior probability of the data.
An algorithmic solution based on two fixed-point equations is proposed.

In <cit.>, the signal model is $ \mby = \mbB \mbPsi \mbs $
with $ \mbB \in \R^{m\times n} $ and $ \mbPsi \in \R^{n\times L} $ ($ n < L $).
The goal is to minimize the $ t $-averaged mutual coherence $ \mu_t(\mbB \mbPsi) $ with resepect to $ \mbB $.
To compute $ \mu_t(\mbB \mbPsi) $, the columns of $ \mbB \mbPsi $ are normalized and
then the Gram matrix $ \mbG = (\mbB \mbPsi)^\tp (\mbB \mbPsi) $ is computed.
The average of all entries $ |[\mbG]_{ij}| \geq t $ is defined to be $ \mu_t(\mbB \mbPsi) $.
The proposed iterative solution algorithm proceeds in two alternating stages.
Starting from some initial $ \mbB^{(1)} $,
in iteration $ i $ the Gram matrix $ \mbG^{(i)} = (\mbB^{(i)} \mbPsi)^\tp (\mbB^{(i)} \mbPsi) $ is computed
and its entries are modified such that $ \mu_t(\mbB^{(i)} \mbPsi) $ decreases,
yielding a new matrix $ \hat{\mbG}^{(i)} $.
In order to decrease $ \mu_t(\mbB^{(i)} \mbPsi) $, a down-scaling factor $ \gamma \in (0, 1) $,
which is chosen before the algorithm's main iterations start,
is used to shrink all entries of $ \mbG^{(i)} $ whose absolute values are larger than $ t $ by multiplying them by $ \gamma $.
In the second stage, a decomposition $ \hat{\mbG}^{(i)} \approx (\mbB^{(i+1)} \mbPsi)^\tp (\mbB^{(i+1)} \mbPsi) $ is found
to obtain a new matrix $ \mbB^{(i+1)} $.
These two stages are alternated for a number of iterations.

Similar to <cit.>, the authors of <cit.> work with the Gram matrix $ \mbG \in \R^{L\times L} $.
Their goal is to choose $ \mbB \in \R^{m\times n} $ such that $ \mbG $ is as close to an identity matrix as possible.
To this end, $ \mbB $ is initialized randomly and then a KSVD-like algorithm consisting of $ m $ (number of rows of $ \mbB $)
steps is computed.
Additionally, an extension is proposed where also the dictionary $ \mbPsi $ can be optimized.

The goal of <cit.> is to choose $ \mbB $ such that a bi-Lipschitz condition is established.
For given data points $ \{ \mbh_i \}_{i=1}^{T} $, this condition is expressed by replacing $ \mbh $ in (<ref>) with differences $ \mbh_i - \mbh_j $.
Therefore, the authors first define the secant set
$ \mc{S} = \{ \frac{\mbh_i -\mbh_j}{\| \mbh_i - \mbh_j \|}, 1 \leq i < j \leq T \} $
and then express the bi-Lipschitz criterion for this set as $ | \| \mbB \mbv \|^2 - 1 | = |\mbv^\tp \mbB^\tp \mbB \mbv - 1| \leq \delta $ with $ \mbv \in \mc{S} $.
Using the Gram matrix $ \mbG = \mbB^\tp \mbB $,
this motivates the following optimization problem to find a matrix $ \mbB \in \R^{m\times n} $:

\begin{multline}\label{eq:lipschitz_op}
\min_{\mbG\in\R^{n\times n}} \sup_{\mbv\in\mc{S}} | \mbv^\tp \mbG \mbv - 1|
\\\text{s.t.}\quad \mbG \succeq 0, \quad \mr{rank}(\mbG) = r, \quad \mr{trace}(\mbG) = b.
\end{multline}

The solution algorithm is based on a relaxation of (<ref>).

A related problem is studied in <cit.>:

\begin{equation}
\min_{\mbG\in\R^{n\times n}, \mbG = \mbG^\tp, \mbG \succeq 0} \mr{trace}(\mbG)
\quad\text{s.t.}\,
\sup_{\mbv\in\mc{S}} | \mbv^\tp \mbG \mbv - 1 | \leq \delta.
\end{equation}

Once a feasible $ \mbG^* $ is determined, the matrix $ \mbB $ is found
via an eigendecomposition of $ \mbG^* $.

An approach motivated by neural network learning is presented in <cit.>.
The signal model is $ \mby = \mbB \mbs $
with nonnegative sparse vectors $ \mbs \in \R_+^n $ and $ \mbB \in \R^{m\times n} $.
Motivated by an $ \ell_1 $-norm minimization approach,
the authors propose to utilize the following projected subgradient update rule to learn $ \mbB $:

\begin{equation}\label{eq:pgd}
\mbs^{(i+1)} = \mbs^{(i)} - \frac{\beta}{i} (\mbI - \mbB^\tp \mbB) \mr{sign}(\mbs^{(i)})
\end{equation}

with the initialization $ \mbs^{(1)} = \mbB^\tp \mby $ and $ \beta \in \R $.
Every gradient iteration (<ref>) is interpreted as a layer and $ I $ such gradient layers are concatenated.
This is called gradient unrolling.
The final output and estimate for $ \mbs $ is given by $ \hat{\mbs} = \mr{ReLU}(\mbs^{(I+1)}) $,
where $ \mr{ReLU} $ denotes the usual rectified linear unit
which performs the operation $ \max(x, 0) $ on every element $ x $ of its argument.
Then, $ \mbB \in \R^{m\times n} $ and $ \beta \in \R $ are interpreted as learnable parameters
and stochastic gradient descent is employed to learn $ \mbB $ and $ \beta $.
Specifically, given training data $ \{ \mbs_i \}_{i=1}^T $, a stochastic gradient descent method is used to solve

\begin{equation}
\min_{\mbB \in \R^{m\times n}, \beta\in\R} \frac{1}{T} \sum_{i=1}^T \| \mbs_i - \hat{\mbs}_i \|^2.
\end{equation}

This gradient unrolling method was again employed in <cit.> for channel state information feedback.
To this end, a channel $ \mbh \in \C^n $ is compressed as $ \mby = \mbB [ \Re(\mbh)^\tp, \Im(\mbh)^\tp ]^\tp \in \R^m $
with $ \mbB \in \R^{m\times 2n} $ such that the base station
can recover $ \mbh $ from the feedback signal $ \mby $ via compressive sensing methods.
A modification of <cit.> for channel estimation is described in <cit.>,
where real and imaginary parts of channels $ \mbh \in \C^n $ are treated as independent data points
such that a real matrix $ \mbB \in \R^{m\times n} $ can be learned.

§.§ Learning-Based Compressive Subsampling (LBCS)

LBCS <cit.>

[1]
training data $ \{ \mbh_i \}_{i=1}^{\Ttrain} $
matrix $ \mbV \in \C^{n\times n} $, number $ m $ of rows to select
normalize the training data: $ \{ \htilde_i \}_{i=1}^{\Ttrain} \leftarrow \left\{ \frac{\mbh_i}{\| \mbh_i \|} \right\}_{i=1}^{\Ttrain} $
$ r = 1 $ to $ n $
$ \alpha_r \leftarrow \sum_{i=1}^{\Ttrain} |\langle \mbv_r, \htilde_i \rangle|^2 \quad $ (with $ \mbv_r $ row $ r $ of $ \mbV $)
$ \Omega \leftarrow $ indices $ r $ which correspond to $ m $ largest $ \alpha_r $
$ \mbP_\Omega \mbV \quad $ (learned measurement matrix)

One challenge with the methods presented so far is that general real-valued matrices $ \mbB \in \R^{m\times n} $ are found
whereas we are seeking a complex-valued matrix $ \mbA \in \C^{m\times n} $ where all entries $ [\mbA]_{k,l} $ have the same absolute value.
This constant modulus constraint is the main challenge.
In particular, methods which are based on a decomposition of the form $ \mbG = \mbB^\tp \mbB $ are difficult to apply
since such a decomposition need not exist when the entries of $ \mbB $ are required to have a constant modulus.
However, a modification of the approach published in <cit.> allows us to incorporate the constraint.

The approach investigated in <cit.> is based on the model

\begin{equation}\label{eq:lbcs_signal_model}
\mby = \mbP_{\Omega} \mbV \mbh
\end{equation}

where $ \mbV \in \C^{n\times n} $ is a basis (a unitary matrix, $ \mbV^\herm \mbV = \mbI $)
and $ \mbP_\Omega \in \C^{m\times n} $ performs subsampling by selecting $ m = |\Omega| $ rows of $ \mbV $.
Once the row index set $ \Omega \subset \{1, 2, \dots, n\} $ is chosen, the measurement matrix is given by $ \mbB = \mbP_{\Omega} \mbV \in \C^{m\times n} $.
One strategy is to select the indices $ \Omega $ randomly.
However, the authors of <cit.> argue that one can improve
upon this by considering a data-based selection procedure.

A criterion for selecting $ \Omega $ for which an exact solution is provided in <cit.> can be described as follows.
Let $ \{ \mbh_i \}_{i=1}^T $ be given training data with norm one.
Then, select those $ m $ rows $ \mbv_r \in \C^n $ of $ \mbV $ which maximize
$ \frac{1}{T} \sum_{i=1}^T \sum_{r\in\Omega}|\langle \mbv_r, \mbh_i\rangle|^2 $.
This corresponds to maximizing the average captured energy $ \| \mbP_\Omega \mbV \mbh_i \|^2 $ of the observed data.
We call the procedure after the paper's title and it is outlined in <Ref>.

§.§ Adaptation of LBCS

Since (<ref>) considers the noiseless case and since it is assumed that a unitary matrix is already given,
we suggest the following adaptation of the algorithm to obtain a matrix $ \mbA $
which can be employed in the context of the current paper.

Because we need a constant modulus matrix,
the first modification is to divide the entries of $ \mbV $ by their respective absolute values.
<Ref> can then be used to select $ m $ rows of this constant modulus matrix.
Second, we do a Monte Carlo search over different initializations of the unitary matrix $ \mbV $.
To explain, we draw a unitary matrix $ \mbV $ uniformly at random,
divide its entries by their absolute values,
and run <Ref> to obtain a constant modulus matrix $ \mbA $.
An algorithm for generating random unitary matrices can, e.g., be found in <cit.>.
Then, validation data $ \{ \mbh_i \}_{i=1}^{\Tval} $ is used to evaluate the channel
estimation performance of the current $ \mbA $ in terms of mse.
The best matrix $ \mbA_{\mr{MC}} $ of $ I $ such iterations is then the final result of the Monte Carlo search.
The whole procedure is outlined in <Ref>.

§.§ Comparison between MMD-Based Learning and LBCS

Both <Ref> and <Ref> are data-based approaches to determine a matrix $ \mbA $.
In both cases, an outer loop over different initializations is used.
<Ref> would be combined with multiple random searches,
and in case of <Ref>,
we evaluate <Ref> multiple times for different randomly initialized uniform matrices.
These are the training phases of the respective algorithms.
A main difference is that <Ref> uses a validation mse criterion
to choose the best matrix, which is dependent on the snr.
Thus, in the numerical experiments in <Ref>,
we run the algorithm for every snr.
This is in contrast to <Ref> where a single matrix is determined
which can be employed for all snr.
Another difference is that <Ref> selects rows of a random initialization $ \mbV $,
but doesn't modify its initial entries.
In comparison, <Ref> continually updates a random initialization.

Monte Carlo LBCS

[1]
training and validation data $ \{ \mbh_j \}_{j=1}^{\Ttrain} $ and $ \{ \mbh_j \}_{j=1}^{\Tval} $
$ \mr{SNR} $, number $ m $ of rows to select, iterations $ I $
$ \mbA_{\mr{MC}} \leftarrow \mbzero $, $ \mr{MSE}_{\mr{MC}} \leftarrow \infty $
$ i = 1 $ to $ I $
draw a random unitary matrix $ \mbV \in \C^{n\times n} $
divide all entries of $ \mbV $ by their absolute values
normalize the rows of $ \mbV $
$ \mbA^{(i)} \leftarrow \mr{LBCS}(\{ \mbh_j \}_{j=1}^{\Ttrain}, \mbV, m) $
$ \mr{MSE}^{(i)} \leftarrow \mr{Evaluation}(\mbA^{(i)}, \{\mbh_j\}_{j=1}^{\Tval}, \mr{SNR}) $
$ \mr{MSE}^{(i)} < \mr{MSE}_{\mr{MC}} $
$ \mr{MSE}_{\mr{MC}} \leftarrow \mr{MSE}^{(i)} $
$ \mbA_{\mr{MC}} \leftarrow \mbA^{(i)} $
$ \mbA_{\mr{MC}} $

§ NUMERICAL EXPERIMENTS

In numerical experiments, we cover the cases
(i) training with sparse vectors,
(ii) training with vectors that have an exact sparse representation,
and (iii) training with vectors that are only approximately sparse.
Channel estimation is always performed with omp (<Ref>).
While there are other choices for recovery algorithms, like $ \ell_1 $-norm minimization <cit.>,
the exact choice is not too critical
since we are only interested in the relative performance of the different approaches to determine a measurement matrix.
omp is a cheap option and widely used.

§.§ Data Models

Of the following three data models, only the last would actually be considered a model for communications channels.
Nonetheless, for simplicity, we continue to speak of channels and channel estimation.
We investigate these three models because they represent typical cases of cs applications.

First, channels

\begin{equation}\label{eq:model1}
\mbh = \mbs
\end{equation}

are modeled as sparse vectors $ \mbs \in \C^{n} $ with $ p $ nonzero entries.
Second, channels are modeled as sparse with respect to the dft basis $ \mbF \in \C^{n\times n} $:

\begin{equation}\label{eq:model2}
\mbh = \mbF \mbs.
\end{equation}

In both cases, each of the $ p $ nonzero entries of $ \mbs $ is drawn from a Gaussian distribution $ \mc{CN}(0, \frac{1}{p}) $.
Third, channels are modeled as explained around (<ref>) such that we have

\begin{equation}\label{eq:model3}
\mbh = \sum_{k=1}^p s_k \mba(\theta_k).
\end{equation}

The path gains $ s_k $ are drawn from $ \mc{CN}(0, \frac{1}{p}) $, the angles $ \theta_k $ are drawn uniformly in $ [0, 2\pi] $.
To perform channel estimation, we use the dictionary $ \mbPsi_L $ from (<ref>) with $ L = 16 n $.

To estimate a channel $ \mbh $ from an observation $ \mby = \mbA \mbh + \mbn $,
omp (<Ref>) is used with $ \mbC = \mbA \mbPsi $
where $ \mbPsi $ is chosen as (i) $ \mbPsi = \mbI $, (ii) $ \mbPsi = \mbF $, or (iii) $ \mbPsi = \mbPsi_L $,
depending on the channel model.
In all cases, omp yields a $ p $-sparse vector $ \hat{\mbs} $ and the channel estimate is computed as $ \hat{\mbh} = \mbPsi \hat{\mbs} $.
In the following numerical experiments,
the described procedure is repeated for $ T_{\mr{test}} = 10000 $ randomly drawn channels
such that an overall relative mse $ \frac{\expec[\|\hat{\mbh} - \mbh\|^2]}{\expec[\|\mbh\|^2]} $ can be estimated.
For a given measurement matrix $ \mbA $ determined via learning (<Ref>) or via (<Ref>),
the evaluation process is outlined in <Ref>.

[1]
measurement matrix $ \mbA $, data $ \{ \mbh_i \}_{i=1}^{T} $
$ \mr{SNR} $, sparsity $ p $, dictionary $ \mbPsi $
$ e \leftarrow 0 $, $ \Delta \leftarrow 0 $
$ i = 1 $ to $ T $
get $ \sigma $ with $ (\mbA, \mbh_i) $ via (<ref>), draw $ \mbn \sim \mc{CN}(\mbzero_m, \sigma^2 \mbI_m) $
$ \mby \leftarrow \mbA \mbh_i + \mbn \quad $ (compute observation)
$ \hat{\mbh} \leftarrow \mbPsi \mr{OMP}(\mbA\mbPsi, \mby, p) \quad $ (estimate channel $ \mbh_i $)
$ e \leftarrow e + \| \mbh_i \|^2 $, $ \Delta \leftarrow \Delta + \| \hat{\mbh} - \mbh_i \|^2 \quad $ (squared error)
$ \frac{\Delta}{e} \quad $ (relative mean squared error)

As already pointed out in <Ref>,
the related methods which are based on decompositions of a Gram matrix are difficult to employ
when the measurement matrix is constrained to have constant modulus entries
because a corresponding decomposition does generally not exist.
However, the method from <cit.> could be adapted to the constraint
and is therefore used as a reference, see <Ref>.

We compare the two methods of determining a measurement matrix $ \mbA $ (<Ref>)
with the alternative approach of drawing a new random measurement matrix for every channel.
That is, instead of first fixing $ \mbA $ and then running <Ref>,
the first step in the for-loop (between lines 2 and 3) always consists of drawing a new random measurement matrix
(with entries as explained in (<ref>))
such that as many measurement matrices are drawn as there are channels.
This approach is labeled as “random” in the following plots.

§.§ Implementation Details

Whenever we run <Ref>,
we use $ \Ttrain = 100000 $ training and $ \Tval = 1000 $ validation data samples which are randomly generated according to one of the three channel models.
Further, for every given snr, we do a Monte Carlo search over $ I = 100 $ matrices.

In order to run the learning <Ref>, a number of hyperparameters need to be set.
First of all, since we use mmd with a Gaussian kernel (<ref>), the parameter $ \sigma > 0 $ needs to be chosen.
As argued in <cit.>, finding the optimal $ \sigma $ which makes mmd most efficient is an open problem,
but a mixture of kernels may serve the intended purpose.
Therefore, we use $ \mmd_k $ in (<ref>) with the kernel

\begin{equation}
k(x, y) = \sum_{\sigma\in\mc{S}} k_\sigma(x, y)
\end{equation}

where $ \mc{S} = \{2, 5, 10, 20, 40, 80\} $.

We use Pytorch <cit.> to implement the stochastic gradient optimization in <Ref> with the Adam optimizer <cit.>.
There are three hyperparameters: batch size $ T $, learning rate (gradient step size) $ l_r $, and exponential learning rate decay $ \beta $.
As described in <cit.>, we determine these three hyperparameters via random search.
In detail, we draw $ (T, l_r, \beta) \in [150, 1500] \times [10^{-6}, 5\cdot 10^{-3}] \times [0.94, 1] $ randomly and run <Ref>.
This is repeated 64 times and validation data is then used to pick the best of the 64 so-obtained matrices $ \mbA $.
The number of training samples is always $ \Ttrain = 50000 $ in all cases and settings.
For reasons explained later in <Ref>, training data points $ \{ \mbh_i \}_{i=1}^{\Ttrain} $ are normalized as follows:

\begin{equation}\label{eq:normalization_expec}
\htilde_i = \frac{\mbh_i}{\sqrt{\frac{1}{\Ttrain}\sum_{j=1}^{\Ttrain} \| \mbh_j \|^2}}, \quad \forall i = 1, 2, \dots, \Ttrain.
\end{equation}

The termination criterion for <Ref> is early stopping <cit.>.

§.§ Discussion of Results for Sparse Vectors

We first concentrate on the two models (<ref>) and (<ref>) where channels have an exact sparse representation
(either in the canonical basis $ \mbI $ or in the dft basis $ \mbF $).
The constant modulus measurement matrix $ \mbA \in \C^{m\times 128} $ has the dimensions $ m \times 128 $ and we vary the sparsity $ p \in \{1, 5, 10\} $.

xmin=-5, xmax=30,
ymin=0.00005, ymax=10,
xlabel=SNR [dB],
ylabel=relative MSE,
title=model (<ref>),
legend style=at=(0.0, 0.0), anchor=south west, legend columns=1,

[omp1, discard if not=algorithmrandom]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, omp1, discard if not=algorithmrandom]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, omp1, discard if not=algorithmrandom]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[lbcs1, discard if not=algorithmlbcs]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, lbcs1, discard if not=algorithmlbcs]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, lbcs1, discard if not=algorithmlbcs]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[learned1, discard if not=algorithmlearned]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, learned1, discard if not=algorithmlearned]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, learned1, discard if not=algorithmlearned]
table [ignore chars=",x=snr,y=mean,col sep=comma]

(axis cs:12.5,1.4) node[ellipse, darkgray, inner xsep=0pt, inner ysep=0pt, minimum width=0.15cm, minimum height=0.4cm, draw] (e10) ;
[inner sep=1pt] (n10) at (axis cs:20,4) $ p = 10 $;
[darkgray] (n10.west) – (e10.north);

(axis cs:12.5,0.35) node[ellipse, darkgray, inner xsep=0pt, inner ysep=0pt, minimum width=0.15cm, minimum height=0.4cm, draw] (e5) ;
[inner sep=1pt] (n5) at (axis cs:20,0.06) $ p = 5 $;
[darkgray] (n5.west) – (e5.south);

(axis cs:12.5,0.0035) node[ellipse, darkgray, inner xsep=0pt, inner ysep=0pt, minimum width=0.15cm, minimum height=0.4cm, draw] (e1) ;
[inner sep=1pt] (n1) at (axis cs:20,0.008) $ p = 1 $;
[darkgray] (n1.west) – (e1.north east);

xmin=-5, xmax=30,
ymin=0.00005, ymax=10,
xlabel=SNR [dB],
title=model (<ref>),
legend style=at=(0.0, 0.0), anchor=south west, legend columns=1,

[omp1, discard if not=algorithmrandom]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, omp1, discard if not=algorithmrandom]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, omp1, discard if not=algorithmrandom]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[lbcs1, discard if not=algorithmlbcs]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, lbcs1, discard if not=algorithmlbcs]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, lbcs1, discard if not=algorithmlbcs]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[learned1, discard if not=algorithmlearned]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, learned1, discard if not=algorithmlearned]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, learned1, discard if not=algorithmlearned]
table [ignore chars=",x=snr,y=mean,col sep=comma]

(axis cs:12.5,1.4) node[ellipse, darkgray, inner xsep=0pt, inner ysep=0pt, minimum width=0.15cm, minimum height=0.4cm, draw] (e10) ;
[inner sep=1pt] (n10) at (axis cs:20,4) $ p = 10 $;
[darkgray] (n10.west) – (e10.north);

(axis cs:12.5,0.5) node[ellipse, darkgray, inner xsep=0pt, inner ysep=0pt, minimum width=0.15cm, minimum height=0.4cm, draw] (e5) ;
[inner sep=1pt] (n5) at (axis cs:20,0.13) $ p = 5 $;
[darkgray] (n5.west) – (e5.south);

(axis cs:12.5,0.005) node[ellipse, darkgray, inner xsep=0pt, inner ysep=0pt, minimum width=0.15cm, minimum height=0.4cm, draw] (e1) ;
[inner sep=1pt] (n1) at (axis cs:20,0.014) $ p = 1 $;
[darkgray] (n1.west) – (e1.north);

Evaluation with channel models (<ref>) (left) and (<ref>) (right).
The constant modulus matrices $ \mbA $ have size $ 16 \times 128 $ and the sparsity $ p \in \{1, 5, 10\} $ is varied.
Evaluation is shown with random matrices (<ref>),
with the matrices from <Ref> (<ref>),
and with the matrices from <Ref> (<ref>).

The left plot in <Ref> visualizes evaluation results with model (<ref>) for $ m = 16 $.
Generally, we can see that recovering channels with sparsity $ p = 10 $ from $ m = 16 $ measurements is not possible:
the relative mse is larger than one.
The case $ p = 5 $ shows better performance and the case $ p = 1 $ seems to show perfect recovery up to the remaining error which is due to noise.
Given the fact that we try to recover $ p $ nonzero entries of an $ n = 128 $-dimensional sparse vector from only $ m = 16 $ measurements,
the qualitative behavior of the displayed curves can be expected.

It is worth pointing out that while the performance of the three displayed methods of generating measurement matrices is aligned in the case $ p = 1 $,
we only have one matrix per snr when <Ref> is used
and we only have a single matrix for the whole curve when the proposed <Ref> is used.
In contrast, the random matrices approach generates a new random matrix for every channel.
All three methods lead to equally bad performance for $ p = 10 $ where the limiting factor seems to be the too small number of measurements $ m = 16 $.
Further, for $ p = 5 $, both <Ref> improve on random matrices.

The right plot in <Ref> visualizes the same setting but with model (<ref>).
According to compressive sensing theory, there should not be a performance difference
between the two channel models (<ref>) or (<ref>) when random matrices are employed.
This is due to the fact that random matrices enjoy a universality property
which means that they are incoherent to any basis <cit.>.
However, the case $ p = 1 $ in the right plot clearly behaves differently from the corresponding case in the left plot.
We can understand this behavior when we take into account that both the random constant modulus matrices $ \mbA $ as well as the dft basis $ \mbF $ have
entries of the form $ e^{\imag \phi} $.
This can lead to a large coherence between $ \mbA $ and $ \mbF $ which can degrade the performance.
Remarkably, <Ref> show equally good performance in both cases:
the respective training processes are able to find incoherent enough measurement matrices.
Even in the simple case $ p = 1 $, the observed behavior of the algorithms is already interesting.
It emphasizes the difference in the obtained matrices.

xmin=-5, xmax=30,
ymin=0.00001, ymax=10,
xlabel=SNR [dB],
ylabel=relative MSE,
title=model (<ref>),
legend style=at=(0.0, 0.0), anchor=south west, legend columns=1,

[omp1, discard if not=algorithmrandom]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, omp1, discard if not=algorithmrandom]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, omp1, discard if not=algorithmrandom]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[lbcs1, discard if not=algorithmlbcs]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, lbcs1, discard if not=algorithmlbcs]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, lbcs1, discard if not=algorithmlbcs]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[learned1, discard if not=algorithmlearned]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, learned1, discard if not=algorithmlearned]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, learned1, discard if not=algorithmlearned]
table [ignore chars=",x=snr,y=mean,col sep=comma]

(axis cs:12.5,0.13) node[ellipse, darkgray, inner xsep=0pt, inner ysep=0pt, minimum width=0.15cm, minimum height=0.4cm, draw] (e10) ;
[inner sep=1pt] (n10) at (axis cs:20,0.6) $ p = 10 $;
[darkgray] (n10.south west) – (e10.north);

(axis cs:12.5,0.017) node[ellipse, darkgray, inner xsep=0pt, inner ysep=0pt, minimum width=0.15cm, minimum height=0.4cm, draw] (e5) ;
[inner sep=1pt] (n5) at (axis cs:24,0.2) $ p = 5 $;
[darkgray] (n5.south west) – (e5.north east);

(axis cs:12.5,0.0019) node[ellipse, darkgray, inner xsep=0pt, inner ysep=0pt, minimum width=0.15cm, minimum height=0.4cm, draw] (e1) ;
[inner sep=1pt] (n1) at (axis cs:5,0.0008) $ p = 1 $;
[darkgray] (n1.north) – (e1.south west);

xmin=-5, xmax=30,
ymin=0.00001, ymax=10,
xlabel=SNR [dB],
title=model (<ref>),
legend style=at=(0.0, 0.0), anchor=south west, legend columns=1,

[omp1, discard if not=algorithmrandom]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, omp1, discard if not=algorithmrandom]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, omp1, discard if not=algorithmrandom]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[lbcs1, discard if not=algorithmlbcs]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, lbcs1, discard if not=algorithmlbcs]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, lbcs1, discard if not=algorithmlbcs]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[learned1, discard if not=algorithmlearned]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, learned1, discard if not=algorithmlearned]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, learned1, discard if not=algorithmlearned]
table [ignore chars=",x=snr,y=mean,col sep=comma]

(axis cs:12.5,0.17) node[ellipse, darkgray, inner xsep=0pt, inner ysep=0pt, minimum width=0.15cm, minimum height=0.4cm, draw] (e10) ;
[inner sep=1pt] (n10) at (axis cs:20,0.6) $ p = 10 $;
[darkgray] (n10.south west) – (e10.north);

(axis cs:12.5,0.02) node[ellipse, darkgray, inner xsep=0pt, inner ysep=0pt, minimum width=0.15cm, minimum height=0.4cm, draw] (e5) ;
[inner sep=1pt] (n5) at (axis cs:24,0.2) $ p = 5 $;
[darkgray] (n5.south west) – (e5.north east);

(axis cs:12.5,0.0019) node[ellipse, darkgray, inner xsep=0pt, inner ysep=0pt, minimum width=0.15cm, minimum height=0.4cm, draw] (e1) ;
[inner sep=1pt] (n1) at (axis cs:5,0.0008) $ p = 1 $;
[darkgray] (n1.north) – (e1.south west);

Evaluation with channel models (<ref>) (left) and (<ref>) (right).
The constant modulus matrices $ \mbA $ have size $ 32 \times 128 $ and the sparsity $ p \in \{1, 5, 10\} $ is varied.
Evaluation is shown with random matrices (<ref>),
with the matrices from <Ref> (<ref>),
and with the matrices from <Ref> (<ref>).

<Ref> repeats the experiments for $ m = 32 $.
It can be seen that now all cases $ p \in \{1, 5, 10\} $ show a relative mse below one
which can be interpreted to mean that the number of measurements $ m $ starts to be large enough.
Additionally, we observe that the case $ p = 5 $ starts to be parallel to $ p = 1 $
which is to be expected once $ m $ is large enough.
Further, the incoherence problem for model (<ref>) in contrast to model (<ref>) is not as pronounced anymore
which further supports the provided explanation.
Still, a degradation can be observed in the right plot for $ p = 5 $ and again <Ref> can alleviate the problem.
Lastly, we see how the two learning-based methods improve on random matrices in particular for $ p = 10 $.

We also simulated $ m = 64 $ and $ m = 96 $ where all curves essentially are straight lines and all methods are aligned (not shown here).
At this point, we would like to emphasize again that this is a satisfying result
since, e.g., the proposed <Ref> gives us a single measurement matrix that we use for all snr,
which may offer attractive options for potential technical implementations.

§.§ Discussion of Results for Approximately Sparse Vectors

This subsection concentrates on channel model (<ref>).
<Ref> shows the evaluation of matrices $ \mbA \in \C^{m\times 128} $ with $ m \in \{32, 64, 96\} $
for $ p = 5 $ (left) and $ p = 10 $ (right) paths.
We see that all curves are eventually constant with increasing snr.
The reason for this phenomenon is that the channels are only approximately sparse: $ \mbh \approx \mbPsi_L \mbs $.
Thus, even in the noiseless case, perfect recovery is not possible
whenever angles $ \theta_k $ of some steering vectors lie in between two grid points of the dictionary $ \mbPsi_L $.
There exist different approaches to combat this problem.
On overview of grid-less methods can, e.g., be found in <cit.>.
However, since we are only interested in studying the behavior of different measurement matrices
rather than in the actual channel estimation itself, it suffices for our purposes to work with the on-grid omp algorithm.

In all cases, <Ref> can significantly improve on random matrices.
Consider for example the cases $ m \in \{64, 96\} $ in the left plot of <Ref>.
The performance of the matrices for $ m = 64 $ of <Ref> reaches the performance of random matrices for $ m = 96 $.
In this sense, the learning processes lead to an improvement of the matrix $ \mbA $ which corresponds to a gain of 32 measurements.
Additionally, it can be kept in mind that <Ref> only need one matrix
(per snr in case of <Ref>) in contrast to the random matrices approach.

xmin=-5, xmax=30,
ymin=0.005, ymax=1,
xlabel=SNR [dB],
ylabel=relative MSE,
title=$ p = 5 $,
legend style=at=(1.0, 1.0), anchor=north east, legend columns=1,

[omp1, discard if not=algorithmrandom]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, omp1, discard if not=algorithmrandom]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, omp1, discard if not=algorithmrandom]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[lbcs1, discard if not=algorithmlbcs]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, lbcs1, discard if not=algorithmlbcs]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, lbcs1, discard if not=algorithmlbcs]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[learned1, discard if not=algorithmlearned]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, learned1, discard if not=algorithmlearned]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, learned1, discard if not=algorithmlearned]
table [ignore chars=",x=snr,y=mean,col sep=comma]

(axis cs:7.5,0.17) node[ellipse, darkgray, inner xsep=0pt, inner ysep=0pt, minimum width=0.15cm, minimum height=0.55cm, draw] (e32) ;
[inner sep=1pt] (n32) at (axis cs:18,0.2) $ m = 32 $;
[darkgray] (n32.west) – (e32.north east);

(axis cs:7.5,0.06) node[ellipse, darkgray, inner xsep=0pt, inner ysep=0pt, minimum width=0.15cm, minimum height=0.55cm, draw] (e64) ;
[inner sep=1pt] (n64) at (axis cs:20,0.09) $ m = 64 $;
[darkgray] (n64.south) – (e64.east);

(axis cs:7.5,0.033) node[ellipse, darkgray, inner xsep=0pt, inner ysep=0pt, minimum width=0.15cm, minimum height=0.55cm, draw] (e96) ;
[inner sep=1pt] (n96) at (axis cs:1,0.025) $ m = 96 $;
[darkgray] (n96.north) – (e96.west);

xmin=-5, xmax=30,
ymin=0.005, ymax=1,
xlabel=SNR [dB],
title=$ p = 10 $,
legend style=at=(1.0, 1.0), anchor=north east, legend columns=1,

[omp1, discard if not=algorithmrandom]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, omp1, discard if not=algorithmrandom]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, omp1, discard if not=algorithmrandom]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[lbcs1, discard if not=algorithmlbcs]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, lbcs1, discard if not=algorithmlbcs]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, lbcs1, discard if not=algorithmlbcs]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[learned1, discard if not=algorithmlearned]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, learned1, discard if not=algorithmlearned]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[forget plot, learned1, discard if not=algorithmlearned]
table [ignore chars=",x=snr,y=mean,col sep=comma]

(axis cs:7.5,0.53) node[ellipse, darkgray, inner xsep=0pt, inner ysep=0pt, minimum width=0.15cm, minimum height=0.55cm, draw] (e32) ;
[inner sep=1pt] (n32) at (axis cs:18,0.6) $ m = 32 $;
[darkgray] (n32.west) – (e32.north east);

(axis cs:7.5,0.13) node[ellipse, darkgray, inner xsep=0pt, inner ysep=0pt, minimum width=0.15cm, minimum height=0.55cm, draw] (e64) ;
[inner sep=1pt] (n64) at (axis cs:20,0.07) $ m = 64 $;
[darkgray] (n64.north) – (e64.east);

(axis cs:7.5,0.074) node[ellipse, darkgray, inner xsep=0pt, inner ysep=0pt, minimum width=0.15cm, minimum height=0.55cm, draw] (e96) ;
[inner sep=1pt] (n96) at (axis cs:1,0.045) $ m = 96 $;
[darkgray] (n96.north) – (e96.west);

Evaluation with channel model (<ref>).
The constant modulus matrices $ \mbA $ have size $ m \times 128 $ where $ m \in \{32, 64, 96\} $.
The number of paths is $ p = 5 $ (left) and $ p = 10 $ (right).
Evaluation is shown with random matrices (<ref>),
with the matrices from <Ref> (<ref>) and from <Ref> (<ref>).

We see a considerable performance degradation for the case $ m = 32 $
when $ p $ is increased from 5 to 10 (compare left and right plots in <Ref>).
This may again be interpreted as having a too small number of measurements $ m $.

So far, in all considered cases, <Ref> were at least as good or better than random matrices
and the proposed <Ref> was at least as good or better than <Ref>.
However, this changes if we consider $ m = 96 $ and $ p = 1 $, which is displayed in <Ref>.
In theory, every matrix which can be found via <Ref> is a matrix which could also be found via <Ref>,
so that we could expect the performance of <Ref>
to be comparable to the one of <Ref>.
In addition to the discussed hyperparameters, which we determine via random search,
another important factor of <Ref> is the initialization of the matrix $ \mbPhi $.
In all considered numerical results, for initialization, every element was drawn independently and uniformly in the interval $ [0, 2\pi] $.
It is now an interesting experiment to initialize <Ref> with <Ref> and to observe
whether the learning procedure does at least not degrade the quality of the final matrix.

This was done in <Ref> where we used <Ref> with
$ \mr{SNR} = 20 $ dB to initialize <Ref> before
we run the 64 random searches as described in <Ref>.
The corresponding curve reveals that the two algorithms harmonize well.
Finding a good initialization is a general issue of gradient descent algorithms.
The observation in <Ref>
suggests to make initialization via <Ref> part of the random search:
in one half of all cases, the initialization would be random, in the other half, the initialization would be based on <Ref>.

xmin=-5, xmax=30,
ymin=0.0025, ymax=0.1,
xlabel=SNR [dB],
ylabel=relative MSE,
legend style=at=(1.0, 1.0), anchor=north east, legend columns=1,

[omp1, discard if not=algorithmrandom]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[lbcs1, discard if not=algorithmlbcs]
table [ignore chars=",x=snr,y=mean,col sep=comma]

[learned1, discard if not=algorithmlearned]
table [ignore chars=",x=snr,y=mean,col sep=comma]

table [ignore chars=",x=snr,y=mean,col sep=comma]

Evaluation with channel model (<ref>).
The constant modulus matrices $ \mbA $ have size $ 96 \times 128 $.
The number of paths is $ p = 1 $.
Evaluation is shown with random matrices (<ref>), and
with the matrices from <Ref> (<ref>) and from <Ref> (<ref>).
Further, the result of initializing <Ref> with <Ref> is depicted (<ref>).

§.§ Training Data Normalization

Lastly, we want to discuss training data preprocessing for <Ref>.
The initial idea described in <Ref> is to regard $ \mbA \in \C^{m\times n} $ as a mapping
from one hypersphere (in $ \C^n $) to another hypersphere (in $ \C^m $).
This implies to normalize the training data $ \{ \mbh_i \}_{i=1}^{\Ttrain} $ via

\begin{equation}\label{eq:normalization}
\htilde_i = \frac{\mbh_i}{\|\mbh_i\|}, \quad \forall i = 1, 2, \dots, \Ttrain
\end{equation}

such that $ \htilde_i $ represents a point on the hypersphere in $ \C^n $.

However, if the distribution of $ \mbA \htilde $ now equaled the distribution of $ \mbu $
(cf. (<ref>)), we would have $ \| \mbA \htilde \| = 1 $
which means we would have found a matrix with rip constant $ \delta $ equal to zero,
see (<ref>).
This can be expected to be difficult if not impossible.
In particular, if the number of rows, $ m $, is significantly smaller than the number of columns, $ n $,
then $ \delta $ can be expected to be relatively large.
On the other hand, if $ m $ is larger,
the matrix $ \mbA $ is closer to a square matrix and it is easier to find an (approximate) isometry.
With this intuition in mind, the optimization problem may be difficult to solve
if the data is normalized such that every $ \htilde_i $ lies exactly on the hypersphere in $ \C^n $,
i.e., if we have $ \| \htilde_i \| = 1 $.
To alleviate the problem, we heuristically introduce some variation in the norm of the training data.
Specifically, we normalize the training data as described in equation (<ref>),
where the denominator approximates $ \sqrt{\expec[\|\mbh\|^2]} $.
Consequently, the training data points are only “on average” on the hypersphere in $ \C^n $.
Now, the distribution $ \mbA \htilde $ can be equal to the distribution of $ \mbu $,
but we would not have $ \delta = 0 $.

The effect of the different normalizations is studied in <Ref>
where we consider two extreme cases: few measurements $ m = 16 $ but only a single path $ p = 1 $
and many measurements $ m = 96 $ with many paths $ p = 10 $.
It can be seen that the normalization (<ref>) leads to better results than (<ref>).
The difference is particularly pronounced in the case $ m = 16 $.
This fits the intuition provided above: as $ m $ decreases, $ \delta $ potentially needs to increase.
Hence, training with (<ref>) is more restrictive in that case.
Motivated by <Ref>,
normalization (<ref>) was used in all other numerical experiments presented in this section.

xmin=-5, xmax=30,
ymin=0.005, ymax=2,
xlabel=SNR [dB],
ylabel=relative MSE,
legend style=at=(1.0, 1.0), anchor=north east, legend columns=1, cells=anchor=west,

table [ignore chars=",x=snr,y=mean,col sep=comma]

$ m = 16 $,

[style2, discard if not=delta0.0]
table [ignore chars=",x=snr,y=mean,col sep=comma]

$ m = 16, \| \cdot \| $

[style3, discard if not=delta1.0]
table [ignore chars=",x=snr,y=mean,col sep=comma]

$ m = 16, \sqrt{\expec[\| \cdot \|^2]} $

table [ignore chars=",x=snr,y=mean,col sep=comma]

$ m = 96 $,

[style5, discard if not=delta0.0]
table [ignore chars=",x=snr,y=mean,col sep=comma]

$ m = 96, \| \cdot \| $

[style6, discard if not=delta1.0]
table [ignore chars=",x=snr,y=mean,col sep=comma]

$ m = 96, \sqrt{\expec[\| \cdot \|^2]} $

Dashed curves show evaluation with model (<ref>) for $ m = 16 $ and $ p = 1 $, solid curves for $ m = 96 $ and $ p = 10 $.
Evaluation with random constant modulus matrices is displayed in dark gray (<ref>, <ref>).
Blue curves (<ref>, <ref>) refer to learning a matrix with training data normalization $ \mbh/\|\mbh\| $,
orange curves (<ref>, <ref>) refer to normalization $ \mbh/\sqrt{\expec[\|\mbh\|^2]} $.

§ CONCLUSION

The anchor point of our studies was the interpretation that a measurement matrix
which has the rip constitutes (approximately) a mapping between hyperspheres.
In order to take measurement noise into account,
we proposed to determine the measurement matrix such
that it maps points uniformly onto the hypersphere in its range.
One of the mmd metrics was used to measure the distance between a uniform distribution on the hypersphere
and the distribution in the range of the measurement matrix.
The recent success of mmd-based machine learning applications motivated a solution of the optimization problem via stochastic gradient descent.
Further, it was shown that a modified method from the literature also leads to good performance
and that it can harmonize with our approach by providing a suitable initialization for the learning algorithm.

The algorithm's learning phase requires training data.
This training data may, for example, be obtained by means of a measurement campaign.
The learned matrix will then depend on the measured scenario.
In our experiments, this is reflected by the matrix's dependency on the number of propagation paths.
In this context, it might be interesting to investigate how the performance behaves when the scenario changes.
One might then attempt to employ a form of domain adaptation <cit.> to adaptively update the measurement matrix.

A notable property of the proposed learning algorithm is that it yields one measurement matrix
which can be employed for all snr.
Numerical experiments demonstrate this property and the great performance of the proposed method.

In this paper, we focused on measurement matrices with a constant modulus constraint.
We translated the constraint into the system of equations shown in (<ref>).
For an arbitrary (not necessarily constant modulus) matrix $ \mbC \in \C^{m\times n} $, (<ref>) takes the form

\begin{equation}
\begin{bmatrix}
\Re(\mbC \htilde) \\
\Im(\mbC \htilde)
\end{bmatrix}
\begin{bmatrix}
\Re(\mbC) & -\Im(\mbC) \\
\Im(\mbC) & \Re(\mbC) \\
\end{bmatrix}
\begin{bmatrix}
\Re(\htilde) \\
\Im(\htilde)
\end{bmatrix}.
\end{equation}

It can be seen that more general structural constraints can be incorporated into the design of $ \mbC $ as well.
For example, the authors of <cit.> prove that random Toeplitz matrices can satisfy a rip condition
and use this result to motivate the application of such matrices in a channel estimation context.
A Toeplitz measurement matrix can also be learned with the approach described in <Ref>.
To this end, weight tying can be employed <cit.>.
That is, parameters of $ \mbC $ would be shared between rows to enforce a Toeplitz structure.
Here it makes sense to normalize the rows during training.
Similarly, other structured matrices can be learned too.