disorder: Clinical and imaging evidence for dissociable effects.
Biological Psychiatry, 2019.

[Vaidyam et al., 2022]
A. Vaidyam, J. Halamka, and J. Torous.
Enabling research and clinical use of patient-generated health data
(the mindlamp platform): Digital phenotyping study.
JMIR Mhealth Uhealth, 10, 2022.

[Vaidyam et al., 2019]
A.N. Vaidyam, H. Wisniewski, J.D. Halamka, M.S. Kashavan, and J.B. Torous.
Chatbots and conversational agents in mental health: A review of the
psychiatric landscape.
Can J Psychiatry, 64, 2019.

[van Erp et al., 2014]
T.G.M. van Erp, A. Preda, D. Nguyen, L. Faziola, J. Turner, ..., and O.P.
Converting positive and negative symptom scores between panss and
Schizophrenia Research, 152, 2014.

[vanderGoot et al., 2021]
W.E. vanderGoot, R.J. Duvivier, N.W.V. Yperen, M.A. Carvalho-Filho, K.E. Noot,
..., and W.F.W. Bierman.
Psychological distress among frontline workers during the covid-19
pandemic: A mixed-methods study.
PLoS ONE, 16, 2021.

[vanLoo and Romeijin, 2015]
H.M. vanLoo and J.W. Romeijin.
Psychiatric comorbidity: fact or artifact?
Theor Med Bioeth, 36, 2015.

[VanRullen and Kanai, 2021]
R. VanRullen and R. Kanai.
Deep learning and the global workspace theory.
Trends in Neuroscience, 44, 2021.

[vanSon et al., 2018]
R.J.J.H. vanSon, C. Middag, and K. Demuynck.
Vowel space as a tool to evaluate articulation problems.
19TH ANNUAL CONFERENCE OF THE INTERNATIONAL SPEECH
COMMUNICATION ASSOCIATION, 2018.

[Vaswani et al., 2017]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez,
L. Kaiser, and I. Polosukhin.
Attention is all you need.
NeurIPS, 2017.

[Vijay et al., 2016]
S. Vijay, T. Baltrusaitis, L. Pennant, D. Ongur, J.T. Baker, and L.P. Morency.
Computational study of psychosis symptoms and facial expressions.
ACM Conference on Human Factors in Computing Systems, 2016.

[Vogelsang et al., 2018]
L. Vogelsang, S. Gilad-Gutnick, E. Ehrenberg, A. Yonas, S. Diamond, R. Held,
and P. Sinha.
Potential downside of high initial visual acuity.
PNAS, 115:0 11333–11338, 2018.

[Voppel et al., 2022]
A.E. Voppel, J.N. deBoer, S.G. Brederoo, H.G. Schnack, and I.E.C. Sommer.
Semantic and acoustic markers in schizophrenia-spectrum disorders; a
combinatory machine learning approach.
Schizophrenia Bulletin, 2022.

[Walenski et al., 2010]
M. Walenski, T.W. Weickert, C.J. Maloof, and M.T. Ullman.
Grammatical processing in schizophrenia: Evidence from morphology.
Neuropsychologia, 48, 2010.

[Wallwork et al., 2012]
R.S. Wallwork, R. Fortgang, R. Hashimoto, D.R. Weinberger, and D. Dickinson.
Searching for a consensus five-factor model of the positive and
negative syndrome scale for schizophrenia.
Schizophrenia Research, 137, 2012.

[Walther and Morrens, 2015]
S. Walther and M. Morrens.
Editorial: Psychomotor symptomatology in psychiatric illnesses.
Frontiers in Psychiatry, 2015.

[Wang and Slotine, 2005]
Wei Wang and Jean-Jacques Slotine.
On partial contraction analysis for coupled nonlinear oscillators.
Biological Cybernetics, 92:0 38–53, 2005.

[Waters et al., 2018]
F. Waters, V. Chiu, A. Atkinson, and J.D. Blom.
Severe sleep deprivation causes hallucinations and a gradual
progression toward psychosis with increasing time awake.
Front Psychiatry, 9, 2018.

[Watson, 2023]
C. Watson.
Academia’s lost luster prompts exodus of health researchers to
Nature Medicine, 29:0 6–9, 2023.

[373]
Eric W. Weisstein.
Positive definite matrix.
URL <https://mathworld.wolfram.com/PositiveDefiniteMatrix.html>.

[Wensing and Slotine, 2020]
Patrick M Wensing and Jean-Jacques Slotine.
Beyond convexity—contraction and global convergence of gradient
Plos one, 150 (8):0 e0236661, 2020.

[Widge et al., 2019]
A.S Widge, S. Zorowitz, I. Basu, A.C. Paulk, S.S. Cash, ..., and D.D.
Deep brain stimulation of the internal capsule enhances human
cognitive control and prefrontal cortex function.
Nature Communications, 10, 2019.

[Wiener, 1948]
Norbert Wiener.
Cybernetics or Control and Communication in the Animal and the
MIT press, 1948.

[Williams et al., 2021]
Alex H Williams, Erin Kunz, Simon Kornblith, and Scott Linderman.
Generalized shape metrics on neural representations.
Advances in Neural Information Processing Systems,
34:0 4738–4750, 2021.

[Williams and Kobak, 2008]
J.B.W. Williams and K.A. Kobak.
Development and reliability of a structured interview guide for the
montgomery–asberg depression rating scale (sigma).
The British Journal of Psychiatry, 192:0 52–58, 2008.

[Wilson and Cowan, 1972]
Hugh R. Wilson and Jack D. Cowan.
Excitatory and inhibitory interactions in localized populations of
model neurons.
Biophysical Journal, 120 (1):0 1–24, 1972.
ISSN 0006-3495.

[Wilson, 1993]
M. Wilson.
Dsm-iii and the transformation of american psychiatry: A history.
Am J Psychiatry, 150, 1993.

[Winston, 2011]
P.H. Winston.
The strong story hypothesis and the directed perception hypothesis.
Association for the Advancement of Artificial Intelligence -
Fall Symposium Series, 2011.

[Wood and Gupta, 2017]
J.M. Wood and S. Gupta.
Using rating scales in a clinical setting: A guide for psychiatrists.
Current Psychiatry, 16:0 21–25, 2017.

[Woods et al., 2014]
S.W. Woods, B.C. Walsh, J. Addington, K.S. Cadenhead, T.D. Cannon, ..., and
T.H. McGlashan.
Current status specifiers for patients at clinical high risk for
Schizophrenia Research, 158, 2014.

[Woody et al., 1995]
S.R. Woody, G. Steketee, and D.L. Chambless.
Reliability and validity of the yale-brown obsessive-compulsive
Behav Res Ther, 33:0 597–605, 1995.

[Woolsten, 2022]
C. Woolsten.
Lab leaders wrestle with paucity of postdocs.
<https://www.nature.com/articles/d41586-022-02781-x>, 2022.
Accessed: 2023-04-16.

[Wortwein et al., 2017]
T. Wortwein, T. Baltrusaitis, E. Laksana, L. Pennant, D. Ongur, J.T. Baker, and
L.P. Morency.
Computational analysis of acoustic descriptors in psychotic patients.
Interspeech, 2017.

[Wosen, 2022]
J. Wosen.
‘the tipping point is coming’: Unprecedented exodus of young life
scientists is shaking up academia.
Accessed: 2023-04-16.

[Wroge et al., 2018]
T.J. Wroge, Y. Ozkanca, C. Demiroglu, D. Si, D.C. Atkins, and R.H. Ghomi.
Parkinson's disease diagnosis using machine learning and voice.
IEEE Signal Processing in Medicine and Biology Symposium,

[Wu et al., 2021]
Ruihan Wu, Chuan Guo, Awni Hannun, and Laurens van der Maaten.
Fixes that fail: Self-defeating improvements in machine-learning
arXiv preprint arXiv:2103.11766, 2021.

[Xu et al., 2015]
K. Xu, E.E. Schadt, K.S. Pollard, P. Roussos, and J.T. Dudley.
Genomic and network patterns of schizophrenia genetic variation in
human evolutionary accelerated regions.
Molecular Biology and Evolution, 32:0 1148–1160,

[Yalincetin et al., 2017]
B. Yalincetin, E. Bora, T. Binbay, H. Ulas, B.B. Akdede, and K. Alptekin.
Formal thought disorder in schizophrenia and bipolar disorder: A
systematic review and meta-analysis.
Schizophrenia Research, 185, 2017.

[Yalsin et al., 2023]
E. Yalsin, R. Martinez-Corral, and M. Chugh.
Retaining postdocs by recognizing their worth.
Nature Biotechnology, 41:0 296–298, 2023.

[Yamins et al., 2014]
Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert,
and James J DiCarlo.
Performance-optimized hierarchical models predict neural responses in
higher visual cortex.
Proceedings of the national academy of sciences, 1110
(23):0 8619–8624, 2014.

[Yang and Molano-Mazon, 2021]
Guangyu R Yang and Manuel Molano-Mazon.
Next-generation of recurrent neural network models for cognition, Apr
URL <psyarxiv.com/w34n2>.

[Yang and Molano-Mazón, 2021]
Guangyu Robert Yang and Manuel Molano-Mazón.
Towards the next generation of recurrent network models for cognitive
Current opinion in neurobiology, 70:0 182–192, 2021.

[Yang et al., 2018]
Guangyu Robert Yang, Igor Ganichev, Xiao-Jing Wang, Jonathan Shlens, and David
A dataset and architecture for visual reasoning with a working
In Proceedings of the European Conference on Computer Vision
(ECCV), pages 714–731, 2018.

[Yang et al., 2019]
Guangyu Robert Yang, Madhura R Joglekar, H Francis Song, William T Newsome, and
Xiao-Jing Wang.
Task representations in neural networks trained to perform many
cognitive tasks.
Nature neuroscience, 220 (2):0 297–306, 2019.

[Yang et al., 2013]
Y. Yang, C. Fairbairn, and J.F. Cohn.
Detecting depression severity from vocal prosody.
IEEE Transactions on Affective Computing, 4, 2013.

[Young and Dahleh, 1997]
P.M. Young and M.A. Dahleh.
Infinite-dimensional convex optimization in optimal and robust
control theory.
IEEE Transactions on Automatic Control, 42:0
1370–1381, 1997.

[Zhang et al., 2022]
T. Zhang, A.M. Schoene, S. Ji, and S. Ananiadou.
Natural language processing applied to mental illness detection: a
narrative review.
npj Digital Medicine, 5, 2022.

[Ziburkus et al., 2013]
J. Ziburkus, J.R. Cressman, and S.J. Schiff.
Seizures as imbalanced up states: excitatory and inhibitory
conductances during seizure-like events.
Journal of Neurophysiology, 109:0 1296–1306, 2013.

CHAPTER: SUPPLEMENT TO CHAPTER <REF>

§ ADDITIONAL RESULTS SECTIONS

§.§ Diary pipeline QC metrics validations

To further characterize the pipeline's QC features, we both used manual quality review methods and looked at relationships between audio and transcript quality metrics. This supplementary section will detail the results of that work.

§.§.§ Cross-checking with manual review

A major part of our workflow in characterizing said dataset was careful manual review of a subset of audio diaries and, where available, matching transcripts. Both student volunteers and a lab RA conducted this process, cross-referencing with outputs from my pipeline where applicable. The manual review of quality of TranscribeMe's outputs that was described above is one such example. Now I will report the results of a few different approaches for manual assessment of audio quality done by the lab.

Because manual review looked at a few different subsets of diaries from BLS, I will first review the overall file counts. Of the 11324 BLS audio files, 11101 were deemed non-empty (i.e. volume not NaN). Of those 11101, intentionally forgiving pipeline cutoffs filtered out another 648 of extreme low quality. Thus 10453 files were sent to TranscribeMe over the duration of BLS. 10271 were able to be transcribed, while 182 were returned to us by the service as non-transcribable.

Prior to most files being sent for transcription, a student reviewed a sample of audio recordings to determine what QC cutoffs we might use. The primary goal was to determine what we could filter out while restricting the false positive rate to near-zero (i.e. send liberally). Original cutoff proposals were volume of at least 30 db and length of at least 5 seconds. Thus from the batch of audio recordings available at the time, the student listened to 14 files with db $\leq 30$ and 53 files with duration $\leq 5$ seconds. Of the 14 with low volume, all were found to be nonsense recordings. However 31 of the 53 short files were found to contain meaningful speech, and of the 22 that did not all had volume below 40 db.

The review was therefore expanded to include 108 files with db $\leq 40$. Of these 108, only 4 were found to contain any transcribable speech, and only 1 was deemed by the student to have no quality issue whatsoever. Based on these results, the QC cutoff used for the majority of transcription decisions became volume of at least 40 db but no duration minimum.

The next major round of manual QC review was to assess the 182 files that were rejected by TranscribeMe, to better characterize false negatives produced by the implemented $40$ db threshold. An RA listened to each audio that was returned to us without a transcript and took notes on the contents. They found that 177 of the files were indeed entirely untranscribable, while 1 was an acceptable recording and 4 were difficult but perhaps possible to at least partially transcribe. They also found that the majority (119) of the files were extremely short in duration ($<1$ second) and contained an unrecognizable sound, while most of the remaining files contained excessive background noise of some form (Table <ref>).

[Manual review of recordings with acceptable volume that were unable to be transcribed.]Manual review of recordings with acceptable volume that were unable to be transcribed. The 182 files that passed a volume threshold of 40 db but were subsequently returned by TranscribeMe were reviewed and categorized by an RA. Counts for each quality problem category are provided here. 177 of these 182 were verified to be completely unsalvageable by the RA. The 119 of short length (row 1) could be easily filtered in the future.

Category     File Count

Unrecognizable sound (duration $<1$ second)     119

Static noise     2

Ambient non-speech background noise (e.g. traffic)     30

Ambient speech background noise (e.g. TV/radio)     11

Poor enunciation or otherwise inaudible speech from participant     15

Presence of another human speaker     5

Therefore, we can generally trust TranscribeMe to return bad files that do slip through. Even if they did choose to partially transcribe a largely inaudible file and charge us for that, the total fee here would have been less than 40 dollars for the 182 bad uploads, and the transcript QC would have easily detected such problematic transcripts. Still, the automated QC cutoffs can be improved from this experience. We chose to not filter length because we decided even a single sentence could contain interesting information, and the cost is of course dependent on file length anyway. However, it is clear from this that sometimes when a participant hits stop immediately after hitting start there may be noise picked up that will cause the recording to pass the volume threshold. For the purposes of file organization, on both our end and TranscribeMe's, it makes sense to instead filter out any records of less than 1 second.

With an extremely short length cutoff added, just 62 files of the 11324 originally screened files would have been sent to TranscribeMe and subsequently deemed non-transcribable, while 990 low quality files would have been filtered out automatically before the TranscribeMe transfer – indicating that our basic QC screening has been largely successful in identifying the problematic files. Furthermore, 10271 files were successfully transcribed of 11324 submissions, indicating that quality of recordings is much more often good than not.

It is unclear if light weight QC metrics would be able to accurately detect the issues in the untranscribable files of longer duration (broadly identified in Table <ref>). With the metrics in the present pipeline, there are some correlations, but nothing strong enough to justify filtering entirely. For example, of the 62 files in the untranscribable set that had duration of at least 1 second, the mean volume was 57.44 db, which is significantly smaller than the mean of 66.82 db from across all files with defined volume (Table <ref>). There were a number of files with volume between 40 and 60 that were transcribable, so it is not necessarily recommended to filter these out; at the same time, it could be recommended to monitor for subjects that are regularly submitting journals in this range via DPDash, as well as for RAs to be on the lookout for and advise participants about the quality concerns listed in Table <ref>.

It is worth noting also that 110 of the 182 problematic recordings that were sent to TranscribeMe, including 22 of the 62 with druation greater than one second, came from the same subject ID (M8MXM). Another subject (GFNVM) submitted 10 of the 62 remaining low quality recordings, despite only accounting for 11 of the larger set of 182 (with a mean duration over the 11 of $\sim 30$ seconds). This suggests that participant GF was unlikely to be submitting bad audio intentionally, but still managed to account for a disproportionate chunk of the low quality files. Thus paying careful attention to subject ID is of clear relevance in managing data collection in an audio diary study.

§.§.§ Automatic methods for validating QC measures

The other major part of our workflow to evaluate potential QC features was statistical analysis of the complete dataset. While manual review can produce a more nuanced characterization of particular diaries, it may be subject to human biases, and it would be a massive undertaking to review the entire corpus. Ultimately, one of the core goals of building this pipeline was to reduce manual label required from data collectors. To fully utilize the tool then, it is necessary to understand how generated features are distributed and what correlation structure exists amongst them. Furthermore, because we obtained a large number of high quality transcriptions via TranscribeMe's premium services, I was able to treat certain transcript quality metrics as a sort of "ground truth" about the audio's suitability for transcribing.

To provide initial context, I calculated summary stats for the distributions of the main QC measures in consideration: duration, volume, flatness, word count, and inaudible, questionable, and redacted counts (Table <ref>). Mean recording duration was $\sim 1 \frac{1}{2}$ minutes, but with high variance across BLS journals. Unsurprisingly, the related transcript word count feature also showed high variance, with a mean $\sim 180$ words. This is consistent with the variation in engagement seen across patients, as well as possible diminished engagement within some patients over time (discussed further in Chapter <ref>).

Many of the other QC features, such as spectral flatness and inaudible count, also had high variance, but additionally had low means - suggesting a majority of diaries that would safely pass a quality threshold, and a smaller set that produces a long feature tail. Indeed, a histogram of inaudible count distribution (Figure <ref>D) shows exactly this pattern. Because inaudibles occurred much more frequently than questionables in the TranscribeMe results and are more concretely defined, I utilized the inaudible count metric as the primary endpoint for measuring transcript quality in subsequent analyses. As short free-form patient journals are not likely to contain much PII, the reported redaction counts are in line with expectations, and thus not of interest for further study. See chapter <ref> for discussion of PII in the context of long clinical interviews.

I will now discuss the distributions of volume (Figure <ref>C) and duration (Figure <ref>B) in more detail. About 250 recordings had volume near 0 and could be directly discarded. Recordings with volume $< 30$ dB were otherwise very rare, as were recordings $> 80$ dB. Fewer than 1000 diaries were submitted with volume $< 55$ dB; while some of these may have been transcribable, they in general produced lower quality transcripts, as reported below. Thus the relatively small number of recordings $< 55$ dB in our dataset suggests that 55 could be a suitable dB threshold for future Beiwe studies.

Recording length is a quality metric where the distribution of specific patient's diaries can be plainly found making up part of the larger study distribution. Nearly 2500 submitted diaries had length $< 15$ seconds, and there is a clear drop-off in the histogram from that point up to 2 minutes. However a second hump can be seen for recordings between 2 minutes and 2 minutes 15 seconds, as well as a final bump at the maximum recording length of 4 minutes. The histograms of diary duration in particular patients of note (3SS93, 8RC89) make this even clearer, as the different participants have distributions that peak at clearly different lengths, from both each other and the larger study distribution (Figure <ref>A, top row).

It is worth noting that recording length need not correspond to the actual amount of content in the journal submission. One big confound, to be discussed in section <ref>, is the presence of pauses and the tendency of particular participants to pause longer than others. Indeed, the estimated distributions of speech duration for 3SS93 and 8RC89 (Figure <ref>A, bottom row) are vastly different when taken in the context of their respective submission duration distributions. In sum, 3SS93 submitted 3768 minutes of successfully transcribed audio, but registered only 1837 speech minutes when run through voice activity detection. 8RC89 submitted 662 total minutes, with 513 minutes registered as speech time.

A different simple metric that does not run into this problem is the transcript word count. 3SS93 had a total of 328,040 words across their transcripts, while 8RC89 had a total of 110,679 words. Thus their ratios of word count, speech duration, and recording duration were $\sim 0.34$, $0.28$, and $0.18$, respectively. This lends credence to the pipeline pause detection algorithm, which will be further validated in subsequent sections. Of course word count is not a perfect metric, as simpler language can generate higher word counts for similar content, and pause time information may itself be clinically meaningful. Moreover, recording duration is much easier to screen than word count as it does not rely on transcript availability.

As mentioned, in addition to inspecting the individual quality control features, I also looked at relationships between audio QC features within the BLS dataset. Both shorter diaries and quieter diaries tended to have higher mean spectral flatness across the entire audio set (Figure <ref>), suggesting a correlation between red flags in different measures of submission quality. However, given the primary goal was to identify features which improve accuracy of the pipeline's quality screening, I was more interested in whether diaries of standard length and volume, but with higher spectral flatness, tended to be of lower quality. Therefore I focused most of my investigation into the connections between audio quality control metrics and the corresponding transcript quality control features. These final analyses were thereby restricted to diaries that were successfully transcribed.

[Relationship between audio diary length, spectral flatness, and volume in the BLS dataset.]Relationship between audio diary length, spectral flatness, and volume in the BLS dataset. All audio recordings of at least 1 second submitted by BLS participants, whether transcribed or not, were assessed for potential relationships between audio quality control features. Each point on this scatter plot represents a single diary submission, with x location corresponding to the recording's length in minutes, y location corresponding to its mean spectral flatness, and color corresponding to its volume (binned - see legend, ranges in dB).

For each diary, the number of inaudibles marked by TranscribeMe was divided by the total length of the recording in minutes to mitigate bias against longer diaries. I then used inaudibles/minute as my primary outcome feature. Pearson correlation between inaudibles/minute and volume was very minimally negative ($r = -0.0413$, $p = 0.00003$), and Pearson correlation between inaudibles/minute and duration was only slightly stronger in effect ($r = -0.0878$, $p < 10^{-18}$). Because the distributional analysis suggested potential nonlinear relationships between features, I also computed the Spearman rank correlation, which resulted in stronger correlations and in the case of duration flipped directionality ($r = -0.0747$, $p < 10^{-13}$ for volume; $r = 0.1627$, $p < 10^{-61}$ for length). These correlations were calculated using the entire set of transcribed non-empty diaries (minimum 1 second duration).

Given the limited utility of very short journals for automated analyses, as well as the large outliers such journals were able to produce in the inaudibles/minute metric, I did a deeper dive into these feature correlations using only submissions of duration $\geq 15$ seconds. A scatter plot of volume versus inaudibles/minute with points colored by duration (Figure <ref>A) shows many diaries of near-maximum length with a non-zero number of inaudibles, regardless of volume. This explains the statistically significant positive Spearman correlation between length and transcript quality, and it is likely a result of poor recording environment and/or enunciation in the submissions from 3SS93 – who accounted for over $\frac{1}{3}$ of all max length diaries (Figure <ref>A) and also submitted a large number of untranscribable diaries (Figure <ref>C). Additionally, many moderate length diaries had an even higher rate of inaudibles, particularly when dB was outside the 65-75 peak distributional range.

[Audio QC metrics versus resulting transcript QC from across BLS diaries.]Audio QC metrics versus resulting transcript QC from across BLS diaries. All transcribed BLS diaries of at least 15 seconds were analyzed for correlation between audio QC features and transcription quality. Here, various audio metrics are compared against the number of inaudibles marked in the transcript per minute. Overall volume (in dB) serves as the x-axis for one such scatter plot (A), while mean spectral flatness serves as the x-axis for the other (B). For clarity, points with flatness > 0.05 were excluded. Both plots have their points colored by the diary length, with the shortest recordings dark blue and the longest dark red.

Because of the clear relationship between mean spectral flatness and recording duration (Figure <ref>), I also scattered flatness against inaudibles/minute with the same duration hue (Figure <ref>B). The lowest quality transcriptions of long recordings actually tended towards having particularly low mean flatness.

Although scatter plots can be informative in understanding the interplay between different values, they are not able to cleanly represent large datasets that have many overlapping values. Here, the majority of returned transcripts had no inaudible words, but that entire part of the distribution is minimized in Figure <ref>. As potential rank relationships were noticed at both extremes of the various quality metrics, I next investigated the distribution of inaudibles/minute in sets of diaries falling into different QC categories.

Of the 8557 transcribed recordings containing 15 or more seconds of audio, 28 had volume between 40 and 50 dB, 886 had volume between 50 and 60 dB, 4147 had volume between 60 and 70 dB, 3472 had volume between 70 and 80 dB, and 24 had volume between 80 and 90 dB. The distribution of inaudibles per minute across these categories shows that volume between 60 and 80 dB is the most likely to result in high quality transcriptions (Figure <ref>). While volume $> 80$ dB or $< 60$ dB should not necessarily be automatically skipped, it may be worth flagging these recordings for RA review or holding them back subject to transcription budget constraints.

[Distribution of words marked inaudible by TranscribeMe in BLS audio diaries of different volume levels.]Distribution of words marked inaudible by TranscribeMe in BLS audio diaries of different volume levels. Transcribed BLS audio journals with duration at least 15 seconds were binned based on their overall volume metric. This Seaborn box and whisker plot shows for each category (40-50 dB, 50-60 dB, 60-70 dB, 70-80 dB, and 80-90 dB) the distribution of the inaudibles/minute diary metric. For clarity, outliers (defined as outside 1.5 times the interquartile range) were excluded from this plot. The median in all cases is 0, but we can see variation in the 75th percentiles (tops of colored boxes) as well as the maximum non-outlier values (tops of whiskers).

In the same set of transcribed audio recordings, 8 had mean spectral flatness above 0.1, 61 had mean spectral flatness between 0.025 and 0.1, 1756 had mean spectral flatness between 0.01 and 0.025, 2926 had mean spectral flatness between 0.0025 and 0.01, and 3806 had mean spectral flatness below 0.0025. In general, the lower the flatness bin, the higher proportion of quality transcripts produced (Figure <ref>A), but only $> 0.1$ had systemic quality issues worthy of a hard cutoff. When focusing in only on diaries within the best dB range (60-80), a similar relationship between flatness and inaudible words remained, indicating that flatness may indeed be a feature worth independently considering (Figure <ref>B). Flatness between 0.025 and 0.1 underwent the largest change in quality distribution between the two datasets, so that using a harsher flatness cutoff when dB is outside the 60-80 range may be advisable. Similarly, for diaries of duration at least 2 minutes, mean flatness below 0.025 (and arguably even 0.01) becomes a recommended hard requirement for quality (Figure <ref>D). In diaries between 15 seconds and 2 minutes however (Figure <ref>C), the relationship is more similar to that seen in the full transcript set (Figure <ref>A). For future study, it may be worth considering an additional flatness bin to capture values nearer 0. There is still some weak evidence, as well as theoretical justification, for concern over extremely low spectral flatness in longer files.

[Distribution of inaudible frequency in BLS diary transcripts varies with recording spectral flatness.]Distribution of inaudible frequency in BLS diary transcripts varies with recording spectral flatness. Transcribed BLS audio journals with duration at least 15 seconds were also binned based on their mean spectral flatness metric. The distribution of the inaudibles/minute diary metric was visualized for each flatness category via violin plot (A). For clarity in the violin plots, inaudibles/minute was capped at 5 - with any exceeding that number rounded down here. An additional violin plot restricting the dataset to only those diaries with volume between 60-80 dB was also produced (B), to determine if any relationship remains when only the samples with volume corresponding to the highest quality are considered. Because of the demonstrated relationship between mean flatness and length, the distributions were examined with further restrictions on length as well, with violin plots considering only diaries between 15 second and 2 minutes (C) and only diaries longer than 2 minutes (D).

As a final characterization of this quality metrics dataset, I looked more closely at the suggested bimodal relationship between diary length and transcript inaudible frequency. Indeed, diaries $> 3$ minutes in length fared notably worse with transcript quality (Figure <ref>A), even when considering only diaries with volume in the best-identified 60 to 80 dB range (Figure <ref>B). This highlights the value of considering features in addition to volume when filtering for QC. However, the specific relationship here is very unlikely to be generalizable to other datasets; as discussed, it is very likely driven by submission patterns of specific BLS patients. Besides that, small correlations could also be an artifact of the normalization method, as short diaries have fewer opportunities for an inaudible to present, but when one does will be more heavily penalized by the length denominator. This latter effect is clearly visible in the shape of the violin plots for the first three length buckets in Figure <ref>.

[Distribution of inaudible frequency in BLS diary transcripts of different lengths.]Distribution of inaudible frequency in BLS diary transcripts of different lengths. In the same fashion as Figure <ref>A, a violin plot of the inaudibles/minute distribution for the different diary length bins (in minutes) was produced (A). As before, a second violin plot further restricting the dataset to only those diaries with volume between 60-80 dB was also produced (B).

Still, it is important to be aware of how recording length can confound various quality features, as there are flaws both in normalizing by it and in not doing so. It is also important to more generally know your own dataset. Hence any quality cutoffs or screening techniques suggested here are meant only as a rough starting point, to be tailored to specific studies based on patient population, primary research questions, and budget (among other things). The above results may be utilized as a blueprint for doing so. The key contribution is not only in providing foundational code and corresponding expected outputs for sanity checks, but also in the detailed documentation of issues to watch out for and potential additional use cases of produced features. This will remain a theme throughout the chapter.

§.§ Detailed distributions of select diary features across the BLS dataset

Using the dataset and features described at the beginning of section <ref> in the main chapter, I first evaluated overall distributions by visualizing histograms with matplotlib. To contextualize the data, it is particularly important to understand how speech production related features are distributed (Figure <ref>). As discussed, information like verbosity and pause duration have been well-established to hold clinical relevance, and these properties also affect other features in a way that is difficult to perfectly normalize for.

Total word count across journals (Figure <ref>A) varied widely, with $\sim 2500$ of the transcripts containing fewer than 100 words and $\sim 2000$ of the transcripts containing at least 300 words. A question for upcoming sections is thus how much of the word count variance occurs within versus between subjects. The percent of time spent speaking feature (Figure <ref>C) also demonstrated a high level of meaningful variance across journals, indicating a strong potential for downstream analyses. About half of the diaries contained participant speech during at least $75\%$ of the recording, while well over 1000 diaries contained participant speech for just $60\%$ or less of the recording duration.

[Distribution of speech production related features in the final BLS audio journal dataset.]Distribution of speech production related features in the final BLS audio journal dataset. Histograms of each per diary feature were generated over the set of quality-filtered BLS submissions ($n=8398$). Features relevant to speech production are shown here: the total number of words (A), the fraction of recording duration that was spent speaking (B), the mean pause duration in minutes (C), and the mean sentence speech rate in syllables per second (D). The plot in (D) was axis limited for clarity, so that 5 diaries with estimated syl/sec $<1$ and 15 with syl/sec $>7$ were excluded. Note that this speech rate metric was derived from transcript timestamps, so it incorporates longer pauses; for a pause-normalized version of the feature, see Figure <ref>A.

Estimated speech rate (Figure <ref>B) was distributed like a Gaussian centered at $\sim 3$ with standard deviation $\sim 1$. With 817 recordings having speech rate below 2 syllables/second, this distribution is quite different than what one would expect from prior knowledge on human speech rate. However, it is not inconsistent with expectations here, because the methodology focused on transcript syllable count and timestamps, such that long pause times were incorporated into the present speaking rate metric. Because this feature distribution looks promising for further analysis, another question for upcoming sections is the separation of long pauses from the speech rate feature.

Linguistic disfluencies (Figure <ref>) are of particular interest due to their relevance for the onsite interview language analysis presented in chapter <ref>. Characterizing the differences between disfluency frequency in the two formats, especially in a patient-specific way, would greatly facilitate future work on the questions posed in section <ref>.

[Distribution of linguistic disfluency features in the final BLS audio journal dataset.]Distribution of linguistic disfluency features in the final BLS audio journal dataset. Histograms of each per diary feature were generated over the set of quality-filtered BLS submissions ($n=8398$). Counts for the four major categories of linguistic disfluency are depicted here: nonverbal edits (A), verbal edits (B), repeats (C), and restarts (D). Recall that each of these features was normalized by transcript word count for analysis. The plot in (C) was axis limited for clarity, so that 3 diaries with repeats/word $>0.1$ were excluded.

Nonverbal fillers (Figure <ref>A) were substantially more common than verbal fillers (Figure <ref>B) in this dataset, which is similar to our observations in Figure <ref>. Nonverbal fillers had no relationship with conceptual disorganization in the analysis of chapter <ref>, and the relationship of verbal filler word use with conceptual disorganization was questionable at best. Because prior work suggests fillers are relevant to other symptom domains in psychotic disorders, it will be interesting to further evaluate them in this broader audio journal dataset. The lack of correlation between nonverbal and verbal edits in the interview dataset (Figure <ref>) further motivates their dissection here.

Restarts (Figure <ref>D) were also noticeably more common than repeats (Figure <ref>C) in the diary dataset. This difference existed in the interview distributions of Figure <ref>, but was much less pronounced. As repeats and restarts were the most impactful linguistic features for modeling conceptual disorganization in chapter <ref>, the healthy distribution of restarts across these audio journals inspires confidence in follow up work using the diary format to assess this kind of disfluency.

It is important to note that the transcribed interview set used in chapter <ref> (<ref>) contained many fewer timepoints from fewer patients than the audio journals do, so distributional differences could be attributable to these factors rather than anything different about the format of a semi-structured interview – though that does have the potential to cause change in disfluency usage. Regardless, the broad consistencies observed between the two formats here is good justification for more research on disfluencies in audio journals, as the journals are much easier to collect and cheaper to carefully verbatim transcribe.

As was discussed in section <ref>, sentence-level sentiment outputs were often 0, and in our hands skewed positive when non-zero. Of course, both the semi-structured interview protocol and the daily audio journal prompt we used for BLS would lend themselves to many neutral sentences, so future research using a prompt of stronger valence ought to evaluate sentiment distribution with that in mind.

Because of these observations in our dataset, I included both mean (Figure <ref>A) and minimum (Figure <ref>B) sentence sentiment as diary-level features under final consideration. The mean was quite evenly centered around $\sim 0.2$, and the vast majority of journals had mean sentiment fall between $-0.25$ and $+0.5$. More than 6500 journals had a negative minimum sentence sentiment, while 1471 had minimum sentiment exactly 0. The length of the journal will have a clear impact on these features, so while the minimum sentiment appears to have differential utility from the mean sentiment based on their distributions, this will be a question for additional study in the upcoming sections.

[Distribution of sentiment and incoherence features in the final BLS audio journal dataset.]Distribution of sentiment and incoherence features in the final BLS audio journal dataset. Histograms of each per diary feature were generated over the set of quality-filtered BLS submissions ($n=8398$). Outputs from the VADER sentiment (top) and Google News word2vec (bottom) NLP tools are represented here. For sentiment, the diary-level features of focus are the mean (A) and minimum (B) sentence sentiment scores. For the word embeddings, the diary-level features of focus are the maximum sentence incoherence (C) and the mean word uncommonness (D). Incoherence for a sentence was defined pairwise, as the mean angle between all words in the sentence, while word uncommonness was the mean vector magnitude.

The word2vec-derived features (Figure <ref>C/D) under consideration have units that are much less immediately interpretable than all the previously discussed features. We are very likely only interested in abnormally high levels of incoherence rather than abnormally low (one of the reasons for taking the maximum instead of the mean), so the 500+ submissions with incoherence $>1.5$ in Figure <ref>C will be a topic of interest in upcoming sections.

Word uncommonness (Figure <ref>D) has been less of a focus of prior literature in psychotic disorders, though high levels of uncommonness have been related to positive symptoms in a task context. It is plausible that low levels of word uncommonness would associate with negative symptoms, but it's unclear how strong the effect would be when verbosity is controlled for. As such, word uncommonness is included here as a more experimental feature.

§.§ Highlighted relationships between diary features in BLS

For the following select feature pairings with interesting Pearson correlation results (Figure <ref>), I created scatter plots to better understand each relationship:

* Fraction of recording duration spent speaking versus mean pause length (Figure <ref>)

* Mean sentence sentiment versus minimum sentence sentiment (Figure <ref>)

* Mean word uncommonness versus maximum sentence incoherence (Figure <ref>)

* Frequency of restarts versus fraction of recording duration spent speaking (Figure <ref>)

All scatters also had their data points colored based on word count, to further elucidate any potential dependency each relationship may have on overall verbosity.

[Fraction of recording duration spent speaking versus mean pause length, word count hue.]Fraction of recording duration spent speaking versus mean pause length, word count hue. Each diary in the final BLS dataset is plotted as a point here, with x-coordinate corresponding to the Speaking Time/Recording Time value for that diary and y-coordinate corresponding to the Mean Pause Length in minutes for that diary. Each diary point is colored according to its word count, from dark blue at $\leq 200$ crossing over to dark red at $\geq 800$.

[ Mean sentence sentiment versus minimum sentence sentiment, word count hue.]Mean sentence sentiment versus minimum sentence sentiment, word count hue. Each diary in the final BLS dataset is plotted as a point here, with x-coordinate corresponding to the Mean Sentence Sentiment for that diary and y-coordinate corresponding to the Minimum Sentence Sentiment for that diary. Each diary point is colored according to its word count, from dark blue at $\leq 200$ crossing over to dark red at $\geq 800$. Note there is no indicator for perfectly overlapping dots, so it may be difficult to interpret the role of 0 sentiment in this relationship.

[ Mean word uncommonness versus maximum sentence incoherence, word count hue.]Mean word uncommonness versus maximum sentence incoherence, word count hue. Each diary in the final BLS dataset is plotted as a point here, with x-coordinate corresponding to the Mean Word Uncommonness value for that diary and y-coordinate corresponding to the Maximum Sentence Incoherence value for that diary. Each diary point is colored according to its word count, from dark blue at $\leq 200$ crossing over to dark red at $\geq 800$.

[ Frequency of restarts versus fraction of recording duration spent speaking, word count hue.]Frequency of restarts versus fraction of recording duration spent speaking, word count hue. Each diary in the final BLS dataset is plotted as a point here, with x-coordinate corresponding to the Restarts Count/Word Count value for that diary and y-coordinate corresponding to the Speaking Time/Recording Time value for that diary. Each diary point is colored according to its word count, from dark blue at $\leq 200$ crossing over to dark red at $\geq 800$. Note there is no indicator for perfectly overlapping dots, so it may be difficult to interpret the role of 0 restarts in this relationship.

The strongest observed correlations (Figure <ref>A) of speaking fraction with mean pause length (Figure <ref>) and of mean versus minimum sentence sentiment (Figure <ref>) had the expected clear trends on display in scatter plot form. In future work, a closer investigation into outliers on these plots could prove quite valuable; outliers may be used to uncover interesting time points in a case study, detect bugs or unwanted noise in feature extraction methods, further dissect relationships between features into interpretable independent components, and serve as important inputs for rare event detection algorithms – among other purposes. However for the aims of the EMA modeling to be performed here in section <ref>, there is lesser value in including both features when they demonstrate a strong relationship, in particular when that relationship itself has some correlation with verbosity (as in Figures <ref> and <ref>).

For mean word uncommonness versus maximum sentence incoherence (Figure <ref>), it is unsurprising that shorter diaries populated both extremes in uncommonness, as well as dominated the low end of max incoherence. However even when considering only short or only moderate length diaries, there was a clear relationship between uncommonness and incoherence, in particular with the lower mean word uncommonness records typically also having lower maximum sentence incoherence. As moderate length diaries (word count $\sim 400$) reflect the broadest swath of participants, this correlation captured a fairly general relationship in the dataset. Given the nature of the trend, it is most likely a property of the maximum incoherence summary stat in part: if max incoherence is high, it means at least one sentence must contain at least somewhat uncommon words (the most common words will have low angle between each other), and because diaries contain a relatively small total number of sentences (even at max 4 minute duration) the presence of that sentence will likely prevent the overall mean uncommonness score from being especially low. Of course, understanding other factors that very well may be at play – such as a link between certain properties across sentences in the same diary – would require further investigation.

§.§ Pilot sentiment analyses in the journals of participant 3SS93

As mentioned in the main chapter, early investigations of VADER sentence sentiment and derived diary-level sentiment scores helped to shape the pipeline, and in part involved looking at item-level EMA Spearman correlations with sentiment in a small subset of 3S's data. Because this provides a small preview into some extensions to the EMA summary score linear modeling performed on a much larger dataset within chapter <ref>, the results of the early sentiment validation work are reported in a bit more detail in this section.

The pilot correlational analysis between language sentiment (mean sentence VADER score) and EMA responses in the first $\sim 6$ months of participant 3S's data uncovered some interesting trends, particularly in consideration of item-level EMA correlations (Figure <ref>). Note that in this analysis, EMA responses were not adjusted to align positive and negative scoring – so a high score on a negative property was bad while a high score on a positive property was instead (generally) good. Overall, rating of happiness was the highest positively correlated question with diary language sentiment, and rating of stress was the most negatively correlated question.

[ A closer look at BLS patient 3SS93: diary sentiment scores correlate with same-day mood reported on daily surveys.]A closer look at BLS patient 3SS93: diary sentiment scores correlate with same-day mood reported on daily surveys. The same $\sim 6$ month journal dataset from 3SS93 as shown in Figure <ref> was also used in a pilot correlational analysis with same-day self reported EMA survey responses. For each EMA question, as well as the sum of all positively-worded questions and the sum of all negatively-worded questions, the Spearman correlation between survey score and mean diary sentiment is reported here, with $95\%$ confidence intervals. Questions are ordered based on $r$ value. Note that the EMA scores by default are non-negative integers, with stronger agreement corresponding to a higher score.

Of course there were some EMA items with little immediate relevance to diary sentiment, such as stomach pain on the negative side or sleep quality on the positive side. Presence or absence of these item-based relationships with diary properties in an individual patient may itself be informative, as there can be a modulating effect from both the topics a participant chose to talk about in the diaries as well as the level of interference that a specific symptom assessed by EMA (e.g. sleep disruption) had on their daily life.

§.§ Characterization of the pipeline's sentence to sentence incoherence feature

Of the $n=8398$ journals in the final BLS analysis dataset of section <ref> in the main chapter, $15$ contained only one sentence as marked by TranscribeMe. Therefore we had $n=8383$ final records with a defined mean sentence to sentence (sequential) incoherence score. As described, this between sentence incoherence metric was calculated by obtaining a vector for a given sentence via the mean of all word2vec-derived word vectors in the sentence. The angles between the vectors for subsequent sentences were then averaged across the transcript. Significant subject-specific distributional differences were again seen in the between sentence incoherence feature (Figure <ref>). The KS test results for all 3 subject comparisons were significant by the previous described criteria, with $D=0.13$ ($p < 10^{-10}$) for 3S versus all of BLS, $D=0.32$ ($p < 10^{-35}$) for 8R versus all of BLS, and $D=0.36$ ($p < 10^{-35}$) for 5B versus all of BLS.

[ Comparison of sentence-to-sentence incoherence distributions between different patients of interest.]Comparison of sentence-to-sentence incoherence distributions between different patients of interest. Using the same methodology as in Figure <ref>, additional subject-specific feature distributions are compared against each other and against the distribution across the final BLS dataset, with both violin plots (A) and smoothed distribution curves (B). Here we consider the mean sentence-to-sentence incoherence score of each transcript, as a potential supplement to the previously discussed mean word uncommonness and maximum (within-)sentence incoherence metrics.

Of course, to some extent significant subject-specific distributional differences were to be expected based on relative diary lengths. For a mean over sentences we would expect lower variance in longer diaries, which did appear to be the primary driver of the distributional difference between 3S and BLS at large. As the angle for each sentence was determined by the mean of the contained words, we would also expect sentence length to play a role in this feature distribution. However the fact that 8R and 5B both had sentence-to-sentence incoherence distributions shifted upwards towards higher incoherence (Figure <ref>) does suggest a potential participant-dependent role for this feature independent of verbosity. This would be consistent with prior clinical expectations for 8R, and also would align with preliminary conclusions from the in depth review of the maximum within sentence incoherence metric for 5B. Indeed, differences in the sentence-to-sentence incoherence between 3S, 8R, 5B, and the rest of BLS can be clearly seen to transcend the total transcript word count when visualized as a scatter plot (Figure <ref>).

[ Participant-dependent distributional variation in sentence-to-sentence incoherence is not entirely explainable by verbosity differences.]Participant-dependent distributional variation in sentence-to-sentence incoherence is not entirely explainable by verbosity differences. Mean between sentence incoherence values were negatively correlated with transcript word count across the final BLS dataset (Pearson's $r=-0.338$), as can be seen in the scatter of all transcripts presented here; each point has x-coordinate corresponding to word count and y-coordinate corresponding to mean sentence-to-sentence incoherence score (A). However, the subject-specific distributional differences seen in Figure <ref> were not purely an artifact of the verbosity differences amongst the highlighted subjects. I also scattered word count against the between sentence incoherence measure in the participant subsets only (B). 8RC89 (green) in particular displayed an upward shift in mean sentence-to-sentence incoherence across the range of observed transcript lengths.

In order to further understand the role of verbosity in the reported results, as well as the relationship between the sentence-to-sentence incoherence and the other word2vec-derived features, I next looked at the correlations of these features of interest with the sentence-to-sentence incoherence metric across BLS. There were moderate magnitude and highly significant correlations (all $p < 10^{-200}$) found with mean uncommonness (Pearson's $r=0.383$), max sentence incoherence (Pearson's $r=0.358$), and word count (Pearson's $r=-0.338$). On the other hand, the correlation with sentence count was negligible (magnitude of $r < 0.04$) and insignificant after multiple testing correction – though we of course would still expect higher variance in the feature in journals with smaller sentence counts. Note that the results of the nonlinear Spearman rank correlation between all of these feature pairings were very similar to the linear results.

As was presented in section <ref> of the main chapter, correlation magnitude $> 0.3$ was relatively high amongst the considered journal features. Furthermore, note that the correlation between maximum within sentence incoherence and word count was positive, as would be mathematically expected, so it is interesting that the between sentence coherence had a fairly strong positive correlation with max sentence incoherence alongside a fairly strong negative correlation with word count. The correlation between mean uncommonness and sentence-to-sentence incoherence was also notably stronger than that between mean uncommonness and within sentence incoherence. Taken together, it is unclear how much information the between sentence incoherence would actually add that is independent of the originally considered feature set (Figure <ref>).

[ Mean word uncommonness versus mean sentence-to-sentence incoherence, maximum sentence incoherence hue.]Mean word uncommonness versus mean sentence-to-sentence incoherence, maximum sentence incoherence hue. Each diary in the final BLS dataset is plotted as a point here, with x-coordinate corresponding to the Mean Word Uncommonness value for that diary and y-coordinate corresponding to the Mean Sentence to Sentence Incoherence value for that diary. Each diary point is colored according to its Maximum Sentence Incoherence (pairwise) metric, from dark blue at $\leq 1.2$ crossing over to dark red at $\geq 1.8$. The between sentence incoherence feature turned out to more strongly correlate with both mean uncommonness and max (within-)sentence incoherence than those two features did with each other (Figure <ref>).

Taken together with the lack of correlation with transcript sentence count, there was likely an especially strong relationship between mean sentence-to-sentence incoherence and words per sentence. Such a relationship would also be consistent with expectations based on the definition of the sentence-to-sentence incoherence feature. It is therefore important to consider how the mean transcript words per sentence varied over the final dataset and how it related to our other features of interest before moving on to further analyses. This is especially true because the sentence splitting was based on human transcriber judgement rather than automatic tokenization, which may capture interesting trends in speaking patterns but may also somewhat confound a subset of the downstream features extracted.

To finish the evaluation of the between sentence incoherence metric, I reviewed the relationship of sentence length with sentence-to-sentence incoherence (Figure <ref>). The negative correlation was exceptionally strong for this dataset (Pearson's $r = -0.733$); combined with the relationship of sentence-to-sentence incoherence with the other word2vec features reported above (Figure <ref>), the marginal value of including sentence-to-sentence incoherence in the intentionally limited modeling feature set is highly questionable.

[ Words per sentence versus sentence-to-sentence incoherence, word count hue.]Words per sentence versus sentence-to-sentence incoherence, word count hue. Each diary in the final BLS dataset is plotted as a point here, with x-coordinate corresponding to the mean words per sentence in that diary and y-coordinate corresponding to the mean between sentence (sequential) incoherence value for that diary. Each diary point is colored according to its word count, from dark blue at $\leq 100$ crossing over to dark red at $\geq 800$. Of the NLP journal features considered in this section, the sentence to sentence incoherence feature was thus by far the most correlated with basic verbosity and transcript structure measures.

§.§ Model results for psychosis-related EMA responses from GFNVM

Hallucinations EMA summary scores were binned into binary labels indicating some self-reported hallucination experience or no self-reported hallucination experience. For the delusions EMA score, binning was also performed, into baseline ($= 2$), lower symptoms ($< 2$), and higher symptoms ($> 2$) categories. This was done despite the nicer distribution because there was only a single delusions-related EMA item in the primary BLS EMA, so linear regression would have high error due to the large gaps between possible scores. 0/1 and 3/4 responses were combined due to the relative infrequency of 0 and 4. I then performed linear regression with discrete labels using the logits (and mnlogits) function of statsmodels. I tested verbosity-only models first, but prediction was hardly above chance and model fit was insignificant for both EMA categories.

I then independently tested pause-related, disfluency-related, incoherence-related, and sentiment features in separate models for each of the feature categories and each of the 2 EMA types. However training classification performance remained generally poor at near chance for most of these models. Only the model of pause-related features for hallucinations EMA seemed potentially promising, with the speech fraction feature the main contributor. A logistic regression model using speech fraction alone to predict hallucinations EMA response produced the best performance of this set, though still not at all strong. It could classify $\sim 60\%$ of the training points correctly without class bias, and had model fit $p = 0.013$. Considering the distribution of speech fraction in the two different hallucinations cases (Figure <ref>), the difference being fit by the model was clear, with more of the overall records that had speech fraction $> 0.8$ corresponding to days where non-zero hallucinations were self-reported.

[ Fraction of audio diary containing detected speech versus self-reported hallucinations from GFNVM.]Fraction of audio diary containing detected speech versus self-reported hallucinations from GFNVM. In the psychosis-related EMA training set, the speech fraction feature derived from the pipeline's pause detection was the most relevant journal feature for linear prediction of participant GF's survey responses, though this relationship was weak (if real). Here the distribution of speech fraction over diaries where the corresponding EMA self-reported no hallucinations (blue) is compared against the distribution of speech fraction over diaries where the corresponding EMA self-reported some level of hallucinations (transparent red). Journals with $> 80\%$ of their duration filled with detected vocals were more likely to match to a non-zero hallucinations EMA, while those with $< 80\%$ were more likely to match to a zero hallucinations EMA.

In the test set there were 9 of 15 total non-zero hallucination records with speech fraction $> 0.8$ (7 of those exceeding $0.85$), but there were only 9 of 18 zero hallucinations records with speech fraction $< 0.8$ (with 13 below $0.85$). It is therefore not possible to draw any clear conclusion about the relationship between speech fraction and self-reported psychosis symptoms in GF. Recall also that GF tended to speak with particularly poor enunciation at times, and some of their most mumbling files caused pauses to be missed by the current pause detection algorithm. It is therefore plausible that there is a real relationship here that is concentrated in the extremes, though it may not relate directly to pausing.

Ultimately, there were no meaningful relationships between diary features and same day psychosis-related EMA summary scores in participants GF that could be expressed via basic logistic regression. It is unclear to what extent this is a lack of information in the diary features for this type of prediction versus a timescale mismatch for such predictions versus a weakness of linear coefficients versus a random noise problem in GF's hallucinations and delusions EMA responses.

§.§ Extra diary examples from manual review

§.§.§ Sentiment

[top=-0.25cm,bottom=0.4cm,left=-0.25cm,right=-0.25cm]00:01.32         Um, yesterday I ended up feeling kind of sick in the morning, but then I wasn't a– um, getting manic like I thought I would when I saw my boyfriend and felt, um, [inaudible] hypermania, um, and it's a good feeling, I guess.

00:21.07        I mean, again, I know it's not natural, but, um, it's been a while since I've been hypermanic so it's kind of nice to have a– I don't feel terrible right now, um, but I know I'm kind of sleep deprived because I couldn't fall asleep very easily and get much sleep the past few nights, and I'm, I'm hoping I can just get some sleep.

[top=-0.25cm,bottom=0.4cm,left=-0.25cm,right=-0.25cm]00:03.28         I understand it all– it started off– I have to record it, my day, which started off draining a little bit and dragging, but I drank a lot of caffeine, and I felt better seeing my boyfriend and should be able to sleep well tonight.

[ The two most positive GFNVM diaries by mean sentence sentiment were not in the manual review top 5.]The two most positive GFNVM diaries by mean sentence sentiment were not in the manual review top 5. The 2 GF transcripts with the highest mean VADER sentence sentiment scores are reproduced here, in rank order. These journals were not selected as especially positive by our manual reviewer, and were likely assigned an inflated sentiment by the pipeline due to the way TranscribeMe split these sentences – though that likely itself contains information of interest on GF's pattern of speech on these days, and it is particularly interesting in light of the multiple mentions of hypomania in the $\#1$ diary.

§.§.§ Incoherence (within sentence)

The top 4 journals by within sentence incoherence are reproduced for GFNVM in Figures <ref>-<ref>, as 1 overlapped with the between sentence list. Finally, the top 5 journals by within sentence incoherence are reproduced for 8RC89 in Figures <ref>-<ref>.

[top=-0.5cm,bottom=0.1cm,left=-0.5cm,right=-0.5cm]00:02.78         Um, my boyfriend had, um, made a comment, um, that he was only attracted to skinny girls and that I could lose some weight.

00:29.07         And I know I'm fat and all and I, I have body issues and I have eating disorder issues from my past.

00:38.18         And I don't know.

00:42.28         It really hurt me because I thought– I thought he thought I was pretty.

00:54.45         And then I was like, "Why are you with me?"

00:57.07         And I mentally broke.

01:01.68         And I still– thoughts won't leave my head.

01:05.67         They keep attacking me and telling me just to run and just to kill– like, I don't wanna die.

01:17.49         But this is like, why live?

01:19.29         It's just like all these hopeless despairing thoughts, you know?

01:24.72         They just won't shut up, no matter what.

01:30.11         And it's driving me mad ever since.

01:32.84         And I also found out the, the month, um, he had broken up with me, he really had like a serious relationship with a girl who was like, [inaudible] away and that was sexual.

01:51.85         And, like, I don't know.

01:57.0        I just– and then he just comes crawling back to me and it's like, "How dare you?"

02:02.16         Apparently just didn't go well with her.

02:08.49         And it's like, wha-what am I then?

02:11.25         A-and I freaked out on him and, um, [inaudible] and just, was, like, all was just crying.

02:19.26         And it was– I couldn't mentally handle it.

02:22.45         I couldn't mentally keep it inside anymore.

02:24.72         And then he had a panic attack while driving, thinking I was gonna leave him and he didn't want me to leave him.

02:32.34         And I just don't know– I don't know what to do anymore.

02:39.83         I'm just, like– my perfect world is destroyed.

02:43.95         And I'm trying not to let my eating disorders come back.

02:51.99         But I'm trying– I, I am really not trying.

02:57.68         I am reducing my diet and restricting and monitoring things closely again.

03:03.96         And I just need to make sure it doesn't get out of control because, like, I would want to think I'm pretty.

03:12.28         Other than that, at the same time, I'm like– it's like I want to hurt him.

03:19.52         And then I– and the– but I want to, like, just cut myself and make him happy with me.

03:26.11         Like, it's just, like– It's a– I'm just all– I don't know what to do.

03:31.52         And I'm so lost and sad and confused.

03:36.45         And I hate myself again, more than ever.

03:38.95         And I never wanted to be back in this position.

03:42.65         And I didn't think– I didn't think he was like that.

03:47.1          I would s– I don't know.

03:51.58         Guess I just wish it was said earlier.

03:54.03         So just don't get back with me if you're not physically attracted to me.

03:58.61         I don't know.

[top=-0.5cm,bottom=0.1cm,left=-0.5cm,right=-0.5cm]00:02.22         Well, I ended up coming off my Abilify, so, um, I started off just kind of oversleeping and feeling like crap and then started having really vivid dreams and, um, after that, I was able to finally kinda get up, get showered, try to get myself moving.

00:21.86         I missed an appointment.

00:22.88         I didn't really care.

00:23.73         I was just so out of it and slept through my alarm and was just too late by the time I realized, and, um, now I'm just, like, really antsy, and, um, I feel like crap.

00:39.47         I mean, it's just, I guess from stopping the medicine, I'm just, like, weird, like, at night now.

00:44.9          I'm feeling antsy and, like, jittery and can't move or stop moving, and, like, all day I've been just, like, oversleeping and, like, acting like my depressed state where I just sleep too much when I get in that kinda zone.

00:58.64         And now I'm, like, antsy but tired and– but I can't sleep 'cause I'm antsy 'cause I'm moving my legs and stuff, but I'm just, like, so unsettled and, like, can't feel restful, can't feel, like, relaxed.

01:10.15        It's just, like, constant just, like, ugh, this weird energy, and I just, I don't know, I'm just– I'm not looking forward to going back to how I was, but at the same time I am in so many ways 'cause I can't stand the way that I feel right now, and that's how I've been feeling.

01:24.15        And, and so, like, I've experienced both today and it's like well, they both suck, but, you know, what am I gonna do?

01:30.2          And I'm just in a weird [inaudible].

[ Within sentence incoherence metrics identify a different kind of anomalous journal for GFNVM.]Within sentence incoherence metrics identify a different kind of anomalous journal for GFNVM. The $\# 1$ most incoherent GFNVM journal as ranked by our maximum within sentence incoherence (pairwise) summary feature was the same as the $\# 5$ most incoherent journal chosen by the sentence to sentence incoherence feature in Figure <ref>, discussing orientation. The remaining 4 transcripts in the top 5 of the within sentence incoherence ranking were not particularly high on sentence to sentence incoherence and spanned a fairly wide range. However they appeared to identify a different type of speaking pattern sometimes used by GF that may also be of interest. Those 4 journals are thus reproduced in rank order, with the first two in this Figure and the following two in Figure <ref>. The connection between sentences is generally more clear than what was seen in the transcripts selected by the sentence to sentence ranking (Figure <ref>), with more total sentences here as well. On the other hand, the transcripts selected by this within sentence ranking tended to contain more "run on" sentences, and more extreme thoughts can be found within some of them. For example, within the first journal depicted here, GF says: "And I still– thoughts won't leave my head. They keep attacking me and telling me just to run and just to kill– like, I don't wanna die."

[top=-0.25cm,bottom=0.3cm,left=-0.25cm,right=-0.25cm]00:00.41        Uh, a totally good day overall with my boyfriend.

00:02.81         I got my homework done even though it was a little late.

00:05.94         Um, I did have one mental breakdown just because I was triggered by, um, my new habits and [inaudible] and feeling guilty.

00:16.94         I wasn't respecting my diet as I should and just being triggered and crying and just– this was just over the top emotional for really no reason.

00:30.95         He apologized and said it was [inaudible].

00:34.43         Like [inaudible] is fine now and [inaudible] kind of passed on to, like, a [inaudible] just to kind of review my system [inaudible].

00:47.43         And we watched a movie and, um, just hang around [inaudible] time together [inaudible] again tomorrow.

01:00.35         I'm glad to have had the day off [inaudible] even though I did get a little kind of irritable and probably frustrated with him earlier where I didn't [inaudible] come across mean, but I was kinda mean.

01:14.52        [inaudible] best interest [inaudible].

01:19.88         I apologized.

[top=-0.25cm,bottom=0.3cm,left=-0.25cm,right=-0.25cm]00:01.25         Um, I didn't have a good day today.

00:03.29         I, uh– I'm not–

00:08.98         I'm, I'm fat, you know?

00:12.06         My boyfriend just casually said that he is only attracted to pretty girls.

00:18.16         And then he realized what he said was offensive.

00:20.62         And he was like, "But you could just lose a little weight, you know?"

00:26.61         And [I'm like?], "[inaudible] spot."

00:31.72         I don't–

00:39.77         [Making?] sense.

[ More journals from GFNVM displaying high within sentence incoherence.]More journals from GFNVM displaying high within sentence incoherence. Follow-up to Figure <ref>, now presenting the number 4 and 5 ranked transcripts by within sentence incoherence for subject GF.

[top=-0.5cm,bottom=0.1cm,left=-0.5cm,right=-0.5cm]00:00.41         Uh, Monday, the 25th.

00:03.01         Um, we wen– we got coffee with my mom and her brother, Uncle [redacted].

00:08.84          He's visiting.

00:09.54         He's going back tomorrow.

00:11.62         Uh, it's good.

00:13.17         Things went well.

00:13.85         [redacted] got along with everybody, got along me, [redacted], them.

00:18.68         Then I worked from 5 to 11, pretty easy shift, um, just reading through people's files, was learning stuff about them.

00:27.21         Um, that's about all.

00:28.87         Hanging in there.

00:30.23         Not sure if I'm gonna stop at the bar after work or not.

00:32.38         I have the car this time.

[top=-0.5cm,bottom=0.1cm,left=-0.5cm,right=-0.5cm]00:00.68        Uh, Wednesday, I, uh, got up at the last possible moment, but I managed to get on the bus, get to a psychiatrist's appointment at [redacted], which is like two miles away, um, on time.

00:10.74         Saw her, um, told her about my vitamin regimen, said my sleep is getting better but still not totally normal.

00:17.39         Sometimes I sleep way more than I should, but it's getting more regular.

00:20.78         I can track it with my Fitbit now.

00:22.59         Um, also saw my, uh, s– therapist.

00:27.45         Um, then I went to group therapy.

00:29.61         All good times.

00:31.2          I didn't tell anyone about hooking up with my ex.

00:33.39         Um, keeping that to myself.

00:36.68         Um, I mean, I could, but I chose not to.

00:41.66         Um, you know, no one's perfect.

00:43.64        Uh, yeah, so then I'm going to [redacted] at, um, 7:00.

00:49.09         There's a speaker.

00:51.08         Um, so yeah, I had an okay day, got what I needed to get done, done.

00:55.93         I was supposed to hang out with [redacted], uh, tonight, but, um, she said she wasn't feeling well and postponed it to Friday.

01:03.35         I said, "I gotta be up early. I'm doing the [inaudible] walk on Saturday, so can we do Saturday?" and she said, "Yeah. That sounds great."

01:09.53         So, yeah, looking forward to that.

01:11.37         I'll be good to see her, um, you know, dinner and a movie, and hopefully some time alone together will be nice.

01:19.64         So yeah, that's about all.

01:21.66        I'm doing well.

01:24.05         Um, gonna work tomorrow and the next day.

01:26.6          Um, so, yeah, doing all right.

[top=-0.5cm,bottom=0.1cm,left=-0.5cm,right=-0.5cm]00:00.81         Uh, Wednesday I didn't go to DBSA.

00:04.67         Um, I, uh– was that the day I was so tired I went home at 2:00?

00:10.67         I don't know.

00:11.01         I didn't ask my boss, but somehow word got around that I was just dead

00:14.7          She let me go early.

00:16.94         So I guess I'm getting some, uh, you know, benefits to having a– being a new parent.

00:26.47         So that's all right.

00:28.01         I just wish they wouldn't have scheduled me six days in a row.

00:30.59         It just it gets very tiring.

00:34.01         And then having, like, more than two or three days off, there's just kind of a long [inaudible] return.

00:40.32          Not, not that beneficial, but that's my whole– that's what I am.

00:45.23         I'm a relief employee.

00:46.33         I'm a relief staff, so my whole purpose is to fill in the gaps.

00:49.95         So I don't know.

00:51.75         Hopefully, my schedule will be more regular as time goes on, but I'm not married to this job.

00:57.37         It'd be nice if I get a higher-paying job that's not horrible.

01:02.92         We'll see.

[ Within sentence incoherence metrics identify a different kind of anomalous journal for 8RC89.]Within sentence incoherence metrics identify a different kind of anomalous journal for 8RC89. The top 5 most incoherent 8RC89 journals as ranked by our maximum within sentence incoherence (pairwise) summary feature are reproduced in rank order, with the first three in this Figure and the following two in Figure <ref>. They were quite different than those identified by the mean sentence to sentence incoherence score for 8R (Figure <ref>). However the within sentence metric appeared to uncover interesting thoughts buried in longer diaries that were not identified in the sentence to sentence case – like the same metric ranking did for GFNVM (Figure <ref>). From this review we can also identify recurring topics of importance to the patient, as they often pop up in journals even when disconnected from other points. For 8R, a closer look into mentions of romantic relationships and of alcohol consumption over time could be of particular salience.

[top=-0.55cm,bottom=0.05cm,left=-0.5cm,right=-0.5cm]00:02.13         Um, been doing all right.

00:05.46         Uh, working.

00:06.68         Worked today.

00:07.04         It was pretty calm.

00:08.8         People acted up only a little bit, um.

00:13.28         I, uh, tidied up my room because I wanna have this girl [redacted] over that I'm dating.

00:16.78          We're supposed to hang out Saturday, tomorrow, but I haven't heard from her about time and place to meet up.

00:21.87         I hope I hear from her.

00:22.7          I hope she didn't decide, for some reason, that she doesn't like me or doesn't want to see me or talk to me.

00:29.3          Um, you know, I'm probably just being paranoid, though.

00:32.3          She's probably just busy with her own stuff.

00:36.83         You know, she's running around trying to get some benefits from social security, um, so it's a lot of paperwork and lawyers and stuff.

00:46.1          Every night this week, I've slept over at my ex [redacted]'s, and, um, she's, like, more willing to cuddle, which is nice, uh, s– you know, sort of giving me some of what I need, I guess.

00:57.65         But, you know, I fee– I kind of feel like saying to her sometimes, "Regardless of the reason you, uh– you dumped me, you really hurt me."

01:06.14         "I'm not over it."

01:09.34         "On some level, I still wanna be with you, but I know you'll probably just break my heart a third time."

01:14.64         But almost like what's the point?

01:18.71         Gonna go see, um– see her later.

01:20.63         Her friend [redacted]'s in town.

01:23.54         So I'm gonna see her and [redacted] and her friend [inaudible] and this random guy that likes [redacted].

01:28.68         And, uh, you know, last time [redacted] was here, we kind of hooked up.

01:31.8         Um, probably nothing will happen between us, but, um, you know, be tempting if she makes any overtures, you know, like.

01:42.49         She checks a lot of boxes for me, the red hair, the pale skin, um, yeah, so.

01:51.22         We'll have to see what happens.

01:52.4          Oh, I've been drinking off and on.

01:55.68         Uh, my mom found out.

01:56.98         She was not happy, but at least I don't have to go to a program so far.

02:04.04        Um, I might have to go back [inaudible] tomorrow, so I might have some drinks tonight, stop and buy something on the way.

02:11.35         I don't know, we'll see how it goes.

02:14.41         Doing all right.

[top=-0.55cm,bottom=0.05cm,left=-0.5cm,right=-0.5cm]00:00.97         Thursday I worked.

00:03.03         Um, I think I spent the night at my mom's house.

00:07.0          Um, it was a smooth day at work.

00:09.76         [redacted] didn't get out, unfortunately.

00:11.32         Unfortunately, I'm still banned from seeing her.

00:13.74         Got a real Nurse Ratched type up there at the psych ward.

00:16.03         They're very strict.

00:17.27         Um, yeah, hanging in there.

00:21.41         Hanging out with the cats when I'm there doing laundry, washing their dresses, hanging them up to dry.

00:26.74         The ones that, you know, have special care instructions.

00:30.17         It's the least I can do.

00:31.86         Um, yeah, that's about it.

00:35.08          I'm not getting that much rest.

00:36.4           I'm not sleeping that well.

00:38.01         Fall asleep watching Netflix.

00:40.53         Then the cats wake me up digging their razor sharp little claws into me.

00:44.87         Yeah, hard times.

00:47.16         We'll maintain, we'll survive.

00:49.89         Hopefully, stay together.

00:51.12         I love her so much.

[ More journals from 8RC89 displaying high within sentence incoherence.]More journals from 8RC89 displaying high within sentence incoherence. Follow-up to Figure <ref>, now presenting the number 4 and 5 ranked 8R transcripts by within sentence incoherence.

§.§ Other supplemental figures

§.§.§ More example pipeline outputs

[ Distributions of a few diary features of clinical interest for the BLS study as a whole, compared to two patients of note.]Distributions of a few diary features of clinical interest for the BLS study as a whole, compared to two patients of note (3S and 8R). Audio length and speech length distributions are compared to illustrate differences in patient diary recording times and pause times (A). The distribution of recording length across the study (left) is skewed towards shorter submissions, but both 3S (middle) and 8R (right) make longer submissions in general (top) - particularly 3S. However it is also clear from these distributions that 3S takes up much more time with pauses in speech than 8R does, as can be seen in the speech length histograms (bottom). Similarly, transcript-detected disfluency occurrences also vary by patient and by disfluency type (B). Patient 3S has a relatively low number of verbal edits (top) compared to diaries of a similar length in the larger study, yet they frequently engage in restarts (bottom).

[ Pearson correlation matrix of key diary features from across the BLS study.]Pearson correlation matrix of key diary features (both acoustic and linguistic) from across the BLS study. Features are ordered using hierarchical clustering results from the same dataset. Matrix is colored using the PrGn matplotlib colormap, with limits of -1 to 1. Note these are the core day-level features output by the standard pipeline, described in section <ref>.

[ Effects of pause filtering on BLS diary OpenSMILE results.]Effects of pause filtering on BLS diary OpenSMILE results. Distributions of select features of particular relevance (pitch and jitter) from Table <ref> (A, right) are contrasted with the same features computed without labeled pauses removed from the OpenSMILE results i.e. taking the mean over the raw output (A, left). To illustrate what is occurring on a diary level, distributions over the 10ms OpenSMILE output bins are plotted for the underlying pitch (top) and jitter (bottom) features in an example diary from patient 3SS93 (B). This patient was chosen due to their frequent use of pausing. The particular diary shown is from day 16, the same used to demonstrate VAD spectrograms in Figure <ref>.

§.§.§ More on participant-specific diary submission dynamics

[ Submission day versus submission time with word count hue for BLS subject 3SS93.]Submission day versus submission time with word count hue for BLS subject 3SS93. The methodology of Figure <ref> is repeated here for only the diaries submitted by participant 3SS93. However, the x-axis is no longer artificially limited, so that the full time course of data from 3S is depicted here.

[ Submission day versus submission time with word count hue for BLS subject 8RC89.]Submission day versus submission time with word count hue for BLS subject 8RC89. An analogous plot to the one created for participant 3SS93 in Figure <ref> was also created for 8RC89 here.

[ Submission time versus word count with study day hue for BLS subject 5BT65.]Submission time versus word count with study day hue for BLS subject 5BT65. The methodology of Figure <ref> is repeated here for only the diaries submitted by participant 5BT65. Thus this scatter represents a small subset of the points on the original Figure <ref>, demonstrating an interesting subject-specific dynamic in the relationships between these features.

§.§.§ Further characterization of modeling results: participant-dependent effects

[ Subject-specific effects in word count versus negative EMA.]Subject-specific effects in word count versus negative EMA. Figure <ref> from within chapter <ref> is reproduced here without a limit on the x-axis, to show the full range of 8R's data.

[ Diary word count and mean sentence sentiment had a strong nonlinear relationship with self-reported mood scores in longer 5BT65 recordings.]Diary word count and mean sentence sentiment had a strong nonlinear relationship with self-reported mood scores in longer 5BT65 recordings. For the primary model training dataset, data points with journal duration $< 15$ seconds were filtered out, which removed a large chunk of records for subject 5B in particular. Despite this, word count remained relevant to EMA scores in participant-specific model fitting. Of the new feature set, sentiment was also found to be highly relevant. Although linear model fits were already quite strong, scattering word count and sentiment against each other with EMA hue revealed additional potential nonlinear structure. Here, diary word count is plotted on the x-axis and mean diary sentiment is plotted on the y-axis in each panel, across the 5B diaries in the duration-filtered modeling training set. Positively-worded (A) and negatively-worded (B) EMA item means color the two scatters using the coolwarm colormap with seaborn. The positive map (A) runs from -1 to 2 here and the negative map (B) from 0 to 3, as 5B rarely submitted extremely high levels of positive emotions or negative emotions.

[ Adding sentiment context to the positive correlation between diary word count and 8RC89 negative emotions self-report.]Adding sentiment context to the positive correlation between diary word count and 8RC89 negative emotions self-report. Word count (x-axis) of diary submissions by 8R in the main model training set was plotted versus mean negatively-worded EMA score (y-axis) from the corresponding day. This is very similar to the 8R relationship depicted in Figure <ref>, but points are now colored according to the diary's mean sentiment score, with range -0.25 to 0.5. It is clear here that sentiment can help to distinguish between journals with a similar word count in predicting negative EMA rating from 8R, though the word count relationship with EMA is stronger in this subject.

§.§ Other supplemental tables

§.§.§ Pipeline summary stats for BLS

[Summary of diary QC metric distributions in BLS.]Summary of diary QC metric distributions in BLS. The first three rows are the key audio quality metrics considered, with mean and standard deviation taken over 11,101 submitted recordings with defined volume (a small subset of the submitted diaries were completely empty and thus returned a dB of -inf). The final four rows are key transcript quality metrics, with mean and standard deviation taken over all 10,271 successfully transcribed diaries. For exact feature calculation details, see section <ref>.

Feature     Mean     Standard Deviation

Duration (minutes)     1.49     1.325

Volume (dB)     66.82     9.51

Spectral Flatness     0.0209     0.0804

Total Word Count     180.3     156.2

Inaudible Word Count     0.34     1.11

Questionable Word Count     0.15     0.52

Redacted Word Count     0.78     1.7

[Summary of highlighted audio feature distributions in BLS.]Summary of highlighted audio feature distributions in BLS. The first three rows are diary-level features derived directly from the pause detection algorithm. The final five rows are diary-level features derived from the OpenSMILE results, with pause times filtered out. The raw OpenSMILE features are returned in 10ms bins, and pause times are used to label those bins that correspond to speech. Each of the features in the table represents the mean of these speech bins over the recording. For both the pause-derived features and the OpenSMILE features, the table presents the mean and standard deviation taken over all 10,271 successfully transcribed diaries (diaries unable to be transcribed were considered also too low quality for acoustic analyses). For exact feature calculation details, see section <ref>.

Feature     Mean     Standard Deviation

Speech Duration (minutes)     1.040     0.836

Pause Duration (minutes)     0.481     0.609

Pause Count     33.3     32.2

Mean Pitch (F0)     20.03     3.97

Mean F1     565.28     55.10

Mean F2     1560.61     77.22

Mean Jitter     0.0407     0.0163

Mean Shimmer     1.025     0.242

[Summary of highlighted transcript feature distributions in BLS.]Summary of highlighted transcript feature distributions in BLS. The first row is an estimation of sentiment. The following four rows measure different types of linguistic disfluencies. The next two rows are features derived from semantic coherence metrics. The final row is an estimate of speaking rate based on computed syllables and the TranscribeMe timestamps. For all of these diary-level features, the table presents the mean and standard deviation taken over all 10,271 successfully transcribed diaries. For exact feature calculation details, see section <ref>.

Feature     Mean     Standard Deviation

Mean Sentence Sentiment     0.164     0.219

Nonverbal Edit Count     8.52     8.60

Verbal Edit Count     2.09     4.25

Restart Count     3.03     4.26

Repeat Count     1.66     2.71

Mean Word Uncommonness     2.27     0.20

Mean Sentence Incoherence (pairwise)     1.35     0.05

Mean Sentence Speaking Rate (syllables/second)     3.14     1.48

§.§.§ Participant-specific distributional comparison (KS test results)

[Subject-specific differences observed in BLS audio journal feature distributions.]Subject-specific differences observed in BLS audio journal feature distributions. For each of the highlighted BLS participants, the two sided Kolmogorov-Smirnov (KS) test, performed with the scipy.stats python package, was used to quantify how different the empirical distribution of each feature in the patient's dataset was from the empirical distribution of the same feature in the overall BLS dataset. The obtained KS D statistics are reported in this table, with one results column per considered subject ID and one results row per considered journal feature. For all D stats that met the conservative Bonferroni-corrected cutoff of $p \leq 0.0005$, p-value information is also given in parentheses. The vast majority of these comparisons were highly significant with large effect size, demonstrating that subject-specific factors are highly relevant to analysis of audio journal features.

Feature Distribution Compared (subject vs all BLS)     3SS93 D stat     8RC89 D stat     5BT65 D stat

Word Count     0.553 ($p<10^{-14}$)     0.104 ($p<10^{-4}$)     0.57 ($p<10^{-101}$)

Speaking Time/Recording Time     0.724 ($p<10^{-5}$)     0.198 ($p<10^{-16}$)     0.282 ($p<10^{-22}$)

Mean Pause Length (minutes)     0.71 ($p<10^{-6}$)     0.27 ($p<10^{-30}$)     0.219 ($p<10^{-13}$)

Speech Rate (syllables/second)     0.636 ($p<10^{-8}$)     0.488 ($p<10^{-105}$)     0.291 ($p<10^{-24}$)

Restarts Count/Word Count     0.476 ($p<10^{-15}$)     0.105 ($p<10^{-4}$)     0.352 ($p<10^{-36}$)

Repeats Count/Word Count     0.509 ($p<10^{-15}$)     0.096  ($p<0.0003$)     0.348 ($p<10^{-35}$)

Nonverbal Edits Count/Word Count     0.3 ($p<10^{-15}$)     0.102 ($p<10^{-4}$)     0.07 NS

Verbal Edits Count/Word Count     0.276 ($p<10^{-15}$)     0.148 ($p<10^{-8}$)     0.501 ($p<10^{-76}$)

Mean Sentence Sentiment     0.222 ($p<10^{-15}$)     0.187 ($p<10^{-14}$)     0.108 NS

Minimum Sentence Sentiment     0.294 ($p<10^{-15}$)     0.075 NS     0.294 ($p<10^{-25}$)

Maximum Sentence Incoherence (pairwise)     0.157 ($p<10^{-15}$)     0.206 ($p<10^{-17}$)     0.071 NS

Mean Word Uncommonness     0.194 ($p<10^{-15}$)     0.166 ($p<10^{-11}$)     0.12 ($p<0.0002$)

[Using random simulation to identify subject-specific distributional differences attributable to verbosity (KS test results).]Using random simulation to identify subject-specific distributional differences attributable to verbosity (KS test results). The same protocol used to compare the patient-specific feature distributions to the study-wide ones in Table <ref>, including the same significance cutoff, was repeated here on simulated datasets. Using the true distribution of diary sentence counts for each subject and for all of BLS, sentence-level feature values were randomly and independently sampled from underlying distributions designed to simulate the sentiment and incoherence features (exact methods detailed above). Distributions of analogous diary features were then obtained by taking the appropriate summary stats over the simulated data. This table thus presents the D stats obtained by running KS tests on the simulated subject-specific distributions versus the simulated study-wide ones. The stats here are from the final run of simulations, but for all iterations on the underlying sentence distribution parameters, every D stat was within $+/- 0.05$ of the numbers reported here, and the non-significant (NS) comparisons remained consistent. These D stats serve as an estimate of the contribution of verbosity to the observed D stats for the corresponding sentiment and coherence features in Table <ref>.

Feature Distribution Compared (subject vs all BLS)     3SS93 D stat     8RC89 D stat     5BT65 D stat

Randomized Mean Sentiment     0.073 NS     0.058 NS     0.104 NS

Randomized Minimum Sentiment     0.214 ($p<10^{-15}$)     0.117 ($p=0.000003$)     0.238 ($p<10^{-16}$)

Randomized Maximum Incoherence     0.205 ($p<10^{-15}$)     0.085 NS     0.272 ($p<10^{-21}$)

Randomized Mean Uncommonness     0.09 ($p<10^{-5}$)     0.072 NS     0.084 NS

§.§.§ Participant-specific submission stats

[Diary participation stats for top BLS subjects - frequency of back to back submissions.]Diary participation stats for top BLS subjects - frequency of back to back submissions. For each of the $25$ patients with at least 100 journals submitted of sufficient length, summary stats were computed about the temporal dynamics of the submissions. The first two counts provided are the total number of recordings $\geq15$ seconds submitted by that subject, and then the number of those days where the subject also submitted a $\geq15$ second recording the following day. The next two counts are the same statistics but restricted to only the first year of the study period for each participant. Then the final four columns are fractions derived from these counts: the observed probability of diary submission given a diary was submitted the prior day, the same probability given it is also year one, the fraction of days in year one where a diary was submitted, and the fraction of the total dataset for that subject that is covered by year one. Heterogeneity in journal participation behavior is observed between participants.

Patient ID     Diary Count     Next Day Count     Diary Count (Year 1)     Next Day Count (Year 1)     Next Day / Total     Next Day / Total (Year 1)     Year 1 / $365$     Year 1 / Total

3SS93     1287     1233     345     325     0.958     0.942     0.945     0.268

7NE49     744     436     164     89     0.586     0.543     0.449     0.22

SBPQM     560     351     268     181     0.627     0.675     0.734     0.479

8RC89     515     172     182     79     0.334     0.434     0.499     0.353

GMQBM     451     265     211     134     0.588     0.635     0.578     0.468

2XC43     369     160     129     59     0.434     0.457     0.353     0.35

5BT65     345     216     141     72     0.626     0.511     0.386     0.409

5KXYM     341     291     314     275     0.853     0.876     0.860     0.921

CYG4M     333     222     231     163     0.667     0.706     0.633     0.694

GFNVM     329     200     231     137     0.608     0.593     0.633     0.702

DQDCM     311     165     156     94     0.531     0.603     0.427     0.502

8ADCM     266     153     134     67     0.575     0.5     0.367     0.504

Z8YRM     262     198     262     198     0.756     0.756     0.718     1.0

M8MXM     256     176     247     174     0.688     0.705     0.677     0.965

V8GMM     203     154     203     154     0.759     0.759     0.556     1.0

8KX53     199     93     178     90     0.467     0.506     0.488     0.895

ACMCM     170     84     163     79     0.494     0.485     0.447     0.959

B3CRM     147     97     147     97     0.66     0.66     0.403     1.0

5QXNM     130     27     65     14     0.208     0.215     0.178     0.5

5CR39     126     98     126     98     0.778     0.778     0.345     1.0

TQJKM     117     63     117     63     0.539     0.539     0.321     1.0

9SU83     111     41     88     33     0.369     0.375     0.241     0.793

V65HM     109     68     95     61     0.624     0.642     0.26     0.872

JVBKM     108     69     108     69     0.639     0.639     0.296     1.0

F6VVM     103     63     103     63     0.612     0.612     0.282     1.0

In addition to daily submission availability, I also looked at patterns in submission timing and potential relation with resulting diary content richness, as mentioned in the main chapter section <ref>. For each possible Eastern Time (ET) submission hour, Table <ref> presents the total number of diaries in the final BLS dataset assigned to that hour bin, as well as the mean and standard deviation of the total word counts of all such diaries. Recall that recordings submitted prior to 4am were counted to the previous day, so that assigned hours ranged from 4 to 27 - thereby ensuring that late night submissions would not be counted as early morning in the correlational analyses mentioned.

[Diary metadata stats by submission time in the final BLS dataset.]Diary metadata stats by submission time in the final BLS dataset. The number of diaries assigned to each possible submission time integer is reported here; the diary counts match the number of points that fall onto that submission time in Figure <ref>. The mean and standard deviation of diary word counts across the subset of each submission hour is also reported, to provide additional context for Figure <ref> - as overlapping points are not clearly visible on the scatter.

Submit Time (integer)     Submit Time (ET)     Diary Count     Mean Word Count     StDev Word Count

4     4am     98     210.2     164.1

5     5am     86     210.8     186.4

6     6am     73     218.5     214.5

7     7am     94     190.1     163.6

8     8am     144     224.1     160.1

9     9am     282     240.6     140.6

10     10am     240     233.4     176.2

11     11am     295     193.1     174.4

12     12pm     474     166.5     159

13     1pm     357     225.3     182.6

14     2pm     270     224     190.3

15     3pm     239     181.5     155.6

16     4pm     381     212.3     136.3

17     5pm     349     186.4     140.4

18     6pm     477     160.3     147.7

19     7pm     521     180.5     146.3

20     8pm     804     184.8     142.4

21     9pm     579     203.5     142.4

22     10pm     675     205.8     121.9

23     11pm     520     235.4     137.7

24     12am (next day)     407     250.1     138.1

25     1am (next day)     475     293     107.9

26     2am (next day)     393     286.3     109.4

27     3am (next day)     165     243.2     128.4

§ ADDITIONAL INFORMATION ON DATA COLLECTION METHODS

The BLS protocol was sanctioned by the Mass General Brigham Institutional Review Board. Protocol Number: 2015P002189 Circuit dynamics underlying longitudinal fluctuations in mood and cognition in bipolar patients, Principal Investigator Name: Baker, Justin T.

§.§ BLS recruitment and demographics

Patients interested in BLS were screened for eligibility over the phone by a trained research assistant. The only inclusion criterion was current diagnosis of a severe affective or psychotic disorder, established based on the Structured Clinical Interview for Diagnosis of DSM-V Disorders. Schizophrenia, Schizoaffective Disorder, Bipolar Disorder (with or without psychosis), and Major Depressive Disorder were among the eligible diagnoses. Individuals under legal or mental incompetence or legal guardianship were excluded. In order to partake in the audio diary collection portion of the study, a prospective patient must have also owned their own smartphone capable of downloading the Beiwe app. The resulting study-wide demographics breakdown can be found in Table <ref>.

[BLS demographics.]BLS demographics. From the time of enrollment, for n = 74 enrolled study participants. Reproduced from [Liebenthal et al., 2022].

Count     Percent

Female     44     59.5%

Male     30     40.5%

African American     8     10.8%

American Indian     1     1.4%

Asian     9     12.2%

White     52     70.3%

Not Reported     4     5.4%

4 Years College     25     33.8%

Part College     31     41.9%

Professional School     11     14.9%

High School     5     6.8%

Not Reported     2     2.7%

48-51     3     4.1%

40-47     3     4.1%

30-39     11     14.9%

24-29     24     32.4%

18-23     23     31.1%

Not Reported     10     13.5%

§.§ Patient diary collection tips

When recording an audio diary, remember to:

* Keep your device level with your voice throughout the entire recording

* Limit the movement of your device throughout the entire recording (keep your device steady)

* Maintain a small distance between your mouth and the microphone on your device throughout the entire recording

* Attempt to limit background noise as much as possible throughout the entire recording

Ideally, when recording an audio diary, the participant will be seated in an area with limited background noise. Being seated allows for the participant to limit movement of the device and subsequent feedback from the device.

§.§ EMA survey design for BLS

Through May 2022, ecological momentary assessment (EMA) self-report surveys were collected from BLS participants using the described Beiwe app [Torous et al., 2016]. The primary survey was a daily questionnaire with 30 items, where each item was a radio button (i.e. select a single option) prompt with 5 response choices ranging from strong disagreement to strong agreement, though individual questions could also be skipped. The aim was for participants to submit a survey response daily, and as such payment was based on daily completion. However, participants were free to submit multiple responses to the same survey in a day if desired. Additionally, supplemental surveys were administered in select situations that could be completed for extra compensation.

Here, I provide detailed information on the EMA items used across different surveys in BLS, broken into subsections. Where applicable, I note areas where EMA design or Beiwe encoding methodology varied over time, and discuss related caveats to be aware of in processing these data. While the dataset used in the EMA modeling of chapter <ref> was carefully curated to include only some of the most consistent and clean EMA items with good availability, understanding the properties of the broader dataset is still important for future study and analysis design.

§.§.§ 30 question EMA

As mentioned, the 30 question was the original and primary EMA survey used in BLS; though there were a few changes in the design over time, the bulk of the questions remained very similar through the duration of the Beiwe study, and many BLS participants submitted answers to these items regularly for multiple years (which does bode well for EMA usability given the length of the survey). The 30 question survey always began with the following info box, before launching into the items.

Over the past 24 hours how much were you:

Each item was then a radio button question, many of which had the following standardized options.

* Very slightly or not at all

* A little

* Moderately

* Quite a bit

* Extremely

The following 15 items used the above mentioned standardized radio button options -

* Happy

* Stressed

* Hostile

* Irritable

* Alert

* Ashamed

* Inspired

* Determined

* Upset

* Afraid

* Lonely

* Outgoing

* Bothered by hearing voices

* Bothered by seeing things others could not

* Bothered by feeling like other people are out to get you or cause you trouble

There were also 17 items that had their own unique radio button options (usually with 5 options but sometimes more), as follows -

* Energetic

* Little energy or motivation to do much of anything

* Enough energy to get by but not enough to be very productive

* Typical energy level with usual productivity

* Plenty of energy to be even more productive than usual

* Unusually high energy feeling hyper or even agitated at times

* Anxious

* Completely relaxed

* Relaxed

* Typical some moments of anxiety

* High anxiety

* Extreme anxiety that dominated my thoughts

* Able to manage stress

* Very unsuccessful

* A little bit successful

* Moderately successful

* Very successful

* Extremely successful

* Social in Person

* I spent almost all of my time alone

* I interacted with others but little more than superficial interactions

* I interacted with others in a meaningful way

* I extensively interacted with close friends or family or a significant other

* I experienced an unusually deep connection with another person

* Social Digitally

* I spent almost no time interacting with others on my phone or computer

* I interacted with others but little more than superficial messages

* I interacted with others on my phone or computer in a meaningful way

* I extensively interacted with close friends or family or significant other

* I experienced an unusually deep connection through a digital interaction

* Physically Active

* Minimal movement (didn't get off the couch)

* Little more than getting around my living space

* Only did what was necessary (walking to do errands)

* Exercised 30 min or less

* Exercised 60 min or less

* Exercised more than 60 min

* Hungry

* Not hungry at all and ate much less than I normally would

* Not as hungry as usual and ate slightly less than normal

* Typical level of hunger and ate all my normal meals and foods

* More hungry than usual and ate a little more than usual (extra portion or snack)

* Surprisingly hungry and ate a lot more than usual (extra meal or binge)

* Bothered by your stomach

* Not at all

* It was a little upset

* It was mildly upset

* It was very upset

* Severe illness

* I did not notice one way or the other

* Experiencing your heart racing

* I am not generally aware of my heart rate

* I am aware of my heart rate and did not notice any changes

* I noticed my heart racing after physical exertion (stairs or exercise)

* My heart was racing but I was able to control it and it went away

* My heart was racing and I felt unable to control it

* Experiencing difficulty breathing

* I am not generally aware of my breathing

* I am aware of my breathing and did not notice any changes

* I noticed I was out of breath after physical exertion (stairs or exercise)

* I noticed I had difficulty breathing periodically throughout the day

* I felt unable to take deep breaths or catch my breath for most of the day

* Having a headache or other physical pain

* Not at all

* A little but it did not interfere with my day

* I had some mild pain that stopped me from engaging in certain activities

* The pain interfered with my ability to work or do other activities

* The pain was severe enough to not leave my house or for me to seek medical attention

* Consuming Caffeine

* None

* One cup of coffee or tea or soft drink

* Two cups of coffee or tea or soft drink

* Three cups of coffee or tea or or soft drink

* More than three cups of coffee or tea or soft drink

* Consuming Alcohol

* None

* One drink

* Two drinks

* Three to five drinks

* Five or more drinks

* Consuming cannabis or CBD

* Zero times

* Once

* Twice

* Three times

* Four or more times

* Sleeping

* Terribly: little or no sleep

* Not so well: got some sleep but not enough

* Sufficient: got enough sleep to function

* Good: got a solid night's sleep and felt well rested

* Overslept: slept more than usual and/or feel groggy

* Taking medications

* None/I don’t usually take prescribed medication

* Less than usual (forgot/skipped or reduced a dose)

* As prescribed (same as usual)

* More than usual (increased dose or took an extra dose)

* Substantially more than usual (took 2 or more extra compared to the usual amount)

* Menstruating

* No

* Uncertain (may be just beginning)

* Yes with light flow or minimal cramping

* Yes with medium flow or moderate cramping

* Yes with heavy flow or very painful cramping

* Not Applicable

Note that 32 questions are listed here – the survey began with 30 questions, but a couple more were added over time. A handful of questions also had slight wording changes over time, usually to fix typos or grammatical errors. The EMA questions focused on in the modeling of chapter <ref> (drawn from within these 30) were not altered, but any future analyses that consider more of the EMA items will need to take care to document all of the wording changes that did occur and account for any that might have scientific impact. Changes occurred at times in both question prompt and answer option wording.

More on analysis caveats.
A number of challenges in analyzing the BLS EMA responses could be prevented through more careful study setup and improvement (or alternatives) to Beiwe conventions. Question IDs issued by Beiwe do not necessarily align across subjects even when the prompted questions have identical text and options, and they also did not necessarily align across time in a subject if the question was edited. It is not clear to me whether this is a Beiwe issue or an issue in the way the lab utilized it, but nevertheless it means that those IDs are functionally useless in aligning questions. It is thus important to minimize unnecessary changes mid-study, which would be easier if initial survey design was not only more carefully reviewed for typos, but also tested on both the app interface and the resulting exported data prior to rollout.

Some of the inconsistencies in survey design lead to small discrepancies that could have had a large impact on results if not for careful data cleaning review. For example, some questions had an explicit "N/A" option at one point, independent of Beiwe's function for marking empty items (which directly specifies not presented or no response as appropriate). The problem is that this option was encoded by default in different ways at different times, and either of these ways was a potential problem – marking it as a 0 would shift the actual strongly disagree response up to a 1 encoding, and marking it as a 5 would imply it is a high agreement response by convention, if one focused on only the numbers and did not cross check the text.

More generally, it is important to think carefully about question wording and how the provided answer options will scale. Though asking about abnormally high energy is an interesting topic for a Bipolar disorder cohort, it also created an odd dynamic in the analysis as the answer options followed the trend of most other questions in being monotonic with respect to symptom severity (as it pertains to low energy in depression here), but then abruptly switched to being a different type of symptom marker with the final option. While it is important to keep the number of questions asked reasonably low, it is also important to obtain a clean dataset, and for such a relevant symptom in both directions it would perhaps have been better to split out the item.

Furthermore, wording is of course highly important to the actual responses elicited and thus the overall dataset quality. Some of the items included in the 30 question BLS EMA that I did not use had in my opinion quite shaky wording. For example, there is a big difference between using cannabis (which implies THC content) and using pure CBD, so responses to a question asking about cannabis/CBD use are not especially meaningful. The answer options to that question were similarly oddly defined, because no units that would be typically associated with a daily amount of marijuana use were supplied.

Similarly, items that centered their options around "typical" amounts are more difficult to interpret; although we are more concerned with longitudinal fluctuations in an individual rather than comparing across individuals, "typical" can change drastically over the course of a few years, and so questions focused on this may be less comparable across the entire study timeline. It is akin to the "exceeds expectations" model of performance review at many tech companies, where for some managers it becomes difficult to keep exceeding expectations, as they literally adjust their expectations each quarter. EMA questions benchmarked in this way can still have appropriate use cases, but it is an important discussion to have, particularly when other questions asking about different symptoms in the same EMA set will not be using that wording.

Ideally issues like the above could have come up in a careful initial design phase for the EMA surveys, rather than after the dataset was already collected and data cleaning/filtering needed to occur – so this is an important lesson for future studies, both in the EMA design process and in how critical it is to still perform careful data review as automated techniques remove some of the labor burden for research data collection. Automated techniques can at times introduce their own data cleanliness issues, e.g. the requirements for writing Beiwe questions that do not interfere with the automatic CSV export process for the submitted responses. The questions themselves should not contain commas, so the lab has mostly used semi-colons. However for the answer option text, it is important to not use semi-colons either, because Beiwe export denotes the split between different answer options in the CSV column using semi-colons.

Broadly, it is worth paying careful attention to details like punctuation early on in the study process, as well as performing testing from start to finish of EMA prompting and data compilation. Considering potential nuances of the intended statistical analyses and the impact of wording on patient response in parallel would likely prove fruitful too. Such planning might also involve testing of multiple app platforms for administering the EMA surveys (and other digital psychiatry data collection).

§.§.§ 7 question EMA

More recently, another daily EMA survey was designed for BLS, this one containing only 7 items – all radio button questions with the same 5 answer options. Note that 4 of the 7 items have analogous questions on the 30 count EMA.

The 7 question EMA began with the info box prompt "Over the past day; how ______ did you feel?", and then presented the following items:

* Happy

* Tense

* Relaxed

* Energetic

* Stressed

* Sad

* Fatigued

The response options for the 7 item survey were consistent across questions, and only slightly different from the generic 5 options provided for half of the 30 question survey EMA items:

* Not at all

* A little

* Moderately

* Quite a bit

* Extremely

As this briefer EMA was more recently designed, availability skews towards newer subjects, and thus has minimal overlap with the Beiwe audio journal dataset that we transcribed for BLS. For that reason, the 7 question EMA was not used in any of the analyses of chapter <ref>, despite being a cleaner dataset than the older 30 question EMA described above.

Shortening EMA.
Note that Beiwe also provides a checkbox option for question answers instead of a radio button, which means that each option can be independently ticked off or not, and not checking a given box is an implicit no. This does confound lack of response with an actual no response, and it lacks some of the resolution that the radio button questions can have. At the same time, it can compact a number of items into one question, making it easier for participants to respond to more different prompts. An example that was used briefly in the BLS dataset was the following prompt –

Consider how you have felt over the past 24 hours and check all that apply:

with the following options –

Irritable; Energetic; Hostile; Inspired; Ashamed; Afraid; Upset; Happy; Lonely; Outgoing; Alert

such that Beiwe would return a subset of this list that indicates which answers were selected. That could then be automatically converted into the same format as the related items in the previously described EMA, but instead encoding each as a binary variable.

§.§.§ Patient-specific EMA supplements

Another option enabled by Beiwe that was used at times in BLS is the administration of surveys about specific topics to target patients. One type of patient-specific supplement administered in BLS was an EMA about drug usage, which would be set up to ask about a specific substance and prompts directed to only relevant participants. This survey used a radio button format but with only yes/no answers for each question, along with a final free form response box - another EMA feature that could be better leveraged in some future studies.

More specifically, the drug survey would ask for some particular substance X:

In the past 24 hours have you used X in any form?

If yes, the following additional yes/no radio button questions would be presented:

* Did you use X for the sole purpose of avoiding withdrawal symptoms?

* Would the amount of X you used be considered excessive?

* Did you experience any side effects?

* Did you feel guilty about using X?

If no, instead an open ended question "What happened to make you choose not to use X?" would be presented with a text box, and then a few other yes/no items:

* Did you experience any cravings or felt like you needed to use X?

* Did you experience any withdrawal symptoms?

In BLS, this survey was most commonly administered to ask about tobacco use.

Note other types of participant-specific surveys with a similar format included items centered around about appetite/eating behavior, and items centered around experienced hallucinations. While this sort of EMA has the potential to be highly relevant in practice, it was not collected consistently or robustly enough during BLS to be used in larger scale analyses. It is good for future study design to be aware of the many possible EMA options however.

§.§.§ COVID EMA supplement

An advantage of Beiwe EMA administration is that it was feasible to release a new type of survey mid-study, in order to capture feelings and behaviors related to the COVID-19 pandemic. While analysis of these EMAs was beyond the scope of the present thesis, they could be particularly interesting when considered in conjunction with passive sensing data collected around the same time. The COVID surveys were weekly EMAs typically prompted on Sundays, and involved both radio button can checkbox items.

The radio button questions included the following, rated in response to "Over the past week..."

* To what extent has COVID-19 affected your work or school situation?

* To what extent has COVID-19 affected your living situation?

* To what extent has COVID-19 affected your physical health?

* To what extent have you been following recommendations to prevent spread of COVID-19?

with the following answer options available:

* Not at all

* A little

* Moderately

* Quite a bit

* Extremely

* N/A - Prefer not to answer

For each of the above questions there were also corresponding checkbox question(s) for further elaboration. Those questions were as follows, with the answer options for each contained in the associated nest listed (the participant being prompted was asked to "check all that apply"):

* Which of the following are true about your work or school situation?

* Working or taking classes from home with full load as before COVID19

* Working or taking classes from home with reduced load from before COVID19

* Working away from home

* Lost job or was furloughed

* Started a new job or educational program

* Not working - same as before COVID19

* Which of the following are true about your living situation?

* Living alone with no pets or roommates

* Living mainly on my own but with a pet or a roommate

* Living with a significant other or family member(s)

* Living with someone (e.g. child or elder) for whom I am the primary or sole caregiver

* Living with someone (e.g. child or elder) where I am responsible for some caregiving

* Living with someone who is my primary caregiver

* Which of the following are true about your relationships?

* Increased interpersonal stress with someone you live with

* Improved interpersonal relationships with someone you live with

* Increased interpersonal stress with someone you are not living with

* Improved interpersonal relationships with someone you are not living with

* Had a breakup or ended a longterm relationship

* Started a new relationship

* Which of the following are true about your physical health? Viral symptoms typically include fever; body aches; cough; fatigue; chills; digestive issues.

* I am not experiencing viral symptoms and as far as I know have not acquired COVID19

* I am not currently experiencing viral symptoms but I know or suspect I have acquired COVID19

* I am currently experiencing mild-moderate viral symptoms which I know or suspect are due to COVID19

* I am currently experiencing severe viral symptoms which I know or suspect are due to COVID19

* Someone in my home or close physical contacts has acquired COVID19

* I previously had COVID19 but am no longer experiencing symptoms

* Which of the following behaviors have you been performing to prevent the spread of COVID-19?

* Staying home as much as possible

* Avoiding indoor social gatherings with anyone who does not live with me

* Avoiding outdoor social gatherings with anyone who does not live with me

* Avoiding any places with 10 or more people

* Staying at least 6 feet away from other people who do not live with me

* Avoiding physical contact with other people who do not live with me

* Over the same period, which of the following behaviors have you been performing?

* Preventing contact with others who live with me

* Using protective gear (e.g. masks or face covering and/or gloves) when around those who live with me

* Using protective gear (e.g. masks and/or gloves) when around those who do not live with me

* Isolation in a medical facility

* Self-monitoring symptoms (e.g. regularly taking temperature)

* Washed hands with soap and water whenever I return home

§ ANALYSIS METHOD DETAILS

§.§ Distributional comparisons

I compared the distributions of the three subsets with the reported distributions of the full BLS dataset. For each of the 12 key features, I generated violin plots and kernel density estimate plots using the seaborn python package to visually compare the distribution of that feature across these 4 datasets. I also performed a two sided Kolmogorov-Smirnov (KS) test using the scipy python package, to measure the difference between the distribution of each feature in each highlighted participant with the distribution of that feature across the entire final dataset (Table <ref>). To correct the KS test results for multiple testing, I used a conservative significance cutoff of $p=0.0005$. This was based on the Bonferroni-based corrected cutoff for 3 comparisons on each of 12 features: $\frac{0.05}{12*3} \approx 0.00139$, which I then divided by 2 to account for the two sidedness of the test and finally rounded down. Table <ref> uses the $p=0.0005$ cutoff to mark the many subject-specific distributions that are significantly different than the study-wide one.

The KS test can be used to compare empirical distributions of different sizes and can capture distributional differences related to both shape and location. The returned $D$ statistic represents the largest observed distance magnitude between the cumulative distribution functions of the two empirical datasets, and therefore ranges between 0 and 1. The corresponding $p$ value represents the probability of observing a $D$ statistic at least as large as actually was, if two samples of the same sizes were drawn from the same underlying population.

§.§.§ Simulated distributions based on diary lengths

I defined the sentence-level feature distributions to be used with random sampling based on the corresponding ones reported for interviews in chapter <ref>, along with the properties of the sentiment and incoherence features observed in the diaries. I then used the real distribution of sentence counts over diaries in the present datasets (i.e. the 8398 BLS-wide diary time points, and the 3SS93, 8RC89, and 5BT65 subsets of size 981, 510, and 344 respectively) to create a dataset of fake "diaries" as described. In the simulated dataset, each record thus had a sentence count corresponding to an actual diary in the real dataset, and for each sentence in that record there was an associated sentiment score, word uncommonness score, and pairwise incoherence score, which had been obtained from random sampling of the defined distributions.

Simulated distributions of diary-level features were then obtained directly from the "diary" records by taking the mean over the randomly sampled sentiment and uncommonness scores, as well as the minimum of the sentiment scores and the maximum of the incoherence scores of each "diary". I ran KS tests (Table <ref>) and generated violin plots and KDE plots (Figure <ref>) comparing the subject-specific simulated distributions to the BLS-wide simulated distributions, just as was done for the real distributional comparisons above. Those results were subsequently utilized to inform interpretation of the corresponding results from the real diary feature distributions.

The simulation protocol was run 5 separate times, each with slightly different settings in the definitions of the underlying sentence distributions. However, the results were very similar each time, so only the final round of simulated distributions (which most closely matched the real BLS-wide distributions) are reported in depth here. The sentence pairwise incoherence and word uncommonness scores had real distribution neatly resembling a Gaussian both in the journal and interview transcript datasets (see Figure <ref>C/D), so all tested sentence-level distributions for those simulated features were randomly sampled from a Gaussian defined using numpy's random.normal function. The final settings used in the depicted simulation round were mean 1.35 with standard deviation 0.075 to represent incoherence, and mean 2.25 with standard deviation 0.325 to represent uncommonness.

To define the sentence-level distribution used to represent sentiment score in simulation, a more complicated scheme was employed. As can be seen in the journal distributions above and in the interview sentence distributions of chapter <ref> (see Figure <ref>), the sentence sentiment distribution does not closely match a convenient theoretical shape like the Gaussian. Zero sentiment sentences are very common, and of sentences with a non-zero sentiment, we found positive scores to be notably more common, and also found the magnitude of non-zero sentiment scores to be biased away from both extremes (0 and 1). Based on these observations, I used the following methodology to randomly sample sentence sentiment in the final round of simulated distributions.