In this section, we introduce the data preparation, model development, and
evaluation of the proposed semantic FOV extension method. Experiments and
analyses were conducted in Python 3.7.4, PyTorch 1.9.0, CUDA 11.3, and R
4.1.2. The pretrained pipeline is available in the form of docker container
and can be accessed by following the instructions at
https://github.com/MASILab/S-EFOV.

Lung Screening CT Dataset In this study, we included two lung cancer screening
CT datasets.

CT arm of National Lung Screening Trial (NLST). NLST Schaapveld2011 is the
largest randomized controlled trial to evaluate the effectiveness of LDCT in
lung cancer screening. 53,454 eligible participants were enrolled in the
program from August 2002 through April 2004. 26,722 were randomly assigned to
the CT arm. Longitudinal data are available, with up to three annual screens
for those who continuously enrolled and have not been diagnosed with lung
cancer during previous screens. The anthropometric measurements, including
height and weight, were self-reported at the time of enrollment right before
the first screen. In this study, we randomly sampled 1,280 subjects from the
CT arm of NLST, with 3,586 available LDCT scans in total.

Vanderbilt Lung Screening Program (VLSP). VLSP
(https://www.vumc.org/radiology/lung) is an on-going LDCT-based lung cancer
screening program conducted at Vanderbilt University Medical Center. In this
study, we used the VLSP data to develop our two-stage pipeline as more
untruncated cases are available. This included 1,490 CT scans of 887 subjects
enrolled since 2013. All data were de-identified and acquired under internal
review board supervision (IRB#181279).

The demographic and imaging protocol statistics of these two study cohorts are
summarized in Table tab:cohort. The smaller DFOV in NLST could be explained by
the strict requirement on the FOV size in the NLST imaging protocol, where
including unnecessary amount of additional body tissue beyond the lung was
even considered as a type of imaging quality issue Gierada2009,
Schaapveld2011. Although this restriction might not be strictly enforced by
later lung screening programs equipped with more advanced scanner platforms
than NLST (e.g., minimum requirement for 16-slice MDCT), optimization of the
FOV to the lung field for each patient is still recommended Kazerooni2014,
ACR2014. To address R3.C6.

Table :  Characteristics of study cohorts. SD=Standard Deviation.  Characteristics | VLSP | NLST
---|---|---
Demographic |  |
No. of subject | 887 | 1280
No. of female (%) | 399 (45.0) | 527 (41.2)
Age at baseline (y) $\pm$ SD |  64.0 $\pm$ 5.6 |  61.4 $\pm$ 4.9
BMI at baseline (kg / m2) $\pm$ SD |  28.3 $\pm$ 6.0 |  27.7 $\pm$ 4.8
Imaging |  |
No. of scans | 1490 | 3586
Effective mAs $\pm$ SD |  45.6 $\pm$ 30.3 |  36.9 $\pm$ 7.9
kVp $\pm$ SD |  119.0 $\pm$ 4.3 |  121.0 $\pm$ 4.4
Display FOV (cm) $\pm$ SD |  36.9 $\pm$ 3.6 |  33.4 $\pm$ 3.4
|  |

##  Data Preparation

###  Image Quality Review

We reviewed all selected lung screening LDCT scans and excluded cases with
severe imaging artifacts. The type of imaging artifacts included: incomplete
imaging data, imaging data corruption, beam hardening, cupping artifact,
severe imaging noise, and non-standard body positioning. For all qualified
images, the FOV mask, lung mask, and body mask were identified using the
procedures described in LABEL:app:the-masks. We further reviewed the generated
region masks and excluded those cases with significant defects. This combined
review process filtered out 5 (0.3%) scans of the VLSP cohort and 74 (2.1%)
scans of the NLST cohort in total. We identified one VLSP scan and two NLST
scans that are associated with cupping artifacts. This accounted for 0.06% in
all included LDCT scans. As evidence to support the resolution for comment
R2.C3 For all included scans, the levels of T5, T8, and T10 were estimated
using the vertebral level identification module developed in [Xu2022]. The
per-slice TCI values were calculated based on Eq (LABEL:eq:TCI).

###  Candidate Slice Identification

We used the untruncated slices in the VLSP dataset to generate the synthetic
data. As the intended application was focused on the BC assessment using the
T5, T8, and T10 axial slices, we first defined an inclusion range to cover
those slices with anatomy close to these locations. Briefly, we defined a
linear body part regression (BPR) score for each slice based on the relative
location to T5 and T10 levels, with T10 level of BPR score 0 and T5 level of
BPR score 1\. Then, all slices with BPR scores between -0.2 and 1.2 were
marked as in-range slices. Among the in-range slices, we further filtered out
those with zero TCI value, which indicated no tissue truncation in these
slices. This process filtered out 89,992 slices from 1,018 CT volumes across
669 unique subjects. Among these subjects, we random sampled 549 subjects to
form a training cohort, with 71,319 candidate slices in total. Within the
remaining subjects, we identified those with at least one T5, T8, or T10 slice
in the candidate slice set, and split them into a 60 subject validation
cohort, and a 60 subject testing cohort. For the validation and testing
cohort, we only considered the T5, T8, and T10 slices, which led to 148 slices
for validation and 145 slices for testing.

[scale=0.5]fig-typical-result-internal

Figure :  Typical results of the developed FOV extension method on synthetic
FOV truncation samples. The two ground-truth slices are selected at the fifth
and eighth thoracic vertebral body levels from two patients with dramatically
different body composition profiles. Synthetic FOV masks are applied to the
raw slices, generating slices with tissue truncation. The predicted complete
body bounding box is compared against the ground-truth bounding box. The
anatomical consistency of the generated structures is evaluated by a prior
developed BC assessment tool. The correction for BC assessment offset is
demonstrated using the bar plots that compare the measurements on truncated or
reconstructed slices against measurements on paired untruncated slices.
FOV=Field-of-view. BC=Body Composition. SAT=Subcutaneous Adipose Tissue.

[scale=0.67]fig-bland-altman

Figure :  Correction of FOV truncation caused BC assessment offset by the
proposed FOV extension method. The Bland-Altman plots compare the slice-wise
body BC assessment performed on truncated or reconstructed slices with
measurement performed on paired untruncated slices. The data are collected on
1,940 synthetic FOV truncation slices with mild to severe truncation severity
level. SD=Standard Deviation. FOV=Field-of-view. BC=Body Composition.

###  Synthetic Data Pairs

For preprocessing of the CT slices, the extraneous information outside the
identified body mask (Section  and LABEL:app:the-masks), e.g., the scan tables
and clothes, were removed by replacing the intensity of pixels with HU
intensity of air. To address R3.C8 Then, the intensity window
$\left[-150,150\right]$ (HU) was applied to highlight relevant tissues. The
resulting slices, together with pre-identified body region masks, were further
resized to $256\times 256$ before the synthetic data generation procedure. We
adopt the following configuration for the synthetic data generation procedure
introduced in Section LABEL:sec:synthetic-data. The probability to generate
three types of truncation patterns $P_{a}$, $P_{b}$, and $P_{c}$, were set to
0.5, 0.3, and 0.2, respectively, with emphasis on the first two types of
truncation patterns. $R_{RFOV}$ was uniformly sampled in range [0.6, 0.9],
which determined the size of RFOV. For the first two types of truncation
patterns, the DFOV was automatically identified once RFOV was given. For the
third type pattern generation, the $R_{DFOV}$ was uniformly sampled in range
[0.7, 1.0], which determined the size of DFOV. The offsets on two dimensions
relative to the image center, $X_{DFOV}$ and $Y_{DFOV}$, were sampled in
$\left[-D/2,D/2\right]$ , where $D=256\cdot(1-R_{DFOV})\cdot R_{RFOV}$ was the
dimension difference between the RFOV diameter and DFOV side length. This
guarantees the generated DFOV contained inside the extent of RFOV. On the CT
slice augmentation, the random scale ratio was sampled between 0.7 and 1.0.
The maximum rotation degree was set as 15°. The maximum translation in ratio
of the image dimension was 0.1 in the anterior-posterior direction and 0.2 in
the transverse direction. As a result of this random augmentation of raw CT
slice, the body region may extrude the original image border in some cases,
resulting in inaccurate body bounding box assessment. We excluded these cases
for the training of the body bounding box estimation model, while keeping them
for the image completion model development. In our implementation, the
randomized synthetic sample generation procedure was integrated into the model
training process and applied to each training CT slice when it was loaded. In
contrast, the paired data used in validation and testing are pre-generated and
specifically configured. Briefly, we generated 1,000 samples for each slice
using the same randomized synthetic data generation procedure for training
data. Cases with body regions that extruded the image boundary were excluded.
To best represent the application situation in lung cancer screening, we
excluded cases with incomplete lung regions. To balance the data regarding
different truncation severity, we set the limit of maximum five cases for each
severity levels defined in Section LABEL:sec:synthetic-data for each slice.
This resulted in 2,600 samples for validation and 2,657 samples for testing.

##  Pipeline Development

The development of the models in the proposed two-stage pipeline was based on
the same sets of training and validation data. The models were first trained
on the training set, and the best epoch was selected as the one with the best
performance on the validation set. For all models, the input slices were
normalized from $\left[-150,150\right]$ to range $\left[-1,1\right]$, with
non-FOV region imputed with value 0. FOV border extension. We implemented the
body bounding box prediction model using ResNet-18 pre-trained on ImageNet as
backbone, with the last layer replaced with a fully connected layer with four
output channels representing the pixel-space coordinates of the bounding box.
We empirically set the weight $\lambda$ to 1500 to balance the MSE term and
GIoU term in loss function (Eq LABEL:eq:combined_obj). The model was trained
with batch-size of 20, optimized using Adam optimizer with weight decay
$1\times 10^{-4}$. The learning rate was set as $2\times 10^{-3}$ . The model
was trained for 200 epochs in total. The trained model achieved the
performance of $0.976\pm 0.015$ in IoU on testing samples. With the predicted
bounding box, the extension ratio $R$ defined as in Eq (LABEL:eq:extension-
ratio) was determined to extend the image border symmetrically to cover the
estimated body region. Using the estimated $R_{est}$ alone (set $R=R_{est}$ in
Eq LABEL:eq:extension-ratio) generated extended FOV covering complete body
region only for 78.3% of cases in the test set, indicating the necessity of
the extra extension $R_{0}$. In our evaluation, a 5% extra extension
($R_{0}=1.05$) was able to consistently produce extended FOV border that
covers the complete body region, with 98.4% success rate on the test set. To
address R3.C13. Why the extra extension is necessary Image completion. For
image completion stage, we evaluated the three published methods mentioned in
Section LABEL:sec:image-completion-methods. The detailed training
configurations are given in LABEL:app:comp-image-completion.All contents
relevant to the comparison of the three image completion solutions are moved
to appendix to address R4.C2.

##  Evaluations and Results

###  Evaluation on Synthetic Paired Data

We evaluated the developed FOV extension models on pre-generated synthetic
paired data following the methods described in Section LABEL:sec:evaluation-
synthetic-data. Visual Turing test. We randomly sampled 100 synthetic samples
with TCI $>0.3$ from the withheld testing dataset (Section ). We prepared the
data following the practice of visual Turing test (Section
LABEL:sec:evaluation-synthetic-data). Two trained clinical experts
independently classified each case into fake or real category. The mean
accuracy of the two raters was 0.71, and inter-rater consensus was 0.68. On 35
out of 50 synthetic cases, at least one rater identified the case properly.
These cases were associated with slightly higher TCI value (indicating more
severe truncation) comparing to the rest of the cases (0.47 $\pm$ 0.11 vs.
0.42 $\pm$ 0.08). To address R1.C2. easiest / most difficult to distinguish by
the observers. Fig  demostrate the results of three example cases selected
from the synthetic group.Added to address comment R1.C2 and R3.C12.

[scale=0.67]fig-visual-turing-test-example

Figure :  Example cases in visual Turing test to assess the effectiveness of
the image completion stage (Section ). Two clinical experts were asked to
identify synthetic cases independently from a mixed set of equally numbered
synthetic and real cases. All three demonstrated cases are selected from the
synthetic group, where the reconstructed (synthetic) slices were provided
along with the truncation patterns to the raters instead of the real slices.
For Case 1, both raters misclassified it as real. For Case 2, one rater
classified it as real while the other rater classified it as synthetic. For
Case 3, both raters successfully identified the case as synthetic. TCI=Tissue
Truncation Index. Added this figure as part of resolution for comment R1.C2
and R3.C12.

Correction of BC measurement shift. Fig  shows the results on two samples for
qualitative evaluation. Both samples were generated using slices without
tissue truncation (zero TCI value). The BC assessment results on untruncated
slice were considered as the ground-truth measurements. The measurements on
truncated slice and FOV extended slice were compared against the ground-truth
to evaluate the effectiveness of the correction. In Fig , we use Bland-Altman
plot to evaluate the capability of the method to systematically correct the
under-estimation of area ($\text{cm}^{2}$) in the BC assessment caused by FOV
truncation. Table  shows the effectiveness of the image completion method in
restoring missing body tissues in the truncated regions, which was assessed by
pixel-wise RMSE, DSC, and BC measurements including area ($\text{cm}^{2}$) and
attenuation (HU) of SAT and muscle.To address R1.C1 and R2.C1. Added the
evaluations based on pixel-wise RMSE and BC attenuation. The metrics to
characterize the difference between the ground-truth assessment and assessment
without correction are included to provide a reference.

Table :  Evaluation of the effectiveness of the image completion method in restoring missing body tissues in the truncated regions. The experiments were conducted on synthetic truncation pairs generated by the procedure described in Section LABEL:sec:synthetic-data, by evaluating the improvement comparing reconstructed slices to truncated slices in the agreement with untruncated slices. The evaluation was based on pixel-wise RMSE and DSC, as well as the BC measurements including area ($\text{cm}^{2}$) and attenuation (HU) of SAT and muscle, using the untruncated slice and the segmentation and measurements on it as ground-truth. The evaluation was stratified by truncation severity levels as defined in Section LABEL:sec:synthetic-data. The results are displayed in mean and SD for pixel-wise RMSE and DSC, and in RMSE and 95% CI for BC measurements. Statistical significance in difference was assessed by Mann Whitney U Test for pixel-wise RMSE and DSC, and Wilcoxon Signed-rank Test on bootstrap samples for BC measurements. $P<0.001$ for all comparisons between truncated and reconstructed. RMSE=Root Mean Square Error. DSC=Dice Similarity Coefficients. BC=Body Composition. SAT=Subcutaneous Adipose Tissue. HU=Hounsfield Unit. SD=Standard Deviation. CI=Confidence Interval. Updated this table to address comments R1.C1, R2.C1, and R4.C2. Added evaluation based on pixel-wise RMSE, DSC, and tissue attenuation. Moved the comparison between the three models to the appendix section to reduce the complexity of the main text.  | Truncation Severity Level
---|---
(lr)2-6 Metric | Trace ($N=715$) | Mild ($N=715$) | Moderate ($N=713$) | Severe ($N=514$) | Overall ($N=2657$)
Pixel-wise RMSE (HU), Mean $\pm$ SD |  |  |  |  |
Truncated |  5.33 $\pm$ 3.22 |  11.54 $\pm$ 5.23 |  17.18 $\pm$ 5.73 |  24.11 $\pm$ 7.83 |  13.81 $\pm$ 8.66
Reconstructed |  2.18 $\pm$ 1.20 |  4.80 $\pm$ 2.09 |  7.55 $\pm$ 2.77 |  11.49 $\pm$ 3.52 |  6.12 $\pm$ 4.09
Dice Similarity Coefficient, Mean $\pm$ SD |  |  |  |  |
Truncated |  0.98 $\pm$ 0.02 |  0.94 $\pm$ 0.03 |  0.88 $\pm$ 0.05 |  0.79 $\pm$ 0.07 |  0.90 $\pm$ 0.08
Reconstructed |  0.99 $\pm$ 0.01 |  0.98 $\pm$ 0.02 |  0.96 $\pm$ 0.03 |  0.93 $\pm$ 0.04 |  0.97 $\pm$ 0.03
SAT Area ($\text{cm}^{2}$), RMSE (95% CI) |  |  |  |  |
Truncated | 7.88 (7.36, 8.57) | 20.48 (19.26, 22.34) | 39.09 (36.88, 42.09) | 65.41 (62.31, 68.82) | 36.95 (35.46, 38.45)
Reconstructed | 1.69 (1.45, 2.10) | 3.85 (3.37, 4.51) | 8.23 (6.91, 10.32) | 12.91 (11.59, 14.70) | 7.42 (6.79, 8.26)
SAT Attenuation (HU), RMSE (95% CI) |  |  |  |  |
Truncated | 0.57 (0.53, 0.63) | 1.37 (1.28, 1.50) | 2.71 (2.49, 3.09) | 4.33 (4.06, 4.66) | 2.49 (2.36, 2.64)
Reconstructed | 0.34 (0.31, 0.40) | 0.80 (0.72, 0.92) | 1.26 (1.15, 1.45) | 2.04 (1.86, 2.27) | 1.20 (1.13, 1.29)
Muscle Area ($\text{cm}^{2}$), RMSE (95% CI) |  |  |  |  |
Truncated | 3.59 (3.25, 3.96) | 10.81 (10.08, 11.77) | 18.35 (17.46, 19.42) | 29.39 (27.72, 31.41) | 17.08 (16.38, 17.87)
Reconstructed | 0.69 (0.61, 0.77) | 2.18 (1.86, 2.73) | 3.54 (3.14, 4.25) | 5.56 (4.89, 6.50) | 3.28 (2.99, 3.63)
Muscle Attenuation (HU), RMSE (95% CI) |  |  |  |  |
Truncated | 0.39 (0.33, 0.50) | 1.26 (1.12, 1.45) | 2.41 (2.24, 2.62) | 3.10 (2.87, 3.38) | 1.97 (1.87, 2.08)
Reconstructed | 0.20 (0.19, 0.23) | 0.61 (0.55, 0.67) | 1.00 (0.91, 1.14) | 1.48 (1.34, 1.67) | 0.89 (0.84, 0.97)
|  |  |  |  |

###  Evaluation on Real FOV Truncation Scans

In addition to the evaluation of synthetic FOV truncation samples, we
evaluated our pipeline on real lung cancer screening LDCT scans following the
methods described in Section LABEL:sec:evaluation-real. Table  shows the
statistics of the scan-wise truncation severity levels and the anthropometric
characteristics of each level in the two included the lung screening LDCT
datasets. NLST scans were associated with more severe truncation compared to
VLSP scans, which consistent with the even restricted DFOV size in NLST
compared to VLSP (Table ). Scans with more severe truncation were associated
with higher weight, BMI, and FM index in both of the datasets. As part of
resolution for R4.C1. For LDCT scans in both VLSP and NLST,To address R3.C11.
Add VLSP results. we obtained the BC assessment results using both the
original version of BC assessment pipeline developed in [Xu2022] and the
enhanced version with the FOV extension module integrated to correct the
measurement offsets caused by FOV truncation.

Table :  Anthropometric characteristics of scan truncation severity levels. The included scans are those filtered out by image quality review and with complete height and weight data. BMI=Body Mass Index. FM=Fat Mass. FFM=Fat-free Mass. To address R4.C1.  Cohort | Severity Level | No. of Scans (%) | Height (m) | Weight (kg) | BMI (kg/m2) | FM Index (kg/m2) | FFM Index (kg/m2)
---|---|---|---|---|---|---|---
6*VLSP | None | 78 (6.3%) |  1.70 $\pm$ 0.10 |  70.3 $\pm$ 15.7 |  24.1 $\pm$ 4.3 |  6.8 $\pm$ 3.2 |  17.4 $\pm$ 1.7
| Trace | 690 (55.4%) |  1.72 $\pm$ 0.10 |  79.4 $\pm$ 16.8 |  26.6 $\pm$ 4.6 |  8.4 $\pm$ 3.5 |  18.2 $\pm$ 1.7
| Mild | 355 (28.5%) |  1.72 $\pm$ 0.11 |  90.1 $\pm$ 17.9 |  30.5 $\pm$ 5.1 |  11.6 $\pm$ 4.1 |  18.9 $\pm$ 1.8
| Moderate | 107 (8.6%) |  1.69 $\pm$ 0.10 |  98.5 $\pm$ 25.2 |  34.4 $\pm$ 7.5 |  15.1 $\pm$ 5.9 |  19.2 $\pm$ 2.3
| Severe | 16 (1.3%) |  1.67 $\pm$ 0.07 |  104.2 $\pm$ 20.6 |  37.2 $\pm$ 5.9 |  18.1 $\pm$ 5.0 |  19.1 $\pm$ 1.5
(lr)2-8 | Overall | 1,246 |  1.72 $\pm$ 0.10 |  83.8 $\pm$ 19.5 |  28.3 $\pm$ 5.8 |  9.9 $\pm$ 4.6 |  18.4 $\pm$ 1.9
6*NLST | None | 18 (0.5%) |  1.70 $\pm$ 0.09 |  66.1 $\pm$ 17.9 |  22.6 $\pm$ 4.1 |  5.65 $\pm$ 2.5 |  17.0 $\pm$ 1.9
| Trace | 638 (18.2%) |  1.75 $\pm$ 0.10 |  79.4 $\pm$ 17.2 |  25.7 $\pm$ 4.3 |  7.4 $\pm$ 3.1 |  18.3 $\pm$ 1.6
| Mild | 1,190 (34.0%) |  1.75 $\pm$ 0.09 |  82.9 $\pm$ 16.4 |  27.0 $\pm$ 4.1 |  8.4 $\pm$ 3.1 |  18.6 $\pm$ 1.6
| Moderate | 1,214 (34.7%) |  1.71 $\pm$ 0.10 |  83.1 $\pm$ 18.3 |  28.4 $\pm$ 4.7 |  10.1 $\pm$ 3.5 |  18.3 $\pm$ 1.9
| Severe | 442 (12.6%) |  1.68 $\pm$ 0.10 |  89.0 $\pm$ 20.4 |  31.2 $\pm$ 5.5 |  12.7 $\pm$ 4.2 |  18.6 $\pm$ 1.9
(lr)2-8 | Overall | 3,502 |  1.73 $\pm$ 0.10 |  83.0 $\pm$ 18.0 |  27.8 $\pm$ 4.8 |  9.3 $\pm$ 3.7 |  18.4 $\pm$ 1.8
|  |  |  |  |  |  |

Expert review for application validity. We randomly sampled 100 NLST scans
with moderate to severe tissue truncation (TCI $>0.3$) and presented the FOV
extended images and final segmentation masks to two trained clinical experts
who independently reviewed and rated the quality of the results for downstream
application. None of the cases were rated as Failed (quality score $\geq 7$).
39 cases were labeled as Acceptable, with quality score between 4 and 6, by at
least one rater and with at least one clear description for identified issue.
Even with identifiable defects, these results were considered as valid for
downstream analysis by both raters. Those labeled as Acceptable were
associated with slightly higher TCI value (more severe truncation) comparing
to those considered Succeed (0.46 $\pm$ 0.10 vs. 0.43 $\pm$ 0.10). BMI
distributions were similar between those labeled as Acceptable (28.7 $\pm$
5.8) and those labeled as Succeed (28.6 $\pm$ 4.7).To address R1.C2. Add
additional characterizations for the Failed/Acceptable stratification The
identified defect include: [label=(0)] unrealistic scapula shape, 24 cases;
unrealistic intensity in subcutaneous region, 4 cases; body extrude the image
border, 2 cases; and unrealistic breast implant shape in female, 1 case.
However, the inter-rater consistency was poor, with Intraclass Correlation
Coefficient between the two raters being 0.15. By categorizing the ratings
into Failed (score $\leq 3$) and Acceptable (score $>3$), the Cohen’s Kappa
Coefficient was 0.09 (none to slight agreement) between the two raters. To
address comment R4.C4 The distribution of rating scores given by the two
raters is presented in Fig . Fig  shows the pipeline results, quality scores,
and review comments on four cases included for this quality review. To address
comment R1.C2.

[scale=0.34]fig-nlst-qa-rating-concordance

Figure :  Distribution of quality scores given by two expert raters on the FOV
extension quality of 100 random NLST scans with moderate to severe tissue
truncation. The quality score system was defined from 1 (exceptional) to 9
(poor). As none of the sampled scans was assigned with score higher than 6
(Failed), we reduce the range as from 1 to 6 in the confusion matrix plot. To
address comment R1.C2. Add a table/fig for expert review results.

[scale=1.10]fig-nlst-qualitative

Figure :  Results of BC assessment with FOV extension (image grid) and expert
reviews (blue box) on lung screening LDCT with limited FOV. The pipeline
identifies the axial slices corresponding to the T5, T8, and T10 vertebral
bodies from 3D chest CT volume (first column). The FOV of each slice is
extended with missing body tissue imputed (second column). The measurements of
BC are based on the segmentation masks predicted on the FOV extended images
(third column). The four selected cases are among the 100 NLST scans with
moderate to severe truncation (TCI $>$ 0.3) included for the expert quality
review conducted with two trained experts (Section ). All CT slices are
displayed using window [-150, 150] HU. BC=Body Composition. FOV=Field-of-view.
BMI=Body Mass Index. TCI=Tissue Truncation Index. Updated this figure to
address comment R1.C2. Expand to four cases and add the rating scores to the
image so the reader can understand what kind of results were typically
represented by the quality review categories, togather with brief case
descriptions.

Intra-subject consistency. For NLST, we identified all longitudinal pairs,
e.g., the baseline screen and second follow-up screen of the same subject,
which resulted in 3,110 pairs. As the time distance between two consecutive
screens for the same subject was approximately fixed at one year based on NLST
protocol, we put the identified pairs into two categories: 1-year-pair (2,081
pairs) and 2-year-pair (1,029 pairs). In consistency with NLST, we identified
those longitudinal pairs in VLSP with time distance between 0.5 and 1.5 years
and categorized these pairs as 1-year-pair (505 pairs), while longitudinal
pairs with time distance between 1.5 and 2.5 years were identified and
categorized as 2-year-pair (191 pairs). The longitudinal pairs were further
stratified into different truncation severity level, where the pair-wise
severity level was defined as the maximum severity level of the two paired
scans. The correlations between the measurement results on two longitudinal
scans were assessed using Spearman’s rank correlation coefficients with and
with FOV extension. Statistical significance in difference between the
correlations with and with FOV extension were assessed by the method of
[silver2004testing], which compared two dependent correlations with non-
overlapping variables. The results are summarized in Table . To address R3.C11
and R4.C1. Update experiments on intra-subject consistency. Correlation with
anthropometric approximation. We obtained the anthropometric approximations of
FFM, and FM indexes (kg/m2) based on the formulas described in Section
LABEL:sec:evaluation-real. For VLSP, the required height and weight data were
obtained before each of LDCT screen, with 1,246 scans with available
corresponding anthropometric metrics. For NLST, the anthropometric data were
obtained at enrollment right before the baseline screens. For this reason, the
effectiveness of the approximation in NLST was expected to be strongest for
the baseline screens and decreased for the follow-up screens. Thus, we
categorized the NLST scans based on screen years: 1) Screen-0 (1,232 scans);
Screen-1 (1,158 scans); and Screen-2 (1,112 scans). Screen-(0, 1, 2) represent
the baseline screen, first follow-up screen, and second follow-up screen,
respectively. We further categorized the scans by truncation severity levels.
We assessed the correlation between measured SAT index and FM index and the
correlation between muscle index and FFM index using Spearman’s rank
correlation coefficients with and with FOV extension. Statistical significance
in difference between the correlations with and with FOV extension were
assessed by the method of [hittner2003monte], which compared two dependent
correlations with overlapping variables. The results are summarized in Table .
To address R3.C11 and R4.C1. Update the anthropometric correlation tests

Table :  The comparisons of intra-subject consistency between BC measurements on longitudinal scan pairs with and without FOV extension. The scan-wise BC measurements were defined as the summation of BC areas measured on the cross-sectional slices at T5, T8, and T10. For NLST, the intra-subject pairs were formed by the possible combinations of two screens for each subject, categorized into 1-year-pair and 2-year-pair based on the years of trial the two scans were conducted. For VLSP, pairs with time distance between 0.5 and 1.5 years were categorized as 1-year-pair, while pairs with time distance between 1.5 and 2.5 years were categorized as 2-year-pair in consistency with NLST. The pairs are further categorized into different truncation severity levels defined as the maximum of scan-wise truncation severity levels of the two paired scans. Spearman’s rank correlation coefficients with 95% confidence interval are reported. Results on groups with too few cases (e.g., less than six pairs) are not included. As FOV extension is not needed for pairs without truncation, only results on raw images are displayed when it is possible. Statistical significance are evaluated by method of silver2004testing for comparison between two non-overlapping dependent correlations. The tests are two-sided, with p-values reported. BC=Body Composition. FOV=Field-of-view. SAT=Subcutaneous Adipose Tissue. To address R3.C11 and R4.C1. 1) Add results on VLSP; 2) provide ”reference” measurements.  |  | SAT area (cm2) | Muscle area (cm2)
---|---|---|---
(lr)3-5(lr)6-8 Cohort | Group (no. of pairs) | Without Correction | FOV Extended | p-value | Without Correction | FOV Extended | p-value
18*VLSP | No Truncation |  |  |  |  |  |
|  1-year-pair (N=13) | 0.967 (0.845, 1.000) | - | - | 0.918 (0.659, 0.994) | - | -
|  2-year-pair (N=6) | 0.886 (0.200, 1.000) | - | - | 0.771 (0.000, 1.000) | - | -
| Trace Truncation |  |  |  |  |  |
|  1-year-pair (N=233) | 0.946 (0.923, 0.961) | 0.941 (0.915, 0.958) |  $<$.001 | 0.940 (0.914, 0.958) | 0.946 (0.923, 0.961) |  $<$.001
|  2-year-pair (N=79) | 0.894 (0.809, 0.947) | 0.893 (0.803, 0.951) | .89 | 0.955 (0.921, 0.971) | 0.943 (0.895, 0.965) | .69
| Mild Truncation |  |  |  |  |  |
|  1-year-pair (N=181) | 0.929 (0.896, 0.949) | 0.933 (0.899, 0.954) | .64 | 0.936 (0.908, 0.956) | 0.946 (0.922, 0.963) | .001
|  2-year-pair (N=72) | 0.886 (0.797, 0.941) | 0.890 (0.805, 0.944) | .69 | 0.911 (0.814, 0.962) | 0.925 (0.860, 0.961) | .49
| Moderate Truncation |  |  |  |  |  |
|  1-year-pair (N=69) | 0.810 (0.701, 0.881) | 0.908 (0.840, 0.947) |  $<$.001 | 0.865 (0.751, 0.931) | 0.944 (0.901, 0.964) |  $<$.001
|  2-year-pair (N=31) | 0.762 (0.502, 0.898) | 0.906 (0.749, 0.967) |  $<$.001 | 0.849 (0.663, 0.945) | 0.904 (0.806, 0.944) | .50
| Severe Truncation |  |  |  |  |  |
|  1-year-pair (N=9) | 0.400 (-0.381, 0.890) | 0.617 (-0.242, 1.000) | .02 | 0.850 (0.352, 1.000) | 0.983 (0.817, 1.000) | .16
|  2-year-pair (N=3) | - | - | - | - | - | -
(lr)2-8 | Overall |  |  |  |  |  |
|  1-year-pair (N=505) | 0.950 (0.937, 0.959) | 0.961 (0.949, 0.969) |  $<$.001 | 0.930 (0.910, 0.946) | 0.952 (0.940, 0.961) |  $<$.001
|  2-year-pair (N=191) | 0.927 (0.895, 0.947) | 0.941 (0.915, 0.959) |  $<$.001 | 0.927 (0.889, 0.953) | 0.938 (0.912, 0.957) | .42
18*NLST | No Truncation |  |  |  |  |  |
|  1-year-pair (N=4) | - | - | - | - | - | -
|  2-year-pair (N=1) | - | - | - | - | - | -
| Trace Truncation |  |  |  |  |  |
|  1-year-pair (N=207) | 0.952 (0.932, 0.965) | 0.946 (0.923, 0.960) |  $<$.001 | 0.934 (0.897, 0.959) | 0.927 (0.880, 0.955) | .028
|  2-year-pair (N=105) | 0.955 (0.920, 0.972) | 0.951 (0.914, 0.970) |  $<$.001 | 0.934 (0.886, 0.962) | 0.937 (0.898, 0.958) | .84
| Mild Truncation |  |  |  |  |  |
|  1-year-pair (N=645) | 0.936 (0.922, 0.949) | 0.936 (0.921, 0.948) | .96 | 0.942 (0.930, 0.952) | 0.936 (0.922, 0.947) | .21
|  2-year-pair (N=329) | 0.885 (0.849, 0.911) | 0.886 (0.852, 0.915) | .50 | 0.912 (0.885, 0.933) | 0.909 (0.880, 0.932) | .76
| Moderate Truncation |  |  |  |  |  |
|  1-year-pair (N=853) | 0.901 (0.877, 0.918) | 0.926 (0.903, 0.942) |  $<$.001 | 0.943 (0.935, 0.950) | 0.952 (0.943, 0.959) |  $<$.001
|  2-year-pair (N=388) | 0.875 (0.836, 0.904) | 0.898 (0.859, 0.927) | .026 | 0.941 (0.928, 0.950) | 0.953 (0.942, 0.960) |  $<$.001
| Severe Truncation |  |  |  |  |  |
|  1-year-pair (N=372) | 0.797 (0.751, 0.837) | 0.916 (0.892, 0.934) |  $<$.001 | 0.861 (0.818, 0.891) | 0.919 (0.893, 0.939) |  $<$.001
|  2-year-pair (N=206) | 0.782 (0.707, 0.841) | 0.894 (0.854, 0.921) |  $<$.001 | 0.833 (0.775, 0.875) | 0.917 (0.877, 0.942) |  $<$.001
(lr)2-8 | Overall |  |  |  |  |  |
|  1-year-pair (N=2,081) | 0.912 (0.901, 0.922) | 0.950 (0.942, 0.957) |  $<$.001 | 0.944 (0.938, 0.949) | 0.951 (0.946, 0.956) |  $<$.001
|  2-year-pair (N=1,029) | 0.880 (0.859, 0.898) | 0.925 (0.909, 0.938) |  $<$.001 | 0.933 (0.924, 0.940) | 0.948 (0.940, 0.955) |  $<$.001
|  |  |  |  |  |  |
Table :  The comparisons of correlation between the height-square normalized BC area indexes and anthropometric approximated BC indexes with and without FOV extension. SAT index (cm2/m2) was compared against FM index (kg/m2), and Muscle index (cm2/m2) was compared against FFM index (kg/m2). For VLSP, the anthropometric approximations were derived using weight and height obtained before each of the scan. For NLST, the approximations were derived from the height and weight information obtained at the baseline. Thus, we further stratified the NLST scans into years of the screen in addition to the truncation severity level. Spearman’s rank correlation coefficients with 95% confidence interval are reported. Statistical significance are evaluated by the method of hittner2003monte for comparison between two overlapping dependent correlations. The tests are two-sided, with p-values reported. BC=Body Composition. FOV=Field-of-view. SAT=Subcutaneous Adipose Tissue. FM=Fat Mass. FFM=Fat-free Mass. To address R3.C11 and R4.C1. 1) Add VLSP results; 2) provide the reference measurements  |  | SAT index (cm2/m2) vs. FM index (kg/m2) | Muscle index (cm2/m2) vs. FFM index (kg/m2)
---|---|---|---
(lr)3-5(lr)6-8 Cohort | Severity Level (no. of scans) | Without Correction | FOV Extended | p-value | Without Correction | FOV Extended | p-value
5*VLSP | None (N=78) | 0.780 (0.657, 0.862) | - | - | 0.692 (0.545, 0.801) | - | -
| Trace (N=690) | 0.798 (0.765, 0.826) | 0.801 (0.769, 0.829) | .27 | 0.658 (0.612, 0.703) | 0.681 (0.638, 0.721) |  $<$.001
| Mild (N=355) | 0.782 (0.734, 0.822) | 0.789 (0.743, 0.828) | .02 | 0.605 (0.532, 0.671) | 0.659 (0.596, 0.717) |  $<$.001
| Moderate to Severe (N=123) | 0.653 (0.535, 0.750) | 0.673 (0.556, 0.767) | .36 | 0.362 (0.172, 0.528) | 0.579 (0.415, 0.709) |  $<$.001
(lr)2-8 | Overall (N=1,246) | 0.837 (0.815, 0.855) | 0.845 (0.823, 0.862) |  $<$.001 | 0.606 (0.567, 0.646) | 0.668 (0.634, 0.699) |  $<$.001
20*NLST | None |  |  |  |  |  |
|  Screen-0 (N=9) | 0.762 (-0.078, 1.000) | - | - | 0.905 (0.315, 1.000) | - | -
|  Screen-1 (N=5) | - | - | - | - | - | -
|  Screen-2 (N=4) | - | - | - | - | - | -
| Trace |  |  |  |  |  |
|  Screen-0 (N=226) | 0.737 (0.653, 0.806) | 0.735 (0.649, 0.805) | .08 | 0.698 (0.611, 0.768) | 0.713 (0.632, 0.778) | .004
|  Screen-1 (N=211) | 0.712 (0.611, 0.792) | 0.717 (0.618, 0.795) | .15 | 0.708 (0.613, 0.785) | 0.725 (0.635, 0.796) | .01
|  Screen-2 (N=201) | 0.702 (0.616, 0.776) | 0.700 (0.615, 0.773) | .41 | 0.704 (0.612, 0.780) | 0.724 (0.633, 0.794) | .19
| Mild |  |  |  |  |  |
|  Screen-0 (N=421) | 0.760 (0.707, 0.812) | 0.762 (0.708, 0.811) | .48 | 0.705 (0.645, 0.754) | 0.714 (0.655, 0.766) | .65
|  Screen-1 (N=379) | 0.727 (0.663, 0.778) | 0.730 (0.670, 0.781) | .84 | 0.716 (0.656, 0.766) | 0.721 (0.661, 0.772) | .49
|  Screen-2 (N=390) | 0.711 (0.648, 0.764) | 0.710 (0.647, 0.767) | .34 | 0.707 (0.645, 0.761) | 0.698 (0.634, 0.754) | .98
| Moderate to Severe |  |  |  |  |  |
|  Screen-0 (N=577) | 0.762 (0.723, 0.795) | 0.818 (0.785, 0.845) |  $<$.001 | 0.709 (0.662, 0.748) | 0.746 (0.705, 0.779) |  $<$.001
|  Screen-1 (N=562) | 0.711 (0.665, 0.750) | 0.758 (0.718, 0.791) |  $<$.001 | 0.705 (0.656, 0.748) | 0.741 (0.696, 0.780) |  $<$.001
|  Screen-2 (N=517) | 0.718 (0.668, 0.764) | 0.745 (0.696, 0.786) |  $<$.001 | 0.668 (0.617, 0.715) | 0.712 (0.669, 0.752) |  $<$.001
(lr)2-8 | Overall |  |  |  |  |  |
|  Screen-0 (N=1,232) | 0.778 (0.750, 0.803) | 0.818 (0.795, 0.839) |  $<$.001 | 0.714 (0.687, 0.741) | 0.752 (0.727, 0.776) |  $<$.001
|  Screen-1 (N=1,158) | 0.759 (0.729, 0.786) | 0.802 (0.777, 0.825) |  $<$.001 | 0.699 (0.667, 0.729) | 0.745 (0.718, 0.770) |  $<$.001
|  Screen-2 (N=1,112) | 0.744 (0.712, 0.773) | 0.782 (0.753, 0.807) |  $<$.001 | 0.694 (0.659, 0.726) | 0.731 (0.700, 0.757) |  $<$.001
|  |  |  |  |  |  |


# Packet Compressed Sensing Imaging (PCSI): Robust Image Transmission over
Noisy Channels 00footnotetext: Presented at ARRL/TAPR DCC 2020

Scott Howard, Grant Barthelmes, Cara Ravasio,
Lisa Huang, Benjamin Poag, & Varun Mannam

(Department of Electrical Engineering, University of Notre Dame)

###### Abstract

Packet Compressed Sensing Imaging (PCSI) is digital unconnected image
transmission method resilient to packet loss. The goal is to develop a robust
image transmission method that is computationally trivial to transmit (e.g.,
compatible with low-power 8-bit microcontrollers) and well suited for weak
signal environments where packets are likely to be lost. In other image
transmission techniques, noise and packet loss leads to parts of the image
being distorted or missing. In PCSI, every packet contains random pixel
information from the entire image, and each additional packet received (in any
order) simply enhances image quality. Satisfactory SSTV resolution (320x240
pixel) images can be received in $\approx$1-2 minutes when transmitted at 1200
baud AFSK, which is on par with analog SSTV transmission time. Image
transmission and reception can occur simultaneously on a computer, and
multiple images can be received from multiple stations simultaneously -
allowing for the creation of “image nets.” This paper presents a simple
computer application for Windows, Mac, and Linux that implements PCSI
transmission and reception on any KISS compatible hardware or software modem
on any band and digital mode.

###### Contents

  1. 1 Introduction
    1. 1.1 Radio Image Transmission Modes Comparison
    2. 1.2 PCSI’s Predecessors
  2. 2 The Magic in the Math
    1. 2.1 Compressed Sensing Imaging
    2. 2.2 Chroma Compression and Color Depth
  3. 3 Implementation: The PDP Specification
    1. 3.1 What is the PDP?
    2. 3.2 PDP Specification Scope
      1. 3.2.1 AX.25 Framing
      2. 3.2.2 SSDV-style Framing
      3. 3.2.3 Framing Comparison
    3. 3.3 PDP Specification Details
      1. 3.3.1 Packet Payload Preparation
      2. 3.3.2 Pseudo-random Number Generation for Picking Pixels
      3. 3.3.3 PCSI Payload base91 Encoding
    4. 3.4 Reconstructing PCSI Images
  4. 4 Future Work

##  1 Introduction

Packet compressed sensing imaging (PCSI) is a solution to the technical
challenge of transmitting a complete image to multiple receivers over a
channel where each receiving station may miss different parts of the
transmission due to channel noise. PCSI is implemented in a way such that a
low-power microcontroller (e.g., Arduino) is capable of transmitting the image
from challenging and remote environments (e.g., high altitude balloon).

PCSI achieves these capabilities by using a technique known as ‘compressed
sensing imaging,” a computational method that allows one to reconstruct a
complete image when only given a random selection of pixels from that image.
Therefore, even after receiving only a single packet of random pixels, the
receiver can begin to reconstruct the complete original image. Each additional
packet received further increasing image quality. PCSI is, therefore, robust
against packet loss.

The initial version of an open-source software tool [1] that implements
sending and receiving PCSI in Python for Windows, macOS, and Linux is
illustrated in Figure 1. The image experienced $\approx 50\%$ packet loss, yet
the image was fully reconstructed (bottom frame). In other techniques, 1/2 of
the image would be completely missing.

Figure 1: Demonstration of PCSI using pcsiGUI, an open-source tool written in
Python for Windows, macOS, and Linux computers. Only 53.6% of the packets were
received (indicated by the red pixels in the top frame), yet the entire image
can be reconstructed (bottom image). The software tool connects to any KISS
compatible TNC or software modem and is capable of simultaneous (duplex)
receiving/transmitting, and capable of receiving multiple images from multiple
stations simultaneously.

###  1.1 Radio Image Transmission Modes Comparison

When transmitting an image, the operator is presented with many possible
methods and must select an image transfer method that best suits the
application. The following is a quick summary of popular techniques.

Analog transmission can either encode an image’s color values using frequency
modulation (as in SSTV) or amplitude modulation (as in radiofax). In all
cases, each point is transmitted sequentially; therefore, channel noise and
signal loss leads to color distortion and pixel loss.

Digital techniques divide an image into individual pixels and transmit the
quantized (i.e., represented in binary) values of each pixel’s color channels
as groups of data in packets. Each packet contains additional information that
allows the receiver to check if the received packet was distorted during
transmission (e.g., a “checksum” as used in AX.25), and more recently, can
correct for some errors using forward error correcting (FEC) codes (as used in
FX.25). In both cases, unrecoverable errors lead to complete loss of pixel
information for parts of the image.

Digital techniques also allow for the use of compression to send image data
efficiently in fewer bits. A great example of this is slow scan digital video
(SSDV) [2]. SSDV was developed by Philip Heron with the UK’s high altitude
ballooning community to transmit high-quality images. The technique takes
images, converts them to a specific type of JPEG file, and transmits sections
of the JPEG file in packets using FEC. This method transmits high quality
images successfully; however, JPEG encoding and FEC generation is
prohibitively complex for low-memory microcontrollers, and signal loss leads
to missing parts of the image.

Hence, there is a need to develop a computationally simple image transfer
method that is robust to signal loss and channel noise.

###  1.2 PCSI’s Predecessors

In developing PCSI, we found that the idea of transmitting limited information
over a noise channel while ‘filling in the blanks” at the receiving end were
present in at least two other technologies.

APRS Vision System: At the 1997 DCC, Bob Bruninga (WB4APR) proposed the “APRS
Vision System” [3]. That approach was an attempt to relay imaging information
over the APRS network. The idea was to transmit APRS packets that contain
increasingly higher amounts of spatial information content. The first packet
contained an extremely low quality image, and each subsequent packet doubled
the resolution. When the receiver feels the image is “clear enough” (i.e., the
receiver can ”fill in the gaps” in the image), the receiver can tell the
sender to stop. This way the receiver can possibly control or react to the
image before the whole image became “clear.” However, this system required
that every previous packet had been received perfectly for subsequent packets
to be used, and the receiver needed to communicate with transmitter to
indicate missed packets and to stop. Any missed packet caused the whole system
to fail from that point forward.

Hellschreiber: This mode, developed in the mid 20th century, is a fascinating
approach that enabled robust transmission of text over noisy channels [4].
Each character of text to be transmitted is converted in to a 7x7 pixel image
of that character. Each character is then transmitted using extremely narrow
bandwidth transmission (e.g., on-off CW keying). The receiver takes the
received data and reconstructs the image. The operator then “reads” the
resulting image text. Fundamentally, the operator’s trained character
recognition neural network (i.e., their brain) does pattern recognition to
“fill the the gaps” in the image that was caused by channel noise and signal
loss. The entire message can therefore be received even if some pixels were
not received or if noise distorted some pixels.

##  2 The Magic in the Math

While it may seem “too good to be true” to be able to fully reconstruct an
entire image using only a few randomly selected pixel values, PCSI is made
possible via the magic in the math. PCSI accomplishes this using two
techniques: compressed sensing imaging and chroma compression.

###  2.1 Compressed Sensing Imaging

The mathematical concept of compressed sensing imaging exploits the fact that
data can often be represented in “sparse” domains. By “sparse,” we mean that
“most of the values are zero.” For example, compare a photograph of a blooming
garden during the day to a photograph of the night sky. The image of the
garden will have lots of red, green, and blue color, and therefore, many non-
zero values for the R, G, and B channels in the image. The night sky is mostly
empty, with a few stars, planets, and the moon. The nigh sky image, therefore,
has many pixels that have zero value in the R, G, or B channels. We would say
the garden image is not sparse while the night sky is sparse.

The magic happens when you convert the image from the spatial domain (i.e., a
photograph) to another “domain.” For example, you can perform a mathematical
operation on an image called the discrete cosine transform (DCT). The
resulting DCT “image” ends up containing all the same information from the
original image, just arranged in a different way. You can then do an inverse
discrete cosine transform (IDCT) on the “DCT image,” and you will restore the
original image accurately. Why would we want to take the DCT of an image? We
do this because practically all images are “sparse” after taking the
DCT.111Technically, the image is sparse in the DCT domain because cameras
typically massively “oversampling” an image. In other words, most of the
values of the DCT of an image are zero! Both the garden and night sky look
sparse after taking the DCT!

How can we use the fact that most images are sparse after taking the DCT? This
is the magic of PCSI. You ask the computer to find the DCT of an image such
that:

  1. 1.

The values for the DCT it finds has a lot of zeros in it (i.e., is sparse).

  2. 2.

When you reconstruct the image using the values for the DCT the computer
found, the resulting real image closely matches whatever values of the pixels
that were received

Mathematically, you are asking the computer to do something similar to “basis
pursuit denoising” to find the simplest image (i.e., the fewest non-zero DCT
values) that also matches the pixels that you have received so far. You can do
that by using a computer program to find the values of the DCT of the final
image ($\mathbf{X}$) such that it gives you the smallest value of the
following expression:

$\sum_{n}{|\text{IDCT}(\mathbf{X})_{n}-b_{n}|^{2}}+C\sum{|\mathbf{X}|}$ (1)

where $b_{n}$ is the value of the $n$th pixel that was received, and $C$ is a
scaling factor (typically in the range of 3-5). The first term is the sum of
the squared error between the values of the pixels in the reconstructed image
and the value of the pixels that were actually received. Ideally, this will be
zero. The second term is the L1 norm, which is adding up all the values of
$\mathbf{X}$, and minimizing that term is a good method for finding a sparse
$\mathbf{X}$. Once you find $\mathbf{X}$, you can take the IDCT of
$\mathbf{X}$ to find the reconstructed image.

While the current PCSI reference implementation does use the above basis
pursuit technique, it is just one of many ways to reconstruct an image from a
collection of random pixels. PCSI actually does not require any particular
reconstruction algorithm as there may be variations in the method that yield
superior results.

###  2.2 Chroma Compression and Color Depth

PCSI image transmission speed is increased by reducing the bit depth of an
image and by utilizing chroma compression. These techniques are described
below.

  * •

Bit Depth: Reducing an image from 24 bit color (8-bit in each of R, B, G) to
12 bit color (or any other color depth) is trivial, may be acceptable for many
applications, and therefore is an option in PCSI.

  * •

Chroma compression: The human eye has 20-times high density of rods (greyscale
photoreceptors) than cones (color photoreceptors), and therefore detects
greyscale with better spatial resolution than color. It is therefore not
necessary to transmit the same resolution for both luma (brightness) and
chroma (color information). JPEG exploits this by representing an image in the
YCbCr color space and sub-sampling the chroma channels (Cb and Cr) relative to
the luma channel (Y). PCSI uses the same general concept and achieves chroma
compression by sending a combination of full-color (YCbCr) and greyscale-only
(Y) pixels in each packet. This step leads to the receiver receiving more Y
channel pixels than Cb and Cr channel pixels. Each channel, separately,
undergoes the compressed sensing basis pursuit to reconstruct the original
channel, and the channels are then converted back to RGB. The resulting image
appears to have much higher quality for the same number of packets.

##  3 Implementation: The PDP Specification

PCSI requires packet payloads such that each individual payload contains all
the information necessary to reconstruct a single image. To achieve that, the
pseudo-random datagram payload (PDP) specification has been developed. Version
1.0.0 is described below.

###  3.1 What is the PDP?

PDP is a specification for the payloads of data packets such that each packet
contains all the information needed to reconstruct a single image. An image is
then transmitted as the collection of datagrams (i.e., packets in a
connection-less network). Unlike other packetized image transmission formats,
the pixels contained in a packet are selected in a pseudo-random, yet
deterministic, way. This allows images to be restored using compressed sensing
techniques regardless of packet loss.

###  3.2 PDP Specification Scope

The PDP spec merely defines the packet payload for the transmission of a
single image. It can be used in any packet protocol or digital mode. Framing
is independent of the specification. This allows for the separation of a
“session” (consisting of a sending station sending one or more images) from
the minimal content required for a single image. The “session” information is
in the framing; the image information is in the payload. The payload is
designed to be similar to SSDV.

For example, a PDP can be placed as the payload in:

  * •

AX.25 amateur radio packets. Transmitted using any mode (e.g., AFSK, PSK,
etc.) Therefore it is compatible with APRS, TNCs, digipeaters, software modems
(direwolf, fldigi, soundmodem, etc.). Example implementation is in Section
3.2.1.

  * •

SSDV-style framing done in a KISS TNC compatible way. Example implementation
is in Section 3.2.2.

  * •

UDP or TCP (although the benefits of PCSI provide more benefit to multicast
UDP packets than connected TCP packets).

####  3.2.1 AX.25 Framing

While not part of the PDP spec, an example of using AX.25 UI framing [5] of a
PDP is given in Table 1. This is easily compatible with existing TNCs. This
framing adds at least 20 bytes of overhead.

Name | Size (bits) |  Description
---|---|---
Flag | 8 |  HDLC flag ‘$\mathtt{0x7E}$’
Dest. Address | 56 |  Callsign of intended receiver OR alias of an image net, encoded following AX.25 spec. ‘PCSI‘ recommended for general use.
Source Address | 56 |  Sender’s callsign encoded following AX.25 spec. ‘PCSI’ recommended for general use.
Digi Addresses | $d\times 56$ |  $d$ optional digipeater addresses, encoded following AX.25 spec.
Control | 16 |  ‘$\mathtt{0x03F0}$’ indicating UI frame with no response requested, and no layer 3 implemented
PDP | $N\times 8$ |  PDP data, $N\leq 256$
FCS | 16 |  CRC-CCITT
Flag | 8 |  HDLC flag ‘$\mathtt{0x7E}$’
Table 1: Example AX.25 framing that could be used for PCSI.

####  3.2.2 SSDV-style Framing

While not part of the PDP spec, a simple session framing of PDP can be done in
a way that is compatible with existing KISS hardware and software TNCs. An
example is seen in Table 2. This example framing is designed to be easy to use
with any KISS TNC. One would simply send the concatenated “Packet Identifier +
Callsign + PDP” and let the TNC add the flags and do the checksum. This
framing adds at least 9 bytes of overhead.

Offset (bytes) | Name | Size (bytes) |  Description
---|---|---|---
0 | Flag | 1 |  HDLC flag ‘$\mathtt{0x7E}$’
1 | Packet Identifier | 1 |  ASCII ‘v’ = ‘$\mathtt{0x76}$’
2 | Callsign | 4 |  Base-40 encoded callsign following SSDV encoding convention
6 | PDP | $N\leq 256$ |  PDP data
N+6 | Checksum | 2 |  CRC-CCITT
N+8 | Flag | 1 |  HDLC flag ‘$\mathtt{0x7E}$’
Table 2: Example SSTV-style framing that could be used for PCSI.

####  3.2.3 Framing Comparison

Both AX.25 framing and SSDV-style framing can be used. AX.25 is more powerful
as it can leverage existing packet radio infrastructure at the cost of of
larger overhead. However, if channel bit error rate (BER) is high (as is
common in longer-distance HF modes), smaller packets are more likely to be
successfully received. The lower overhead SSDV-style framing may be superior
in this case. This trade-off is explored in Figure 2. The net efficiency
(percent of each transmitted bit that will successfully transmit pixel-level
image information to the receiver) is calculated as the product of the
probability that the entire packet will be received properly and the
percentage of bits in a packet that correspond to pixel information.

Net Efficiency AX.25
$\displaystyle=\frac{x-7}{x+20}\times(1-\text{BER})^{8x+160}$ (2) Net
Efficiency SSDV-Style
$\displaystyle=\frac{x-7}{x+9}\times(1-\text{BER})^{8x+72}$ (3)

where $x$ is the total PDP length in bytes and the term $x-7$ comes from the
fact that the PDP has a 7 byte header as described in 3.3.

$10^{1}$$10^{2}$$0$$0.2$$0.4$$0.6$$0.8$$1$Net Efficiency
(%)$\text{BER}=10^{-5}$$10^{1}$$10^{2}$$0$$0.2$$0.4$$0.6$$0.8$$1$$\text{BER}=10^{-4}$$10^{1}$$10^{2}$$0$$0.2$$0.4$$0.6$$0.8$$1$PDP
Size (bytes)Net Efficiency
(%)$\text{BER}=10^{-3}$$10^{1}$$10^{2}$$0$$0.2$$0.4$$0.6$$0.8$$1$PDP Size
(bytes)$\text{BER}=10^{-2}$ Figure 2: Comparison of net bit efficiency versus
PDP payload size for both SSDV-style and AX.25 framing.

Results from Figure 2 give guidelines for ideal PDP length. First, find the
approximate BER by estimating packet loss for an AX.25 packet with a 256 byte
payload using the equation:

$\displaystyle\text{BER}=1-(1-L/100)^{1/2192}$ (5)

where $L$ is packet loss in percent. If the packet loss percentage is known
(or can be estimated), BER can be found using Equation 5, which is depicted by
Figure 3.

Now based on the estimated BER, you can choose the appropriate framing style
and PDP size (refering to Fig. 2):

  * •

For low $\text{BER}\leq 10^{-5}$ Environments: Framing style does not matter
that much, and payloads should be the full 256 bytes long.

  * •

For $\text{BER}\approx 10^{-4}$ Environments: Framing style does not matter
that much, and payloads should be 130 bytes long.

  * •

For $\text{BER}\approx 10^{-3}$ Environments: SSDV-Style framing increases
efficiency (and speed) by $\approx 25\%$ compared to AX.25. Payloads should be
40-50 bytes long.

  * •

For $\text{BER}\approx 10^{-2}$ Environments: AX.25 is practically unusable;
SSDV framing will barely be usable. Payloads should be 10-11 bytes long.

$0$$10$$20$$30$$40$$50$$60$$70$$80$$90$$100$$10^{-5}$$10^{-4}$$10^{-3}$Packets
Loss (%)Bit Error Rate (BER) Figure 3: BER vs packet loss for AX.25 frames
with 256 byte payloads.

###  3.3 PDP Specification Details

Since each packet contains information of the whole frame, each packet MUST be
the same size of every packet in an image (same number of pixels per packet).
Total packet size is determined by the framing protocol used. For example,
AX.25 packet payloads are 256 bytes by default. The payload contains the
following data transmitted in order as described by Table 3.

Offset (bits) |  Name |  Size (bits) |  Description
---|---|---|---
0 |  Image ID |  8 |  Identifies unique images within a PCSI session. (uint8)
8 |  Rows |  8 |  Number of lines in the image divided by 16. (4096 lines max, uint8)
16 |  Columns |  8 |  number of columns in the image divided by 16. (4096 columns max, uint8)
24 |  Packet ID |  16 |  used as the starting point of the pseudorandom pixel list. (uint16)
32 |  Number of YCbCr Pixels |  8 |  Number of full color pixels transmitted in this packet. (uint8)
40 |  Color depth |  8 |  Color depth encoded as (color depth/3 -1). e.g., 24bit color = 7. This only uses 3 bits, so there are 5 bits available for future use. (uint8)
48 |  YCbCr Pixel Data |  (Number of YCbCr Pixels) * (Color bit depth) |  Full color (YCbCr) pixels listed first as a binary stream. For example, if color is transmitted as 12-bit color, each pixel is 12-bits long with the first 4 corresponding to the Y channel, the next four corresponding to the Cb channel, and the final 4 corresponding to the Cr channel.
48 + (YCbCr Pixels) * (Color bit depth) |  Y-only Pixel Data |  N |  Black and white (Y only) pixels follow in a binary stream of Y values encoded as a uint with the same bit depth as a single channel of the YCbCr pixels.
|  Zero padding |  Z |  Zero padding for byte alignment as needed.
Table 3: PDP Specification Version 1.0.0

####  3.3.1 Packet Payload Preparation

Given a bit mapped image to transfer, follow the following procedures

  1. 1.

Using a pseudo-random number generator (see Section 3.3.2), generate the
sequence of pixels to be transmitted.

  2. 2.

Given the number of bits available in the payload (e.g., AX.25 UI frames have
256 bytes minus 7 bytes of image info equals 1992 bits total), the desired
chroma compression level, and the desired color bit depth to transmit,
determine the list of pixels to transmit that will be full color and solely
back and white.

    1. (a)

All packets consist of the same number of pixels (e.g., every packet for an
image has exactly 25 YCbCr pixels and 75 Y only pixels for a total of 100
pixels. You can choose whatever numbers you want, as long as they are the same
for every packet of the image).

  3. 3.

Prepare the packet payload

    1. (a)

Convert full color pixels to YCbCr per ITU-T T.871 [6] and black and white
only pixels to Y as per the same spec.

    2. (b)

If color bit depth is being reduced, approximate the value to be transmitted
using rounding. For example, the 8 bit number 200 will be represented as the 4
bit number round(200/255*15)=12.

####  3.3.2 Pseudo-random Number Generation for Picking Pixels

Compressed sensing imaging requires that the measurements are uncorrelated in
the sparse domain that is used to reconstruct the image. Taking random samples
ensures this condition, however, both the transmitter and receiver need to
know which pixel values correspond to which pixels in the image. To do this,
PCSI uses a Linear Congruential Generator [7] as a deterministic pseudo-random
number generator using GCC’s default coefficients ($\text{modulus}=2^{31}$,
$a=1103515245$, $c=12345$, starting with a $\text{seed}=1$). The pseudo-random
number generator is then used with the modern Fisher Yates shuffle algorithm
[8] to generate a random list of the pixels to be sent. See reference code for
details. This approach will allow all receivers and the transmitter to
generate identical lists of order that pixels will be transmitted. Since every
packet has the same number of pixels, the packet ID will give the receiver the
starting pixel number from which the list of pixels received in the packet can
be extracted.

Pixels are indexed column-first as seen in C, not row first as is typically
done in Python. You therefore have to transpose a matrix before selecting and
assigning pixels if you are working in Python.

####  3.3.3 PCSI Payload base91 Encoding

If you are transmitting over channels that only allow printable ASCII text,
the entire PDP can be converted to base91 as described below. This is a
combination of APRS base91 and basE91 [9]. Compared to basE91, the method used
in PCSI is simpler and deterministic at the cost of slightly more overhead.

While there are 13 bits or more to convert, read in 13 bits Convert those 13
bits to two ASCII bytes using [floor(bits/91)+33] for first and [bits%91+33]
for the second byte. Next, if there are fewer than thirteen and more than
seven bits available (the end of the stream), read in and zero pad (to the
right, i.e., least significant bits) the remaining bits so that there are 13
bits total. Convert those 13 bits to two ASCII bytes using [floor(bits/91)+33]
for first and [bits%91+33] for the second byte If there are 6 or few bits
remaining: Read in and zero pad (to the right, i.e., least significant bits)
the remaining bits so that there are 6 bits total. Convert those 6 bits to one
ASCII byte using bits+33.

###  3.4 Reconstructing PCSI Images

There is no specification or standard on how to reconstruct the images. Users
can experiment with different methods and find what is appropriate. The
reference implementation follows these steps [10]:

  1. 1.

Decode all the pixel values and pixel numbers from as many packets as have
been successfully received.

  2. 2.

For each color channel (Y, Cb, Cr), use OLW-QN for basis pursuit [11] to find
the discrete cosine transform (DCT) coefficients that best fit the received
data and minimizes the L1 norm. This is the key to compressed sensing!

  3. 3.

After finding the DCT coefficients, use the inverse DCT to generate the color
channels for the image.

  4. 4.

Convert from YCbCr to RGB, and save the image.

##  4 Future Work

PCSI is available for use any band and KISS compatible TNC or software modem.
Now that it has been demonstrated, some additional features can be explored:

  * •

Low-power micro-controller transmission client: PCSI transmission is
computationally simple enough to be performed on low-memory micro-controllers.
High-altitude balloons would benefit from integrating PCSI transmission with
Arduino radios such as the HamShield (https://inductivetwig.com/).

  * •

PCSI aggregation server: Since different stations can receive different
packets, and increasing the packets increases image quality, a centralized
server can be used to aggregate packets to improve image quality. This system
would be similar to the Automatic Picture Relay Network (APRN [12]) and to
what is done with SSDV to receive images from high-altitude balloons that move
out of range of the original receiving system [13].

  * •

Integrate PCSI with APRS: While transmitting entire PCSI images over APRS
channels would severely strain the network, APRS could be leveraged to
announce ongoing transmission or upcoming “ImageNets” on non-APRS frequencies.
APRS frequency objects can be transmitted following the conventions of the
Automatic Frequency Reporting System (AFRS) [14] and APRS Local Frequency Info
Initiative [15].

## List of Terms

Notation |  Meaning
---|---
AFSK |  Audio Frequency Shift Keying
APRS |  Automatic Packet Reporting System
ASCII |  American Standard Code for Information Interchange
AX.25 |  Amateur X.25
BER |  Bit Error Rate
CRC |  Cyclic Redundancy Check
CW |  Continuous Wave
DCC |  Digital Communications Conference
DCT |  Discrete Cosine Transform
FCS |  Frame Check Sequence
FEC |  Forward Error Correction
FX.25 |  Extension to AX.25 with FEC
GCC |  GNU Compiler Collection
GUI |  Graphical User Interface
HDLC |  High-Level Data Link Control
HF |  High Frequency
IDCT |  Inverse Discrete Cosine Transform
ITU |  International Telecommunication Union
JPEG |  Joint Photographic Experts Group
KISS |  Keep It Simple, [Silly]
PCSI |  Packet Compressed Sensing Imaging
PDP |  Pseudo-random Datagram Payload (PDP)
OLW-QN |  Orthant-Wise Limited-memory Quasi-Newton
RGB |  Red, Green, Blue
SSDV |  Slow Scan Digital Video
SSTV |  Slow Scan Television
TCP |  Transmission Control Protocol
TNC |  Terminal Node Controller
UDP |  User Datagram Protocol
UI |  User Interface
Y$\mathrm{C_{B}C_{R}}$ |  Luma, blue-difference and red-difference chroma components

## References

  * [1] Scott Howard  “PCSI: Packet Compressed Sensing Imaging”  URL: https://maqifrnswa.github.io/PCSI/
  * [2] “SSDV: Slow Scan Digital Video”  UK High Altitude Society  URL: https://ukhas.org.uk/guides:ssdv
  * [3] Bob Bruninga  “APRS Vision System”  In _TAPR/ARRL DCC Proceedings_ , 1997  URL: https://www.tapr.org/pdf/DCC1997-APRSvision-WB4APR.pdf
  * [4] Frank Durenberg  “The Hellschreiber System - How It Works”  URL: https://www.nonstopsystems.com/radio/hellschreiber-function-operation.htm
  * [5] William A. Beech, Douglas E. Nielsen and Jack Taylor  “AX.25 Link Access Protocol for Amateur Packet Radio, Version 2.2”  TAPR, 1998  URL: https://www.tapr.org/pdf/AX25.2.2.pdf
  * [6] “YCbCr: JPEG Conversion”  URL: https://en.wikipedia.org/wiki/YCbCr#JPEG_conversion
  * [7] “Linear Congruential Generator”  URL: https://en.wikipedia.org/wiki/Linear_congruential_generator
  * [8] Richard Durstenfeld  “Algorithm 235: Random permutation”  In _Comm. ACM_ 7.7, 1964, pp. 420
  * [9] Joachim Henke  “BasE91 Encoding”  URL: https://base91.sourceforge.net/
  * [10] Richard Taylor  “Compressed Sensing in Python”  URL: http://www.pyrunner.com/weblog/2016/05/26/compressed-sensing-python/
  * [11] Galen Andrew and Jianfeng Gao  “Scalable training of L1-regularized log-linear models”  In _Proc. of the 24th Inter. Conf. on Machine Learning_ , 2007
  * [12] Bob Bruninga  “Automatic Picture Relay Network”  URL: http://www.aprs.org/aprn.html
  * [13] “SSDV Live Images”  UK High Altitude Society  URL: http://ssdv.habhub.org/
  * [14] Bob Bruninga  “Automatic Frequency Reporting System”  URL: http://www.aprs.org/afrs.html
  * [15] Bob Bruninga  “APRS Local Frequency Info Initiative”  URL: http://www.aprs.org/localinfo.html