11institutetext: University of Science and Technology of China, Hefei, China
22institutetext: Institute of Artificial Intelligence, Hefei Comprehensive
National Science Center, Hefei, China
22email<EMAIL_ADDRESS><EMAIL_ADDRESS>
33institutetext: Anhui University, Hefei, China
33email<EMAIL_ADDRESS>44institutetext: Qufu Normal University,
Qufu, China
44email<EMAIL_ADDRESS>55institutetext: China University of Mining &
Technology, Beijing, China
55email<EMAIL_ADDRESS>
# PE-YOLO: Pyramid Enhancement Network for Dark Object Detection

Xiangchen Yin 11 2 (🖂) 2 (🖂) Zhenda Yu 2233 Zetao Fei 44 Wenjun Lv 11 Xin Gao
55

###### Abstract

Current object detection models have achieved good results on many benchmark
datasets, detecting objects in dark conditions remains a large challenge. To
address this issue, we propose a pyramid enhanced network (PENet) and joint it
with YOLOv3 to build a dark object detection framework named PE-YOLO. Firstly,
PENet decomposes the image into four components of different resolutions using
the Laplacian pyramid. Specifically we propose a detail processing module
(DPM) to enhance the detail of images, which consists of context branch and
edge branch. In addition, we propose a low-frequency enhancement filter (LEF)
to capture low-frequency semantics and prevent high-frequency noise. PE-YOLO
adopts an end-to-end joint training approach and only uses normal detection
loss to simplify the training process. We conduct experiments on the low-light
object detection dataset ExDark to demonstrate the effectiveness of ours. The
results indicate that compared with other dark detectors and low-light
enhancement models, PE-YOLO achieves the advanced results, achieving 78.0$\%$
in mAP and 53.6 in FPS, respectively, which can adapt to object detection
under different low-light conditions. The code is available at
https://github.com/XiangchenYin/PE-YOLO.

###### Keywords:

Object detection Low-light perception Pyramid enhancement

##  1 Introduction

In recent years, the emergence of convolutional neural networks (CNNs) has
promoted the development of object detection. A large number of detectors have
been proposed, and the performance of the benchmark datasets is getting
enjoyable results [8, 14, 19, 1]. However, most of the existing detectors are
studied in high-quality images under normal conditions. In the real
environment, there are often many bad lighting conditions such as night, dark
light, and exposure, so that the quality of the image decreases affects the
performance of the detector. The visual perception model enables the automatic
system to understand the environment and lay the foundation for subsequent
tasks such as trajectory planning, which requires a robust object detection or
semantic segmentation model. Fig. 1 is an example of dark object detection. It
can be found that if the image is appropriately enhanced and restores more
potential information of the original fuzzy object according to environmental
conditions, the object detection model is adapted to different low-light
conditions, which is also a great challenge in the practical application of
the model.

Figure 1: Example of dark object detection. In dark conditions PE-YOLO can
recover more potential information of object to get better detection results.

Currently, many methods have been proposed to solve the robustness problem in
the dark scenes. Many low-light enhancement models [24, 26, 7, 10] have been
proposed to restore image details and reduce the impact of poor lighting
conditions. However, the structure of the low light enhancement model is
complex, which is not conducive to the real-time performance of the detector
after image enhancement. Most of these methods cannot be end-to-end trained
with the detector, and supervised learning is required for paired low-light
images and normal images. Object detection under low-light conditions can also
be seen as a domain adaptation problem. Some researchers [21, 4, 13] have used
adversarial learning to transfer the model from normal light to dark light.
But they focus on matching data distribution and overlook the potential
information contained in low-light images. In the past few years, some
researchers [15, 11] have proposed the method of using differentiable image
processing (DIP) modules to enhance images and train detectors on in an end-
to-end manner. However, DIP are traditional methods such as white balance,
which have limited enhancement effects on images.

To address the above issues, we propose a pyramid enhancement network (PENet)
that enhance low-light images and capture potential information about objects.
We have joint PENet with YOLOv3 to construct an end-to-end dark object
detection framework called PE-YOLO. In PENet, we first decompose the image
into multiple components of different resolutions using the Laplacian pyramid.
In each scale of the pyramid, we propose detail processing module (DPM) and
low-frequency enhancement filter (LEF) to enhance the components. DPM consists
of context branch and edge branch, which context branch globally enhance
components by capturing long-range dependencies and edge branch enhance the
texture of components. LEF uses a dynamic low-pass filter to obtain low-
frequency semantics and prevent high-frequency noise to enrich feature
information. We only use normal detection loss during model training to
simplify the training process, without the need for clear ground truth of the
image. We validated the effectiveness of our method in the low-light object
detection dataset ExDark [16], the results show that compared with other dark
detectors and low-light enhancement models, PE-YOLO achieved the advanced
results, reaching 78.0% in mAP and 53.6 in FPS respectively, which can adapt
to object detection in dark conditions.

Our contribution could be summarised as follow:

  * $\bullet$

We build a pyramid enhancement network (PENet) to enhance different low-light
images. We propose a detail processing module (DPM) and a low-frequency
enhancement filter (LEF) to enhance the components.

  * $\bullet$

By jointing PENet with YOLOv3, we propose an end-to-end trained dark object
detection framework PE-YOLO to adapt the dark conditions. During training, we
only use normal detection loss.

  * $\bullet$

Compared with other dark detectors and low light enhancement models, our PE-
YOLO achieved advanced results in ExDark dataset, achieving enjoyable accuracy
and speed.

##  2 Related Work

###  2.1 Object Detection

Object detection models are divided into three categories: one-stage models,
two-stage models, and anchor-free-based models. Faster RCNN [20] does not
obtain region recommendations through selective search, but rather through a
region proposal network (RPN). It enables candidate region proposals, feature
extraction, classification, and regression to be trained end-to-end within the
same network. Cai et al. propose Cascade RCNN [2], which cascades multiple
detection heads, and the current level will refine the regression and
classification results of the previous level. YOLOv3 [19] proposed the new
feature extraction network DarkNet-53. Inspiring from the feature pyramid
nework (FPN), YOLOv3 adopts multi-scale feature fusion. In addition, recently
anchor-free-based detectors [23, 12] have appeared, they abandoned anchor and
changed it to key point-based detection.

###  2.2 Low-light Enhancement

The goal of low-light enhancement tasks is to improve human visual perception
by restoring image details and correcting color distortion and to provide
high-quality images for high-level visual tasks such as object detection.
Zhang et al. [26] propose Kind, it can be trained through paired images with
different levels of illumination, without the need for ground truth. Guo et
al. [9] propose Zero DCE, which transforms low-light enhancement tasks into
image-specific curve estimation problems. Lv et al. [17] propose a multi-
branch low light enhancement network (MBLLEN), which extracts features at
different levels and generates output images through multi branch fusion. Cui
et al. [5] propose an Illumination Adaptive Transformer (IAT), through dynamic
query learning to construct an end-to-end Transformer. After the low-light
enhancement model restores the details of the image, the effect of the
detector is improved. However, most low-light enhancement models are complex
and have a great impact on the real-time performance of the detector.

###  2.3 Object Detection in Adverse Condition

Object detection under adverse conditions is crucial for the robust perception
of robots, and robust object detection models have emerged for some adverse
conditions. Some people transfer detectors from the source domain to the
target domain through unsupervised domain adaptation[21, 4, 13], adapting the
model to harsh environments. Liu et al. [15]propose IA-YOLO, which adaptively
enhances each image to improve detection performance. They propose a
differentiable image processing (DIP) module for harsh weather and used a
small convolutional neural network (CNN-PP) to adjust the parameters of DIP.
On the basis of IA-YOLO, Kalwar et al.[11] propose GDIP-YOLO. GDIP proposes a
gating mechanism that allows multiple DIPs to operate in parallel. Qin et al.
[18] propose detection-driven enhancement network (DENet) is used for object
detection in adverse weather condition. Cui et al. [6] propose a multi-task
automatic encoding transform (MAET) for dark object detection, exploring the
potential space behind lighting conversion.

##  3 Method

Dark images have poor visibility due to low-light interference, which affects
the performance of the detector. To address this issue, we propose a pyramid
enhanced network (PENet) and joint YOLOv3 to construct a dark object detection
framework PE-YOLO. The overview of the framework of PE-YOLO is shown in Fig.
2.

###  3.1 Overview of PE-YOLO

PENet decomposes the image into components of different resolutions through
the Laplacian pyramid. In PENet, we enhance the components of each scale
through proposed detail processing module (DPM) and low-frequency enhancement
filter (LEF).

Define the image $I\in R^{h\times w\times 3}$ as input, we obtain sub images
of different resolutions using a Gaussian pyramid.

$G(x)=Down(Gaussian(x))$ (1)

where Down represents downsampling, Gaussian represents Gaussian filter, and
the size of Gaussian kernel is $5\times 5$. After each Gaussian pyramid
operation, the width and height of the image are halved, which means the
resolution is the original $\frac{1}{4}$. Obviously, the downsampling
operation of the Gaussian pyramid is irreversible. In order to recover the
original high-resolution image after upsampling, the lost information is
required, and the lost information constitutes the components of the Laplacian
pyramid. The definition of the Laplacian pyramid is

$L_{i}=G_{i}-Up(G_{i+1})$ (2)

among which $L_{i}$ is the $i^{th}$ layer of the Laplacian pyramid, $G_{i}$
represents the $i^{th}$ layer of the Laplacian pyramid, and Up represents
bilinear upsampling operation. When reconstructing the image, we only need to
perform the reverse operation of (2) to restore the high-resolution image.

Figure 2: Overview of PE-YOLO. We use detail processing module (DPM) and low-
frequency enhancement filter (LEF) to enhance the images.

We obtained four components of different scales through the Laplace pyramid,
as shown in Fig. 3. We found that the Laplacian pyramid pays more attention to
global information from bottom to top, while on the contrary it pays more
attention to local details. They are all information lost during the image
downsampling process, which is also the object of our PENet enhancement. We
enhance the components through detail processing module (DPM) and low-
frequency enhancement filter (LEF), and the operations of DPM and LEF are
parallel. We will provide introduction to DPM and LEF later in next section.
By decomposing and reconstructing the Laplacian pyramid, PENet can be made
lightweight and effective, which helps to improve the performance of PE-YOLO.

Figure 3: Visualization of each layer in the Laplacian pyramid. The second to
fourth column is the component of the Laplacian pyramid, and the resolution is
reduced from left to right.

###  3.2 Detail Enhancement

We propose a detail processing module (DPM) to enhance the components in the
Laplacian pyramid, which is divided into contextual branch and edge branch.
The details of DPM are shown in Fig. 5. Context branch obtains contextual
information by capturing remote dependencies, and globally enhances
components. The edge branch uses two Sobel operators in different directions
to calculate image gradients to obtain edges and enhance the texture of the
components.

Context branch. We use a residual block to process features in before and
after obtaining remote dependencies, and residual learning allows rich low-
frequency information to be transmitted through skip connections. The first
residual block changes the channel of the feature from 3 to 32, and the second
residual block changes the channel of the feature from 32 to 3. Capturing
global information in the scene has been proven to be beneficial for low-level
visual tasks such as low-light enhancement. The structure of the context
branch is described in Fig. 5, which is defined as

$CB(x)=x+\gamma(F_{1}(\hat{x}))$ (3)

where $\hat{x}=\sigma(F_{2}(x))\cdot x$, $F$ is the convolutional layer with
kernel $3\times 3$, $\gamma$ is Leaky ReLU, and $\sigma$ is the Softmax
function.

Edge branch. Sobel operator is a discrete operator that uses both Gaussian
filter and differential derivation. It can find edges by calculating gradient
approximation. We use Sobel operators in both the horizontal and vertical
directions to re-extract edge information through convolutional filters and
use residuals to enhance the flow of information. This process is represented
as

$EB(x)=F_{3}(Sobel_{h}(x)+Sobel_{w}(x))+x$ (4)

where $Sobel_{h}$ and $Sobel_{w}$ represents Sobel operations in the vertical
and horizontal directions, respectively.

###  3.3 Low-Frequency Enhancement Filter

In each scale component, the low-frequency component has most of the semantic
information in the image, and they are the key information for the detector
prediction. To enrich the semantics of the reconstructed image, we propose
low-frequency enhancement filter (LEF) to capture low-frequency information in
the components. The details of LEF are shown in Fig. 5. Assuming the component
$f\in R^{h\times w\times 3}$, we first transform it into $f\in R^{h\times
w\times 32}$ through a convolutional layer. We use a dynamic low-pass filter
to capture low-frequency information, and we use average pooling for feature
filtering, which only allows information below the cutoff frequency to pass
through. The low-frequency thresholds for different semantics are different.
Considering the multi-scale structure of Inception [22], we used adaptive
average pooling with sizes of $1\times 1$, $2\times 2$, $3\times 3$, $6\times
6$, and used upsampling at the end of each scale to restore the original size
of the features. A low-pass filter is formed under average pooling of
different kernel sizes. We divide $f$into four parts through channel
separation, namely $\\{f_{1},f_{2},f_{3},f_{4}\\}$. Each part is processed
using different sizes of pooling, which is described as

$Filter(f_{i})=Up(\beta_{s}(f_{i}))$ (5)

where $f_{i}$ is part of $f$ divided on the channel, Up is bilinear
interpolation sampling, $\beta_{s}$ is an adaptive average pooling of
different sizes of $s\times s$. Finally, after tensor splicing each
{$f_{i},i=1,2,3,4$}, we restore them to $f\in R^{h\times w\times 3}$.

Figure 4: Detail of DPM, contains context branch (CB) and edge branch (EB).

Figure 5: Details of low-frequency enhancement filter (LEF). LEF is composed
of adaptive averge pooling in different sizes to intercept low-frequency
components.

##  4 Experiments

###  4.1 Dataset and Implementation Details

Dataset: We use the ExDark dataset to validate the effectiveness of our PE-
YOLO. ExDark is a low-light object detection dataset used for research on
object detection and image enhancement. It collected a total of 7363 images
under 10 different lighting conditions, from extremely low light to dusk, with
12 bounding box annotations of objects in the images. We divided ExDark into
80$\%$ for training and 20$\%$ for testing, and the specific division is
consistent with IAT [5] and MAET [6].

Details: All trained and tested images are resized to $608\times 608$, and
data augmentation methods such as random cropping, flipping, and multi-scale
resizing are used during training. Batch-size is set to 8, the optimizer uses
SGD, the initial learning rate is set to 0.001, and the weight decay is set to
0.0005. Train PE-YOLO for 30 epochs and run our model on a single RTX 3090
GPU. The deep learning framework is Pytorch, and we use mmdetection[3] to
achieve our model.

Evaluation: We use mAP and FPS to validate the effectiveness of our model. mAP
is the average AP of all categories in the detection model, and a larger value
indicates a higher accuracy of the model. It is represented as

$mAP=\frac{\sum^{C}_{i=1}AP_{i}}{C}$ (6)

where $C$ is the number of categories, and $AP$ is the Averge Precision for
each category, calculated by the area of the Precision Recall curve. FPS is
the number of image frames detected by the model per second, and a larger FPS
indicates a faster model detection speed.

###  4.2 Experimental Results

To verify the effectiveness of PE-YOLO, we conducted many experiments on the
ExDark dataset. Firstly, we compare PE-YOLO with other low-light enhancement
models. Due to the lack of detection capability of the low light enhancement
model, we will use the same detector as PE-YOLO to experiment on all enhanced
images. We set the IoU threshold of mAP to 0.5, and the performance comparison
is shown in Table 1. We found that directly using low-light enhanced models
before YOLOv3 did not significantly improve detection performance. Our PE-YOLO
is 1.2$\%$ and 1.1$\%$ higher on mAP than MBLLEN and Zero-DCE, respectively,
achieving the best results.

Table 1:  Performance comparisons between PE-YOLO and low-light enhancement
models. It shows mAP and AP in each class. The bold number has the highest
score in each column.

Model | Venue | Bicycle | Boat | Bottle | Bus | Car | Cat | Chair | Cup | Dog | Motorbike | People | Table | mAP
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
YOLOv3[19] | arXiv | 79.8 | 75.3 | 78.1 | 92.3 | 83.0 | 68.0 | 69.0 | 79.0 | 78.0 | 77.3 | 81.5 | 55.5 | 76.4
KIND[26] | MM2019 | 80.1 | 77.7 | 77.2 | 93.8 | 83.9 | 66.9 | 68.7 | 77.4 | 79.3 | 75.3 | 80.9 | 53.8 | 76.3
MBLLEN[17] | BMVC 2018 | 82.0 | 77.3 | 76.5 | 91.3 | 84.0 | 67.6 | 69.1 | 77.6 | 80.4 | 75.6 | 81.9 | 58.6 | 76.8
Zero-DCE[9] | CVPR 2020 | 84.1 | 77.6 | 78.3 | 93.1 | 83.7 | 70.3 | 69.8 | 77.6 | 77.4 | 76.3 | 81.0 | 53.6 | 76.9
PE-YOLOv3 (Ours) | - | 84.7 | 79.2 | 79.3 | 92.5 | 83.9 | 71.5 | 71.7 | 79.7 | 79.7 | 77.3 | 81.8 | 55.3 | 78.0

We visualized the detection results of different low light enhancement models,
as shown in Fig. 6. We found that although MBLLEN and Zero DCE can
significantly improve the brightness of the image, they also enlarge the noise
in the image. PE-YOLO mainly captures the potential information of objects in
low-light images, while suppressing noise in high-frequency components, thus
PE-YOLO has better detection performance.

Figure 6: Detection Results in PE-YOLO and other low-light enhancement models.

We compared the performance of PE-YOLO with other dark detectors, as shown in
Table 2. In addition, we visualized the detection results of the dark detector
and PE-YOLO, as shown in Fig. 7, which clearly showed that PE-YOLO was more
accurate in object detection. PE-YOLO is 0.7$\%$ and 0.2$\%$ higher in mAP
compared to DENet and IAT-YOLO pre-trained with the LOL dataset, and our PE-
YOLO is also almost the highest on FPS. The above data indicate that PE-YOLO
is more suitable for detecting objects in dark conditions.

Table 2:  Performance comparisons between PE-YOLO and dark detectors. The bold
number has the highest score in each column.

Model | Venue | Bicycle | Boat | Bottle | Bus | Car | Cat | Chair | Cup | Dog | Motorbike | People | Table | mAP | FPS
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
YOLOv3[19] | arXiv | 79.8 | 75.3 | 78.1 | 92.3 | 83.0 | 68.0 | 69.0 | 79.0 | 78.0 | 77.3 | 81.5 | 55.5 | 76.4 | 54.0
MAET[6] | ICCV 2021 | 83.1 | 78.5 | 75.6 | 92.9 | 83.1 | 73.4 | 71.3 | 79.0 | 79.8 | 77.2 | 81.1 | 57.0 | 77.7 | -
IAT-YOLOV3 (LOL pretrain)[5] | BMVC2022 | 79.8 | 76.9 | 78.6 | 92.5 | 83.8 | 73.6 | 72.4 | 78.6 | 79.0 | 79.0 | 81.1 | 57.7 | 77.8 | 52.6
DENet[18] | ACCV2022 | 80.4 | 79.7 | 77.9 | 91.2 | 82.7 | 72.8 | 69.9 | 80.1 | 77.2 | 76.7 | 82.0 | 57.2 | 77.3 | 54.8
PE-YOLOv3 (Ours) | - | 84.7 | 79.2 | 79.3 | 92.5 | 83.9 | 71.5 | 71.7 | 79.7 | 79.7 | 77.3 | 81.8 | 55.3 | 78.0 | 53.6

###  4.3 Ablation Study

To analyze the effectiveness of each component in PE-YOLO, we conducted
ablation studies, and the results are shown in Table 3. After adopting context
branching, PE-YOLO increased from 76.4$\%$ to 77.0$\%$ in mAP, indicating that
capturing remote dependencies is effective for enhancement. After adopting
edge branch, the mAP increased from 77.0% to 77.6%, indicating that edge
branch can enhance the texture of components and enhance the details of the
enhanced image. After adopting LEF, the mAP increased from 77.6% to 78.0%,
indicating that capturing low-frequency components is beneficial for obtaining
potential information in the image. In the end, our model improved from
76.4$\%$ to 78.0$\%$ on mAP and only decreased by 0.4 on FPS.

Figure 7: Detection Results in PE-YOLO and other dark detectors. Table 3:  Ablation study on PE-YOLO. “CB” represents context branch, “EB” represents edge branch, and LEF represents low-frequency enhancement filter. DPM | LEF | mAP | FPS
---|---|---|---
CB | EB
|  |  | 76.4 | 54.0
✓ |  |  | 77.0 | 53.9
✓ | ✓ |  | 77.6 | 53.8
✓ | ✓ | ✓ | 78.0 | 53.6

##  5 Conclusion

To achieve more robust dark object detection, we propose a pyramid enhancement
network (PENet) that performs detail restoration and captures potential
information. By combining PENet and YOLOv3, we build a dark object detection
framework named PE-YOLO. We first use the Laplacian pyramid to decompose the
image into four components with different resolutions, and propose a detail
processing module (DPM) and a low-frequency enhancement filter (LEF) for
component enhancement. In addition, PE-YOLO trains in an end-to-end way,
without additional loss function. We conducted experiments in the ExDark
dataset, and the experimental results show that compared with the low-light
enhancement models and the dark detectors, PE-YOLO achieves the best results
and can effectively detect objects in dark conditions. However, our model
should be studied on more detectors and further improve performance while
maintaining lightweight.

## References

  * [1] Bochkovskiy, A., Wang, C.Y., Liao, H.Y.M.: Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934 (2020)
  * [2] Cai, Z., Vasconcelos, N.: Cascade r-cnn: Delving into high quality object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6154–6162 (2018)
  * [3] Chen, K., Wang, J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Xu, J., et al.: Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155 (2019)
  * [4] Chen, Y., Li, W., Sakaridis, C., Dai, D., Van Gool, L.: Domain adaptive faster r-cnn for object detection in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3339–3348 (2018)
  * [5] Cui, Z., Li, K., Gu, L., Su, S., Gao, P., Jiang, Z., Qiao, Y., Harada, T.: You only need 90k parameters to adapt light: a light weight transformer for image enhancement and exposure correction. In: 33rd British Machine Vision Conference 2022, BMVC 2022, London, UK, November 21-24, 2022. BMVA Press (2022), https://bmvc2022.mpi-inf.mpg.de/0238.pdf
  * [6] Cui, Z., Qi, G.J., Gu, L., You, S., Zhang, Z., Harada, T.: Multitask aet with orthogonal tangent regularity for dark object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2553–2562 (2021)
  * [7] Dudhane, A., Zamir, S.W., Khan, S., Khan, F.S., Yang, M.H.: Burst image restoration and enhancement. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5759–5768 (2022)
  * [8] Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. International journal of computer vision 88, 303–338 (2010)
  * [9] Guo, C., Li, C., Guo, J., Loy, C.C., Hou, J., Kwong, S., Cong, R.: Zero-reference deep curve estimation for low-light image enhancement. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1780–1789 (2020)
  * [10] Guo, X., Li, Y., Ling, H.: Lime: Low-light image enhancement via illumination map estimation. IEEE Transactions on image processing 26(2), 982–993 (2016)
  * [11] Kalwar, S., Patel, D., Aanegola, A., Konda, K.R., Garg, S., Krishna, K.M.: Gdip: Gated differentiable image processing for object-detection in adverse conditions. arXiv preprint arXiv:2209.14922 (2022)
  * [12] Law, H., Deng, J.: Cornernet: Detecting objects as paired keypoints. In: Proceedings of the European conference on computer vision (ECCV). pp. 734–750 (2018)
  * [13] Li, C., Du, D., Zhang, L., Wen, L., Luo, T., Wu, Y., Zhu, P.: Spatial attention pyramid network for unsupervised domain adaptation. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIII 16. pp. 481–497. Springer (2020)
  * [14] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pp. 740–755. Springer (2014)
  * [15] Liu, W., Ren, G., Yu, R., Guo, S., Zhu, J., Zhang, L.: Image-adaptive yolo for object detection in adverse weather conditions. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 1792–1800 (2022)
  * [16] Loh, Y.P., Chan, C.S.: Getting to know low-light images with the exclusively dark dataset. Computer Vision and Image Understanding 178, 30–42 (2019)
  * [17] Lv, F., Lu, F., Wu, J., Lim, C.: Mbllen: Low-light image/video enhancement using cnns. In: BMVC. vol. 220, p. 4 (2018)
  * [18] Qin, Q., Chang, K., Huang, M., Li, G.: Denet: Detection-driven enhancement network for object detection under adverse weather conditions. In: Proceedings of the Asian Conference on Computer Vision. pp. 2813–2829 (2022)
  * [19] Redmon, J., Farhadi, A.: Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767 (2018)
  * [20] Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems 28 (2015)
  * [21] Sasagawa, Y., Nagahara, H.: Yolo in the dark-domain adaptation method for merging multiple models. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16. pp. 345–359. Springer (2020)
  * [22] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1–9 (2015)
  * [23] Tian, Z., Shen, C., Chen, H., He, T.: Fcos: Fully convolutional one-stage object detection. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 9627–9636 (2019)
  * [24] Wei, C., Wang, W., Yang, W., Liu, J.: Deep retinex decomposition for low-light enhancement. arXiv preprint arXiv:1808.04560 (2018)
  * [25] Wu, A., Han, Y., Zhu, L., Yang, Y.: Instance-invariant domain adaptive object detection via progressive disentanglement. IEEE Transactions on Pattern Analysis and Machine Intelligence 44(8), 4178–4193 (2021)
  * [26] Zhang, Y., Zhang, J., Guo, X.: Kindling the darkness: A practical low-light image enhancer. In: Proceedings of the 27th ACM international conference on multimedia. pp. 1632–1640 (2019)


# On a Constrained Pseudoinverse for the Electromagnetic Inverse Source
Problem

E. Citraro  A. Dély  A. Merlini  F. P. Andriulli

###### Abstract

Inverse source approaches have shown their relevance for several applications
in the past years. They rely on the solution of an ill-posed problem where
near-field/current data is reconstructed starting from far-field (or less
informative field) information. Standard strategies, including the physically
constrained ones using Love conditions, result in linear systems to be
pseudoinverted which are still ill-conditioned due to the lack of information
from the evanescent fields. In this work we present a generalized
pseudoinverse for those problems that allows the inclusion of extra
constraints from the evanescent field space, when available. This is obtained
by dropping some of the standard Moore-Penrose (MP) requirements and using the
resulting degrees of freedom to obtain a generalized pseudoinverse that shows
favorable performance in several cases of practical interest.

On a Constrained Pseudoinverse for the Electromagnetic Inverse Source Problem

${}^{(\ref{})}$

(1) Politecnico di Torino, Turin, Italy

(2) IMT Atlantique, Brest, France

##  I Introduction

Electromagnetic inverse source problems have been effectively used for
tackling a variety of application scenarios including antenna diagnostics and
far to near field reconstructions (see [2] and references therein). In light
of the equivalence theorem, many formulations are allowed and proved to be
comparable, in reconstructing the far-field at least, in terms of speed and
accuracy once discretized and solved [1]. However, all methods inherit the
nature of the same electromagnetic problem and describe a common ill-posedness
which gets more severe as the distance from the source(s) increases. In fact,
when dealing with far-field measurements, the inevitable loss of information
associated with the evanescent modes compromises near-field reconstruction:
all the measured power concentrates in the well discretized propagating modes
[3]. On the one hand this problem has been partially mitigated using algebraic
pseudoinverses, most notably the Moore-Penrose pseudoinverse (MPP), associated
with constraints like the Love one [2], which however has limited impact on
the near-field reconstruction given that the regularization is done with
information from the internal equivalent problem. Other approaches, often in
the context of near to far field transformations and pattern characterization
(see the nice work [4] and related papers) have been instead focusing in
recovering overcomplete solution spaces in which the reconstruction is
performed. In this work, we propose a different approach which is somehow in
between the two. A limited number of physical constraints (a substantially
undercomplete basis) is considered and used to define a generalized
pseudoinverse that still allows for norm minimization or equivalent conditions
and, most importantly, is still compatible with Love constraints that are so
popular in the community. The pseudoinverse we propose compensates for the
loss of evanescent degrees of freedom with a limited number of constraints
obtained by measurements or simulation while still solving the better
conditioned part of the matrix through an algebraic pseudoinversion. The new
pseudoinverse is compatible with most of the existing inverse source
formulations (where it can replace the MPP or equivalent). Theoretical
considerations are complemented by numerical results showing the practical
relevance of the proposed approach.

##  II Notation and Background

A given source can be enclosed in a fictitious equivalent surface $\Gamma$
which delimits an internal/external volume $\Omega^{-}/\Omega^{+}$. While
$\Omega^{+}$ retains the properties of the original medium, $\Omega^{-}$ can
be filled with whatever material; here the same permittivity $\epsilon_{0}$
and permeability $\mu_{0}$ as $\Omega^{+}$ are chosen. The equivalence theorem
allows us to find a set of surface currents densities
$\boldsymbol{M}=\boldsymbol{\hat{n}}\times\left(\boldsymbol{E^{-}}-\boldsymbol{E^{+}}\right)$
and
$\boldsymbol{J}=\boldsymbol{\hat{n}}\times\left(\boldsymbol{H^{+}}-\boldsymbol{H^{-}}\right)$
on $\Gamma$ that radiate the same fields as the source, where
$\boldsymbol{\hat{n}}$ is the unit normal vector field on $\Gamma$ pointing
into $\Omega^{+}$ and $\boldsymbol{E^{+}},\,\boldsymbol{H^{+}}$ (resp.
$\boldsymbol{E}^{-},\,\boldsymbol{H}^{-}$) are the electric and magnetic field
in $\Omega^{+}$ (resp. $\Omega^{-}$).

We then define the electric operator on $\Gamma$ as
${\mathcal{T}}(\boldsymbol{J})(\boldsymbol{r})=ik\,{\mathcal{T}}_{s}(\boldsymbol{J})(\boldsymbol{r})-(ik)^{-1}\,{\mathcal{T}}_{h}(\boldsymbol{J})(\boldsymbol{r})$
with $\boldsymbol{r}\in\Gamma$, $k$ the wavenumber,
${\mathcal{T}}_{s}(\boldsymbol{J})(\boldsymbol{r})=\boldsymbol{\hat{n}}\times\int_{\Gamma}\frac{e^{ik\left|\boldsymbol{r}-\boldsymbol{r}^{\prime}\right|}}{4\pi\left|\boldsymbol{r}-\boldsymbol{r}^{\prime}\right|}{\boldsymbol{J}(\boldsymbol{r}^{\prime})}\,\mathrm{d}\boldsymbol{r}^{\prime}$,
and
${\mathcal{T}}_{h}(\boldsymbol{J})(\boldsymbol{r})=\boldsymbol{\hat{n}}\times\nabla\int_{\Gamma}\frac{e^{ik\left|\boldsymbol{r}-\boldsymbol{r}^{\prime}\right|}}{4\pi\left|\boldsymbol{r}-\boldsymbol{r}^{\prime}\right|}{\nabla_{s}\cdot\boldsymbol{J}(\boldsymbol{r}^{\prime})}\,\mathrm{d}\boldsymbol{r}^{\prime}$.
The magnetic operator is
${\mathcal{K}}(\boldsymbol{J})(\boldsymbol{r})=-\boldsymbol{\hat{n}}\times
p.v.\int_{\Gamma}\nabla\times\frac{e^{ik\left|\boldsymbol{r}-\boldsymbol{r}^{\prime}\right|}}{4\pi\left|\boldsymbol{r}-\boldsymbol{r}^{\prime}\right|}\boldsymbol{J}(\boldsymbol{r}^{\prime})\,\mathrm{d}\boldsymbol{r}^{\prime}$
with $\boldsymbol{r}\in\Gamma$. We extend these definition to the case in
which $\boldsymbol{r}$ belongs to a measurement surface $\Gamma_{m}$ distinct
from $\Gamma$ and denote these extensions as ${\mathcal{T}}_{r}$ and
${\mathcal{K}}_{r}$; their expression are similar to ${\mathcal{T}}$ and
${\mathcal{K}}$ except that the principal value in ${\mathcal{K}}$ is not
necessary. Finally, given a measurement vector $b\in\Gamma_{m}$, the
tangentially projected radiation operator is

${\mathcal{R}}=\begin{bmatrix}-{\mathcal{K}}_{r}&{\mathcal{T}}_{r}\\\
-{\mathcal{T}}_{r}&-{\mathcal{K}}_{r}\end{bmatrix}\,,\quad\boldsymbol{r}\in\Gamma_{m}\,,$
(1)

and its related linear system is

${\mathcal{R}}x={\mathcal{R}}\cdot\begin{bmatrix}-\boldsymbol{M}\\\
\eta_{0}\boldsymbol{J}\end{bmatrix}=b=\boldsymbol{\hat{n}}\times\begin{bmatrix}\boldsymbol{E}^{+}\\\
\eta_{0}\boldsymbol{H}^{+}\end{bmatrix}$ (2)

with $\eta_{0}=\sqrt{\mu_{0}/\epsilon_{0}}$. After discretization (2) reads

$\mathbf{Rx}=\mathbf{b}$ (3)

with $\mathbf{R}$, $\mathbf{x}$ and $\mathbf{b}$ being the Rao-Wilton-Glisson
discretized version of $R$, $x$ and $b$ respectively.

##  III A Constrained Pseudoinverse

Solving the inverse source problem amounts to pseudoinverting $\mathbf{R}$ to
solve the ill-posed problem (3). This can be done after adding direct or
indirect constraints, including single source enforced solutions, Love, and/or
minimum norm conditions (see [1, 2] and references therein). Often, when
$\mathbf{R}$ (or its regularized version) is pseudoinverted, the MPP or an
equivalent iterative pseudoinverse is used. This norm minimizing approach
warranties four fundamental properties for the pseudoinverse, i.e. the MPP
pseudoinverse $\mathbf{R}^{+}$ of $\mathbf{R}$ satisfies 1)
$\mathbf{R}\mathbf{R}^{+}\mathbf{R}=\mathbf{R}$, 2)
$\mathbf{R}^{+}\mathbf{R}\mathbf{R}^{+}=\mathbf{R}^{+}$, 3)
$(\mathbf{R}^{+}\mathbf{R})^{*}=\mathbf{R}^{+}\mathbf{R}$, and 4)
$(\mathbf{R}\mathbf{R}^{+})^{*}=\mathbf{R}\mathbf{R}^{+}$. The MPP is the
unique pseudoinverse that satisfies 1)–4), however, among the above properties
only 1) is unavoidable if we want to find a solution of (3); thus renouncing
to some of the properties 2)-4) may allow to further optimize the
pseudoinverse. A natural starting point is the general pseudoinverse
$\mathbf{R}^{\dagger}$ of $\mathbf{R}$ that only satisfies 1). This would
provide an exact solution for (3) that would allow the injection of a priori
regularity constraints and minimization criteria, but the selection criterion
of the solution may not be sufficient to describe the physics of the problem.
Another pseudoinverse that could be of interest would be the one providing an
approximation of the solution of (3), for a specific set of linearly
independent right hand sides $\mathbf{{\tilde{B}}}$, in the space of linear
combinations of the corresponding known solutions $\mathbf{{\tilde{X}}}$. One
way of designing such a pseudoinverse is
$\mathbf{{\tilde{X}}}\mathbf{{\tilde{B}}}^{\bullet}$ where
$\mathbf{{\tilde{B}}}^{\bullet}$ is a pseudoinverse of $\mathbf{{\tilde{B}}}$
that satisfies 1). Although
$\mathbf{{\tilde{X}}}\mathbf{{\tilde{B}}}^{\bullet}$ provides physically
relevant solutions, when the number of vectors is lower than the degrees of
freedom, it provides approximate solutions that are not necessarily solving
(3). In fact, if $\mathbf{{\tilde{B}}}^{\bullet}$ is a pseudoinverse
satisfying at least 1) then
$\mathbf{{\tilde{X}}}\mathbf{{\tilde{B}}}^{\bullet}$ is a pseudoinverse
satisfying at least 2), so it is not ensured to yield an exact solution of
(3). In this work we propose a suitable operator average of the two
approaches, i.e. a more general pseudoinverse in between the two described
above:

$\mathbf{R}^{\ddagger}=\mathbf{R}^{\dagger}(\mathbf{I}-\mathbf{\mathbf{{\tilde{B}}}\mathbf{{\tilde{B}}}}^{\bullet})+\mathbf{\mathbf{{\tilde{X}}}\mathbf{{\tilde{B}}}}^{\bullet}.$
(4)

The above pseudoinverse has the advantage of providing a part of the solution
as a linear combination of know solutions vectors but, at the same time and
differently from $\mathbf{{\tilde{X}}}\mathbf{{\tilde{B}}}^{\bullet}$,
provides an exact reconstruction of the measurement data since it satisfies 1)
and, in particular,
$\mathbf{R}(\mathbf{R}^{\ddagger}\mathbf{b})=\mathbf{R}(\mathbf{R}^{\ddagger}\mathbf{R}\mathbf{x})=\mathbf{b}$.

##  IV Numerical Results

The constrained pseudoinverse we propose has been numerically analysed. First
the rank-enhancement effect of $\mathbf{R}^{\ddagger}$ with respect to
$\mathbf{R}^{+}$ is shown in Fig. 1 through a semi-analytic spectral analysis
realized with vector spherical harmonics that shows that the novel
pseudoinverse is able to recover the evanescent base truncated by the MPP.

Figure 1: Spectral analysis of the different formulations.

To highlight that the regularization is near-field aware (differently from
other forms of regularization) we have studied a Hertzian dipole oscillating
at $5\text{\,}\mathrm{GHz}$ with a measurement surface at a distance of two
wavelengths from the reconstruction sphere of radius $4\text{\,}\mathrm{cm}$
and we have discretized (3) using the Galerkin method. The measurement vector
is pseudoinverted both with the MPP and the constrained pseudoinverse through
a near-field spherical placement of identical sources. A simplified
representation of the spatial constraining along with reconstruction results
is shown in Fig. 2. Compared to the MPP, $\mathbf{R}^{\ddagger}$ shows an
improvement in near-field reconstruction while not altering the far-field
accuracy, in line with our theoretical considerations.

\begin{overpic}[height=142.26378pt,percent]{error2.eps} \put(35.0,18.0){
\includegraphics[scale={0.22}]{dipoles_merged_comp.png}} \end{overpic} Figure
2: Original (red) and constraining (grey) dipole sources in 3D space and
associated reconstruction error from far-field synthetic measurements.

## ACKNOWLEDGMENT

This work was supported in part by the European Research Council (ERC) through
the European Union’s Horizon 2020 Research and Innovation Programme under
Grant 724846 (Project 321); in part by the Italian Ministry of University and
Research within the Program PRIN2017, EMVISIONING, under Grant 2017HZJXSZ,
CUP:E64I190025300; in part by the Italian Ministry of University and Research
within the Program FARE, CELER, under Grant R187PMFXA4.

## References

  * [1] J. Kornprobst, R. A. M. Mauermayer, O. Neitz, J. Knapp, and T. F. Eibert, “On the Solution of Inverse Equivalent Surface-Source Problems”, PIER, vol. 165, pp. 47–65, 2019.
  * [2] J. L. A. Quijano and G. Vecchi, “Field and Source Equivalence in Source Reconstruction on 3D Surfaces”, PIER, vol. 103, pp. 67–100, 2010.
  * [3] J. E. Hansen, “Spherical Near-ﬁeld Antenna Measurements”, Vol. 26, IEE Electromagnetic Waves Series, Stevenage Herts England Peter Peregrinus Ltd., pp. 8-26, 1988.
  * [4] M. Salucci, M. D. Migliore, G. Oliveri, and A. Massa, “Antenna Measurements-by-Design for Antenna Qualification”, IEEE Trans. Antennas Propagat., vol. 66, no. 11, pp. 6300–6312, Nov. 2018.


# Toward the biological model of the hippocampus as the successor
representation agent

Hyunsu Lee  Department of Anatomy, School of Medicine, Keimyung university,
Daegu, Republic of Korea<EMAIL_ADDRESS>
###### Abstract

The hippocampus is an essential brain region for spatial memory and learning.
Recently, a theoretical model of the hippocampus based on temporal difference
(TD) learning has been published. Inspired by the successor representation
(SR) learning algorithms, which decompose value function of TD learning into
reward and state transition, they argued that the rate of firing of CA1 place
cells in the hippocampus represents the probability of state transition. This
theory, called predictive map theory, claims that the hippocampus representing
space learns the probability of transition from the current state to the
future state. The neural correlates of expecting the future state are the
firing rates of the CA1 place cells. This explanation is plausible for the
results recorded in behavioral experiments, but it is lacking the
neurobiological implications.

Modifying the SR learning algorithm added biological implications to the
predictive map theory. Similar with the simultaneous needs of information of
the current and future state in the SR learning algorithm, the CA1 place cells
receive two inputs from CA3 and entorhinal cortex. Mathematical transformation
showed that the SR learning algorithm is equivalent to the heterosynaptic
plasticity rule. The heterosynaptic plasticity phenomena in CA1 were discussed
and compared with the modified SR update rule. This study attempted to
interpret the TD algorithm as the neurobiological mechanism occurring in place
learning, and to integrate the neuroscience and artificial intelligence
approaches in the field.

###### keywords:

hippocampus , predictive map , successor representation , temporal difference
learning , heterosynaptic plasticity

††journal: under review

##  1 Introduction

The navigating environment, including foraging, nesting, and finding a mate,
is the essential ability for surviving animals. To achieve a navigating goal,
animals developed a specific nervous system, which is believed to be the
hippocampus. After discovery of place cell [18], it emerges the cognitive map
theory which believed that hippocampus encodes spatial memory [14]. With the
case studies of patient H.M. [15], the cognitive map theory has expanded into
a complementary learning system [11]. The hippocampus encodes and transfers
not only spatial memory but also semantic memory and episodic memory to the
neocortex. Episodic memories linking specific locations and events are
essential to animal survival. It is important to remember not only the
location of the foods and the predators, but also its existence. From the
point of view of reinforcement learning (RL) in psychology, we can regard
foods and predators as rewards and punishments, respectively. However, it is
not well known how the hippocampus processes the informations of rewards and
punishments for spatial and episodic memories.

Leveraging the concept of RL derived from psychology is not limited to
neuroscience. In artificial intelligence, they have conducted research using
the concept of RL for a long time [27]. The recent research results, such as
AlphaZero and AlphaStar, have shown to exceed human performance in playing Go
and StarCraft II [24, 29]. From a computational point of view, RL computes the
value of the current state or environment and predicts the value of the future
state after the agent takes action. To achieve this computational goal,
artificial intelligence researchers have developed many algorithms. Among
them, temporal difference (TD) learning is related to neuroscientific
mechanisms of RL found in the brain [22, 26]. TD learning updates the
expectations for the value of the current state by observing the difference
between the predicted and observed values after taking the action: this is the
TD error. It is equivalent to prediction error in the concept of the
predictive coding [17]. Although the prediction error hypothesis gives
integrating insight for psychology, artificial intelligence, and neuroscience,
it is little known about the biological mechanisms of how learning of spatial
memory in the hippocampus is occurring by prediction errors.

One limitation of TD learning is that the agent does not differentiate between
the rewards earned and the states change when it updates the value of the
states after taking an action. Because of that, even if the position of the
reward changes in the same environment, the TD agent has to learn again from
the scratch. But animal learning is more robust and adaptive than this. To
overcome this problem, successor representation (SR) learning has been
proposed [5]. This algorithm decomposes the reward and state transition
functions from the value function update. By applying this idea to hippocampal
learning, a recently published study developed predictive map theory [25]. The
study reported that the learned results from exploring the environment
according to the SR learning rule were similar to the responses recorded in
the animal hippocampus. They argued that place cell of the hippocampus
represents the probability of transition the successor location from the
current location rather than a neural correlate of simple geodetic. They,
however, have not explored the biological mechanisms of SR learning.

In this article, we will modify the SR model by the biological basis of the
hippocampus. To compensate for the lack of biological mechanisms in the SR
model, we will derive the synaptic weight update rule from the SR learning
rule. We will treat the term representing states in the SR model as
presynaptic neuron of the CA1 place cells. As a result, we reveal that the co-
activity of the two presynaptic inputs to the CA1 place cell is important to
update the synaptic weights. And it is noteworthy that this update rule is
comparable to the heterosynaptic plasticity of the CA1 place cells.

##  2 Key ideas from the SR learning model

The key idea of predictive map theory is that place cells encode expectations
of future location from a viewpoint of the current location: the firing rate
of the place cells reflects the probability of transition [25]. If the place
cell encodes a location close to the animal’s current location, it means that
the transitional probability is high, therefore the firing rate of the place
cell will also be high. It is noteworthy that the transitional probability
depends not only on the simple Euclidean distances but also on the choices of
the agent. Even if the location is far away from the current location, if
there is a reward, the transitional probability will be high, and vice versa.
Besides reward/punishment, the habitual behavior of the agent, obstacles, and
detours also affects the probability of transition. The predictive map theory
considers these factors. But the classical cognitive map theory does not
consider above factors but considers only the Euclidean distance.

The [25] introduced the SR learning model as an algorithm to elaborate
predictive map theory. The SR learning was derived from the TD algorithm that
discounts the value of a state depending on the distance to the reward. (From
now on, I will use the term state instead of location. The term state is a
term mainly used for reinforcement learning and includes not only a spatial
location but also an agent state, including selectable actions.) The
expectation of distal rewards $R(s_{t})$ multiplied by a discount factor
$\gamma\in[0,1]$ given current state shows the value of state $V(s)$

$V(s)=\mathbb{E}[\sum^{\infty}_{t=0}\gamma^{t}R(s_{t})|s_{0}=s]$ (1)

$s(t)$ indicates the state visited at time $t$.

The essential idea of the SR learning is that the value function is a
combination of prediction of transition to the successor state $s^{\prime}$
and the reward of that state [5]. Thus, we can decompose them as the following
equation.

$V(s)=\sum_{s^{\prime}}M(s,s^{\prime})R(s^{\prime})$ (2)

We can derive the function $M(s,s^{\prime})$ from the value function (1) by
imputing trajectories of the states instead of $R(s_{t})$ as the following
equation.

$M(s,s^{{}^{\prime}})=\mathbb{E}[\sum^{\infty}_{t=0}\gamma^{t}\mathbb{I}(s_{t}=s^{\prime})|s_{0}=s]$
(3)

$\mathbb{I}(s_{t}=s^{\prime})$ is a Boolean function that returns 1 if the
agent visited $s’$ at time $t$, or 0 otherwise.

The agent can learn the environment incrementally by the TD algorithm. After
taking action and observing the next state, it updates the value of the
current state. Therefore, we can update the estimation of $M(s,s^{\prime})$ by
same manner as the following equation.

$M_{t+1}(s_{t},s^{\prime})=M_{t}(s_{t},s^{\prime})+\eta[\mathbb{I}(s_{t}=s^{\prime})+\gamma
M_{t}(s_{t+1},s^{\prime})-M_{t}(s_{t},s^{\prime})]$ (4)

Although the updating TD algorithm manipulates only the information of the
next state, it is the bootstrap method based on previously learned values.
Thus, as defined by equation (3), the horizon of $s^{\prime}$ in
$M(s,s^{\prime})$ can be infinite. Mathematically, it is appropriate for the
agent to initialize the matrix M to the identity matrix; the $M(s,s^{\prime})$
is 1 if $s=s^{\prime}$, otherwise 0. This is because if the agent does not
take action without prior knowledge of the environment, the only successor
state of the current state is the current state. The size of row and column of
matrix $M$ must be equal to the number of the states in the environment that
the agent will explore (I will discuss this in the limitations).

The simulation results of the SR agent shown in the abovementioned paper [25]
are interesting. The estimated $M(s,s^{\prime})$ value after learning seems
similar to the firing pattern of place cells recorded in the hippocampus CA1
of the murine model. In the meantime, the simple geodesic model has not fully
explained the recorded data in the murine model. It cannot reflect the dynamic
response of place cells to environmental changes such as barriers or detour,
or to the direction of the moving animal. These diverse and dynamic
environmental changes that are likely to occur in the real world, however,
change the transitional probability ($M(s,s^{\prime})$ value) of the SR model.
For examples, on the one-way track, place fields often show the ramp patterns
that gradually increase and abruptly decrease, which are asymmetric firing
patterns. Considering the SR model, the transitional probability gradually
increases before the animal arrives at the place field. Since the animal on
the one-way track rarely goes back, the transitional probability to the place
field after passing it is almost zero. Therefore, the SR model explains the
firing pattern of the place cells better than the simple geodesic model.

##  3 Limitations of the SR model

Based on the similarity of the pattern between the estimated $M$ and the place
field, we can interpret the firing pattern of place cells as the transitional
probability to the future state. Despite the similar generated pattern,
however, the SR model has some limitations from the perspective of biological
neurosciences.

The major concern with the SR model is that it did not reflect realistic
interactions between biological agents and the environment. In the SR model,
the artificial intelligence (AI) agent must determine the size of the $M$
matrix based on the number of movable points in the environments to be exposed
in prior: it is given by the human programmer. If we expose the AI agent to a
new environment after learning the other environment once, the agent must not
only re-learn the transitional probability in the new environment but also
change the size of the $M$ matrix according to the new environment. Knowing
the number of place fields in an unfamiliar environment in prior, however, is
not a plausible condition for biological agents. Rather, they generate the
place fields by exploring the unfamiliar environment; it is impossible for the
AI agent of the SR model.

Despite of the differences, the similarity of the response patterns between
$M$ and place cells makes plausible reasoning for predictive coding in the
hippocampus. In order to apply predictive map theory to interpret data from
animal, we must interpret the algorithmic level description of the SR model at
the implementation level of the biological brain, especially at the cellular
level.

The SR model agent observes the result of the taken action; and it compare
$M_{t}(s_{t},s^{\prime})$ with $M_{t}(s_{t+1},s^{\prime})$ to update the $M$
matrix (equation (4)). This is a simple arithmetic at the algorithm level, but
not at the cellular level. After visiting the next place field, the place cell
should update the firing rate for the current place field by according to the
observed result. Based on this assumption, we can raise some biological
questions. How the firing rate of place cell for the same sensory input (same
place field) changes: what is the fundamental mechanism for that? If changing
the synaptic weights between neurons is the answer, what is the update rule
for the synaptic plasticity? And how the neuron perceive the temporal
difference of the inputs from the current and the next place filed? To apply
biological analysis, we should address these issues in the SR model.

##  4 Toward biological model

Figure 1: Schematic figure for CA1 place cells and its presynaptic inputs.

Given that the $M_{t}(s_{t},s^{\prime})$ reflects the activity of place cells,
we can suppose it as a composition of synaptic weight and presynaptic neural
activity as the following equation.

$M_{t}(s_{t},s^{\prime})=W_{t}\cdot Pre(s_{t})$ (5)

where $W_{t}$ and $Pre(s_{t})$ is a vector of synaptic weights and of
presynaptic activities at the time step, respectively. The presynaptic
activities mean sensory inputs from other neural layers or brain regions;
$Pre(s_{t})$ acting like $\mathbb{I}(s_{t}=s^{\prime})$ in equation (3)
estimates the occupancies of the individual states [28]. Thus, we can presume
that the inner product of $W_{t}$ and $Pre(s_{t})$ shows the estimated values
related to the transitional probability, which is equivalent to the place cell
activities. For the convenience of the discussion, we omitted the activation
function of the artificial neural function approximation.

To derive the synaptic update rule for biological neuron from equation (4) of
the SR model, we can replace the $M(s,s^{\prime})$ with $W\cdot Pre(s)$ as
following.

$W_{t+1}\cdot Pre(s_{t})=W_{t}\cdot Pre(s_{t})+\eta[Pre(s_{t})+\gamma
W_{t}\cdot Pre(s_{t+1})-W_{t}\cdot Pre(s_{t})]$ (6)

Note that we still should deal with the presynaptic activities occurring at
$S_{t}$ and $S_{t+1}$ simultaneously: for animals, $S_{t}$ is the place field
just before, and $S_{t+1}$ is the place filed just observed. To resolve this
issue biologically, we can hypothesize that two distinct presynaptic neural
layer convey information of states, $S_{t}$ and $S_{t+1}$, simultaneously. Two
major presynaptic inputs to the CA1 place cells are Schaffer collateral inputs
from the CA3 and temporal-ammonic inputs from the entorhinal cortex
(EC)(Figure 1). Thus, we can rewrite equation (6) as follows.

$\begin{split}W_{t+1}\cdot Pre^{CA3}(s_{t})=&W_{t}\cdot Pre^{CA3}(s_{t})+\\\
&\eta[Pre^{CA3}(s_{t})+\gamma W^{EC}\cdot Pre^{EC}(s_{t+1})-W_{t}\cdot
Pre^{CA3}(s_{t})]\end{split}$ (7)

where $W_{t}$ and $W^{EC}$ are synaptic weights for CA3 inputs and for EC
inputs, respectively. $Pre^{CA3}$ and $Pre^{EC}$ are presynaptic activities of
the CA3 and of the EC, respectively. Here, it is assumed that CA3 and EC
encode information of $S_{t}$ and $S_{t+1}$, respectively, considering the
delay caused by the hippocampal trisynaptic circuit [6, 13].

The change of CA1 firing response for the place field $S_{t}$ occurs by
updating the synaptic weights from CA3 inputs. Based on equation (7), we can
derive the synaptic weight update rule as following.

$\begin{split}\Delta W\cdot Pre^{CA3}(s_{t})&=\eta[Pre^{CA3}(s_{t})+\gamma
W^{EC}\cdot Pre^{EC}(s_{t+1})-W_{t}\cdot Pre^{CA3}(s_{t})]\\\ \Delta
W&=\eta[\gamma W^{EC}\cdot Pre^{EC}(s_{t+1})\cdot
inv(Pre^{CA3}(s_{t}))-W_{t}+1]\end{split}$ (8)

where $inv(Pre^{CA3})$ is the Samelson inverse of $Pre^{CA3}$ vector.

In our derived update rule, $\Delta W$ depends on the dot product of
$Pre^{EC}$ and $Pre^{CA3}$. Biologically, this implies that simultaneous
activation of the CA3 and the EC enhances the synaptic weight from CA3 input
to the CA1 place cells. Modulation of the synapse by cooperative activity of
two distinct presynaptic inputs is called the heterosynaptic plasticity.
Deriving equation (8) from the SR model implies that the heterosynaptic
plasticity occurring at the CA1 place cells is the candidate for the
biological implementation of the predictive map theory, the SR learning model.

##  5 Discussion

In recent years, the convergences of the field of AI and neuroscience have
brought new ideas to each other, which leads to interesting results. The TD
learning idea gave new insights that interpret the CA1 recording results of a
moving animal in spatial navigation task [25]. In the study, the authors
argued that the recorded pattern of CA1 place cells is comparable to the
transitional probability matrix of the SR model, and proposed the predictive
map theory of the hippocampus. In this article, we integrated the SR model
into the hippocampal anatomical architecture, and derived heterogeneous
synaptic plasticity rules from the SR model.

To support the predictive map theory, we assumed that the CA1 place cells
require two distinct presynaptic inputs to update their firing rate with the
heterosynaptic plasticity. The input-timing dependent plasticity (ITDP) is a
good candidate for the biological implementation of synaptic weight
modification for predictive coding [6]. They found that preceded stimulus from
the EC before stimulus from the CA3 potentiated the synaptic strength between
CA3 and CA1 pyramidal neurons. When the stimuli came in the opposite order,
they observed no change in both synaptic strength. Thus, they called this
phenomenon as the ITDP. Based on the simulation, they presumed that the
candidate mechanism for the ITDP is the potentiated $\text{Ca}^{2+}$
transients of proximal dendrite elicited by stimuli to distal dendrite from
EC. The opposite order of stimuli did not sufficiently potentiate the
$\text{Ca}^{2+}$ transients of the proximal dendrite. The propagating
excitatory post-synaptic potential (EPSP) from distal dendrite was enough slow
to be summated with the elicited EPSP at proximal dendrite where receive
inputs from CA3. The summated EPSP opens the NMDA receptor, which is important
for the influx of $\text{Ca}^{2+}$ starting the intracellular signal cascades
for synaptic potentiation.

Conjunctive activation of EC and CA3 input leads not only to eliciting
synaptic plasticity but also to forming a place field of moving rodent. Prior
to forming new place fields, preceding dendritic plateau potential was
observed in vivo experiment [1]. This preceding dendritic plateau potential
required an interaction between the EC and CA3 inputs at the proper time
interval and sequence. Similar to the ITDP, the dendritic potential elicited
$\text{Ca}^{2+}$ transients that induced potentiation of EPSP from CA3. Since
the time window width of this asymmetric synaptic potentiation was a few
seconds, they named the synaptic rule as behavioral time scale synaptic
plasticity [2]. This biological mechanism reported in CA1 pyramidal neurons is
comparable to our heterosynaptic plasticity rule derived from the TD
algorithm: to the best of my knowledge, this is a novel finding.

Based on the anatomy of the hippocampus, the major inputs to CA1 place cells
can be divided into intrinsic input and extrinsic input. The extrinsic input
from outside of the hippocampus conveys sensory information from multiple
sensory cortical areas. Among them, grid cells in the EC project their axons
to the distal dendrites of CA1 place cells via the temporal-ammonic pathway
[7, 8]. The grid cells are characterized by an increase in firing rate as an
animal moves a certain distance in the environment, forming a hexagonal grid
field. The hexagonal grid field acts similarly to the latitude and longitude
of the map, providing an allocentric location for moving animals [3]. Thus,
the extrinsic input from the EC might provide the immediate location of the
agent: it would provide information for $M_{t}(s_{t+1},s^{\prime})$ in the SR
model.

Another component of the SR model, $M_{t}(s_{t},s^{\prime})$, can correspond
to information from CA3 coming along the Schaffer collateral pathway, the
intrinsic input to CA1 place cells. The CA3 receive signals simultaneously
from the EC and the dentate gyrus where receive signals via perforant pathway
from the EC. Together the mossy fiber input and EC input, the recurrent
connection of CA3 is the key feature of regarding CA3 as an attractor network
[20]. The attractor network based on the SR model has been reported to
resemble preplay and rapid path planning, which are key features of the
hippocampus [4]. Based on these anatomical features, there have been
suggestions that CA1 would be a subregion for predictive coding that compares
the outputs of CA3 with sensory information from the EC [13].

In terms of reward learning, however, both original SR model and our model did
not resolve where is the region of brain composing state transitions and
reward functions together, and how the composition occurs. One of the
projection targets of the hippocampus is the ventral striatum, where is
believed to be forming value functions by integrating information from other
brain regions [10, 12, 19]. In relation to value functions, the orbitofrontal
cortex is one region where reward-based decision-making occurs [21]. The
hippocampal-prefrontal replay has been reported to be important for spatial
learning and memory-based decision making [23]. Therefore, further research is
needed on how each region, including hippocampus, ventral striatum, and
orbitofrontal cortex, interacts and learns the value of a given environmental
state [9].

We have discussed integrating the biological findings and algorithmic level to
solve the spatial navigation problem. The functions of the entorhinal cortex
and hippocampus are not limited to representations of places, but are diverse
in concept representation, semantic memory, and episodic memory [3]. The
discovery of these functions in the same anatomical structure suggests that
they employ similar algorithms. We can expect further research to apply the
biological implementation of the SR model to these cognitive functions [16].

##  6 Credit author statement

Hyunsu Lee: Conceptualiztion, Methodology, Writing, Funding acquisition

##  7 Acknowledgements

This study was supported by the National Research Foundation of Korea(NRF)
grant funded by the Korea government(MSIT; Ministry of Science and ICT)(No.
NRF-2017R1C1B507279).

## References

  * Bittner et al. [2015] K. C. Bittner, C. Grienberger, S. P. Vaidya, A. D. Milstein, J. J. Macklin, J. Suh, S. Tonegawa, and J. C. Magee.  Conjunctive input processing drives feature selectivity in hippocampal ca1 neurons.  _Nat. Neurosci._ , 18:1133, 2015.  doi: 10.1038/nn.4062.
  * Bittner et al. [2017] K. C. Bittner, A. D. Milstein, C. Grienberger, S. Romani, and J. C. Magee.  Behavioral time scale synaptic plasticity underlies ca1 place fields.  _Science_ , 357:1033–1036, 2017.  doi: 10.1126/science.aan3846.
  * Buzsáki and Moser [2013] G. Buzsáki and E. I. Moser.  Memory, navigation and theta rhythm in the hippocampal-entorhinal system.  _Nat. Neurosci._ , 16:130–138, 2013.  doi: 10.1038/nn.3304.
  * Corneil and Gerstner [2015] D. S. Corneil and W. Gerstner.  Attractor network dynamics enable preplay and rapid path planning in maze–like environments.  In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, _Advances in Neural Information Processing Systems 28_ , pages 1684–1692. Curran Associates, Inc., 2015.
  * Dayan [1993] P. Dayan.  Improving generalization for temporal difference learning: The successor representation.  _Neural Computation_ , 5:613–624, 1993.  doi: 10.1162/neco.1993.5.4.613.
  * Dudman et al. [2007] J. T. Dudman, D. Tsay, and S. A. Siegelbaum.  A role for synaptic inputs at distal dendrites: instructive signals for hippocampal long-term plasticity.  _Neuron_ , 56:866–879, 2007.  doi: 10.1016/j.neuron.2007.10.020.
  * Fyhn et al. [2004] M. Fyhn, S. Molden, M. P. Witter, E. I. Moser, and M.-B. Moser.  Spatial representation in the entorhinal cortex.  _Science_ , 305:1258–1264, 2004.  doi: 10.1126/science.1099901.
  * Hafting et al. [2005] T. Hafting, M. Fyhn, S. Molden, M.-B. Moser, and E. I. Moser.  Microstructure of a spatial map in the entorhinal cortex.  _Nature_ , 436:801–806, 2005.  doi: 10.1038/nature03721.
  * Hirokawa et al. [2019] J. Hirokawa, A. Vaughan, P. Masset, T. Ott, and A. Kepecs.  Frontal cortex neuron types categorically encode single decision variables.  _Nature_ , 576:446–451, 2019.  doi: 10.1038/s41586-019-1816-9.
  * Johnson et al. [2007] A. Johnson, M. A. van der Meer, and A. D. Redish.  Integrating hippocampus and striatum in decision-making.  _Curr. Opin. Neurobiol._ , 17:692–697, 2007.  doi: doi.org/10.1016/j.conb.2008.01.003.
  * Kumaran et al. [2016] D. Kumaran, D. Hassabis, and J. L. Mcclelland.  What learning systems do intelligent agents need? complementary learning systems theory updated.  _Trends Cog. Sci._ , 20:512–534, 2016.  doi: 10.1016/j.tics.2016.05.004.
  * Lansink et al. [2009] C. S. Lansink, P. M. Goltstein, J. V. Lankelma, B. L. McNaughton, and C. M. Pennartz.  Hippocampus leads ventral striatum in replay of place-reward information.  _PLoS Biol._ , 7:e1000173, 2009.  doi: doi.org/10.1371/journal.pbio.1000173.
  * Lisman [1999] J. E. Lisman.  Relating hippocampal circuitry to function: recall of memory sequences by reciprocal dentate–ca3 interactions.  _Neuron_ , 22:233–242, 1999.  doi: 10.1016/S0896-6273(00)81085-5.
  * McClelland et al. [1995] J. L. McClelland, B. L. McNaughton, and R. C. O’Reilly.  Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory.  _Psychological review_ , 102:419–457, 1995.  doi: 10.1037/0033-295X.102.3.419.
  * Milner et al. [1968] B. Milner, S. Corkin, and H.-L. Teuber.  Further analysis of the hippocampal amnesic syndrome: 14-year follow-up study of hm.  _Neuropsychologia_ , 6:215–234, 1968.  doi: 10.1016/0028-3932(68)90021-3.
  * Momennejad et al. [2017] I. Momennejad, E. M. Russek, J. H. Cheong, M. M. Botvinick, N. D. Daw, and S. J. Gershman.  The successor representation in human reinforcement learning.  _Nature Human Behaviour_ , 1:680–692, 2017.  doi: 10.1038/s41562-017-0180-8.
  * O’Doherty et al. [2003] J. P. O’Doherty, P. Dayan, K. Friston, H. Critchley, and R. J. Dolan.  Temporal difference models and reward-related learning in the human brain.  _Neuron_ , 38:329–337, 2003.  doi: doi.org/10.1016/S0896-6273(03)00169-7.
  * O’Keefe and Dostrovsky [1971] J. O’Keefe and J. Dostrovsky.  The hippocampus as a spatial map. preliminary evidence from unit activity in the freely-moving rat.  _Brain Res._ , 34:171–175, 1971.  doi: 10.1016/0006-8993(71)90358-1.
  * Pennartz et al. [2011] C. Pennartz, R. Ito, P. Verschure, F. Battaglia, and T. Robbins.  The hippocampal–striatal axis in learning, prediction and goal-directed behavior.  _Trends Neurosci._ , 34:548–559, 2011.  doi: 10.1016/j.tins.2011.08.001.
  * Rebola et al. [2017] N. Rebola, M. Carta, and C. Mulle.  Operation and plasticity of hippocampal ca3 circuits: implications for memory encoding.  _Nature Publishing Group_ , 18:209–221, 2017.  doi: 10.1038/nrn.2017.10.
  * Rushworth et al. [2011] M. F. Rushworth, M. P. Noonan, E. D. Boorman, M. E. Walton, and T. E. Behrens.  Frontal cortex and reward-guided learning and decision-making.  _Neuron_ , 70:1054–1069, 2011.  doi: 10.1016/j.neuron.2011.05.014.
  * Schultz [1998] W. Schultz.  Predictive reward signal of dopamine neurons.  _J. Neurophysiol._ , 80:1–27, 1998.  doi: doi.org/10.1152/jn.1998.80.1.1.
  * Shin et al. [2019] J. D. Shin, W. Tang, and S. P. Jadhav.  Dynamics of awake hippocampal-prefrontal replay for spatial learning and memory-guided decision making.  _Neuron_ , 104:1110–1125. e7, 2019.  doi: doi.org/10.1016/j.neuron.2019.09.012.
  * Silver et al. [2017] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, G. van den Driessche, T. Graepel, and D. Hassabis.  Mastering the game of go without human knowledge.  _Nature_ , 550:354–359, 2017.  doi: 10.1038/nature24270.
  * Stachenfeld et al. [2017] K. L. Stachenfeld, M. M. Botvinick, and S. J. Gershman.  The hippocampus as a predictive map.  _Nat. Neurosci._ , 7:1951, 2017.  doi: 10.1038/nn.4650.
  * Suri [2002] R. E. Suri.  Td models of reward predictive responses in dopamine neurons.  _Neural networks_ , 15:523–533, 2002.  doi: doi.org/10.1016/S0893-6080(02)00046-1.
  * Sutton and Barto [2018] R. S. Sutton and A. G. Barto.  1.7. early history of reinforcement learning.  In _Reinforcement Learning_ , pages 13–22. MIT Press, 2018.
  * Vertes and Sahani [2019] E. Vertes and M. Sahani.  A neurally plausible model learns successor representations in partially observable environments, 2019.
  * Vinyals et al. [2019] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, and P. Georgiev.  Grandmaster level in starcraft ii using multi-agent reinforcement learning.  _Nature_ , 575:350–354, 2019.  doi: 10.1038/s41586-019-1724-z.