marginparsep has been altered.
topmargin has been altered.
marginparwidth has been altered.
marginparpush has been altered.
The page layout violates the ICML style. Please do not change the page layout,
or include packages like geometry, savetrees, or fullpage, which change it for
you. We’re not able to reliably undo arbitrary changes to the style. Please
remove the offending package(s), or layout-changing commands and try again.

Zero-Episode Few-Shot Contrastive Predictive Coding:
Solving intelligence tests without prior training

Anonymous Authors1

††footnotetext:  1Anonymous Institution, Anonymous City, Anonymous Region,
Anonymous Country. Correspondence to: Anonymous Author
<EMAIL_ADDRESS>
Preliminary work. Under review by the International Conference on Machine
Learning (ICML). Do not distribute.

# Supplementary Materials

###### Abstract

Video prediction models often combine three components: an encoder from pixel
space to a small latent space, a latent space prediction model, and a
generative model back to pixel space. However, the large and unpredictable
pixel space makes training such models difficult, requiring many training
examples. We argue that finding a predictive latent variable and using it to
evaluate the consistency of a future image enables data-efficient predictions
because it precludes the necessity of a generative model training. To
demonstrate it, we created sequence completion intelligence tests in which the
task is to identify a predictably-changing feature in a sequence of images and
use this prediction to select the subsequent image. We show that a one-
dimensional Markov Contrastive Predictive Coding (M-CPC${}_{\text{1D}}$) model
solves these tests efficiently, with only five examples. Finally, we
demonstrate the usefulness of M-CPC${}_{\text{1D}}$ in solving two tasks
without prior training: anomaly detection and stochastic movement video
prediction.

##  1 Introduction

Figure 1: A sequential intelligence test. The images along the sequence become
darker. The rest of their characterizing features are random. Only the 4th
choice adheres to the shading rule.

Changes in the world are often dominated by the dynamics of a relatively small
number of latent variables. Identifying these variables is useful for making
predictions. For example, in video prediction, the task is to predict an image
from a sequence of its preceding images. To that goal, video prediction models
often assume a small number of latent variables and learn to predict them Liu
et al. (2021). However, the learning of the mapping of these latent variables
to the pixel space requires the training of a generative model, which requires
a large number of examples. The number of examples can be somewhat reduced if
the expressivity of the latent encoder is limited Kumar et al. (2019), or if a
simple structure is imposed on the latent variables Minderer et al. (2019);
Kim et al. (2019); Yang et al. (2018).

Our focus here is on a different class of problems, in which the task is to
learn a predictive latent model only and use it to evaluate the _consistency_
of a given image with its preceding images. Solving this problem enables the
identification of incongruent images, or as we focus in this paper, the
_selection_ of the predictable or congruent image from a set of alternative
choices, rather than creating it. We argue that because no generative model
training is needed, this problem can be solved using only a small number of
examples. In humans, the ability to learn predictive latent variables and use
them to select congruent predictions is quantified with intelligence tests,
whose score is highly correlated with success in the job market and the
academia Sternberg (1977); Raven et al. (1998); Lohman (2000); Kaplan &
Saccuzzo (2009); Siebers et al. (2015). Therefore, we use the framework of
intelligence tests to demonstrate the data-efficiency of a latent-guided
prediction by selection.

Consider the intelligence test depicted in Fig. 1: five ordered images are
presented. The next image, the sixth, is missing. Each image is characterized
by features: the number of objects, their color, shape, size, and positions.
The images were constructed such that one of the features predictably changes
along the sequence according to a simple deterministic rule, while the rest of
the features are either constant or randomly changing. Because of the random
nature of some of the features, predicting the sixth image exactly is
impossible. However remarkably, humans are able to identify the image which is
congruent with the sequence, out of the four alternative choices depicted in
Fig. 1, by finding the predictably changing feature without prior training.

In this paper, we use a Contrastive Predictive Coding (CPC) algorithm Oord et
al. (2018) which was shown useful for finding predictive latent variables
Anand et al. (2019); Henaff (2020); Yan et al. (2020). This self-supervised
algorithm optimizes an infoNCE loss in which consecutive inputs in a sequence
are regarded as positive examples and non-consecutive inputs as negative
examples. Usually, to get accurate data representations, CPC models are
trained over large datasets. However, we hypothesized that the selection task
required for solving intelligence tests such as the one depicted in Fig. 1 are
solvable by inaccurate data representations and therefore that a CPC algorithm
can solve them without _any_ prior training. This is an extreme example of a
few-shot learning in which the training is done using only the five images of
the test, with zero training episodes.

##  2 Intelligence Tests

Inspired by previous intelligence tests generating algorithms Wang & Su
(2015); Barrett et al. (2018), sequences of $K$ gray-scale images
$\mathbf{x}_{j}$ (e.g in Fig. 2) were generated in the following way: each
image included 1-9 identical objects arranged on a 3$\times$3 grid. An image
was characterized by a low dimensional vector of features, $\mathbf{f}_{j}$
where $f^{i}_{j}$ denotes the value of feature $i$ in image $j$. We used the
following five features: the number of objects in an image (possible values: 1
to 9), their shade (6 linearly distributed gray scale values), the shapes
(circle, triangle, square, star, hexagon), their size (6 linearly distributed
values for the shapes’ enclosing circle circumference), and positions (a
vector of grid positions that was used to place the shapes in order). The
image $\mathbf{x}_{j}$ was constructed according to its characterizing
features by a non-linear and complex generative function
$\mathbf{x}_{j}=g\left(\mathbf{f}_{j}\right)$111Code for generating
intelligence tests available on request..

Figure 2: Sequences with various predictive, constant and random features. We
call the random features distractors, and their number determines an
intelligence test’s difficulty.

One of the features $f^{p}$ predictably changed along the sequence according
to a simple deterministic rule $f_{j+1}^{p}=u(f_{j}^{p})$ while the other
features were either constant over the images or changed randomly (values were
i.i.d). We refer to the randomly-changing features as _distractors_ and their
number is considered a measure of the difficulty of the test. After observing
a sequence of $K$ images, the agent’s task was to select the correct
$K+1^{\text{th}}$ image from a set of $n$ optional choice images that were
generated using the same generative function $g$ from the feature space. In
the correct choice, $f^{p}$ followed the deterministic rule
$f_{K+1}^{p}=u(f_{K}^{p})$, whereas in the incorrect choices it did not follow
that rule and was instead randomly chosen from the remaining possible values.
The features that were constant or randomly changing in the sequence were also
constant or changed randomly in all four choices.

##  3 Markov 1D Contrastive Predictive Coding

Our main challenge in solving the intelligence tests is to find a latent
variable that changes in a simple deterministic way along the test’s five
images sequence. Consider an encoder function $Z$, a predictor function $T$,
two images $\mathbf{x}_{a}$ and $\mathbf{x}_{b}$ and the prediction error

$\epsilon_{a,b}\left(Z,T\right)=\bigg{(}T\big{(}Z(\mathbf{x}_{a})\big{)}-Z(\mathbf{x}_{b})\bigg{)}^{2}$
(1)

By construction, for the true encoder and predictor functions $Z^{*}=g^{-1}$
and $T^{*}=u$, $\epsilon_{a,b}\left(Z^{*},T^{*}\right)=0$ if $a$ and $b$ are
two consecutive images ($b=a+1$), and
$\epsilon_{a,b}\left(Z^{*},T^{*}\right)\neq 0$ otherwise.

The challenge is that $Z^{*}$ and $T^{*}$ are unknown. However, given a
sequence of $K$ ordered images, we can approximate $Z^{*}$ and $T^{*}$ by
finding $Z$ and $T$ that minimize the prediction error for consecutive images
and maximize it for the non-consecutive ones. Formally, we define a
contrastive infoNCE loss based on those prediction errors

$\mathcal{L}=-\frac{1}{K-1}\sum_{j=1}^{K-1}\log{\frac{e^{-\epsilon_{j,j+1}}}{\sum_{j^{\prime}}{e^{-\epsilon_{j,j^{\prime}}}}}}$
(2)

and find $Z$ and $T$ that minimize it.

The model’s encoder $Z$ and predictor $T$ are implemented by deep neural
networks. Because the true predictor is one dimensional, we used a
convolutional network for the encoder $Z(\mathbf{x})$ from the 100$\times$100
pixel space to a single neuron; For the predictor, $T(Z(\mathbf{x}))$, we used
a residual network $T(Z(\mathbf{x}))=Z(\mathbf{x})+\Delta T(Z(\mathbf{x}))$
where $\Delta T$ is a fully-connected network. This variant of the CPC
algorithm predicts a 1D latent variable based only on its most recent value.
Thus, we marked it as M-CPC${}_{\text{1D}}$.

##  4 Results

###  4.1 Solving Tests Without Prior Training

To quantify the performance of M-CPC${}_{\text{1D}}$ in solving intelligence
tests without prior training, we applied it on a multitude of intelligence
tests. For brevity, we focused in the main text on a specific predictive
feature, the size of the objects, which increased monotonically throughout.
Tests in which other features changed predictably are shown in the
supplementary material. The 4 remaining features were either constant or
randomly changing, resulting in a total of $2^{4}=16$ test conditions (Fig.
3).

Each intelligence test was solved in the following way: First, we randomly
initialized the networks corresponding to $Z$ and $T$. We then updated these
networks’ weights with a single optimization step in the direction of
minimizing the loss function (Eq. 2)222We found that a single gradient step
achieved comparable results to a full minimization of the loss with more
steps. We used the RMSprop optimizer with learning rate $\eta=4\cdot
10^{-4}$.. After the optimization step, we selected the choice image that had
the lowest prediction error out of the four choices as the answer of this
intelligence test.

Figure 3: Performance Without Prior Training. Accuracy in 16 test conditions
in which the predictive feature was the objects size that increased along the
sequences. The remaining 4 features were either distractors (marked according
to the legend) or constant (not marked). Performance was evaluated using $500$
randomly-generated intelligence tests (see section 2 for details). Error bars
correspond to 95% confidence intervals. The black line and its shade are the
average accuracy per difficulty and the corresponding standard deviation.

Remarkably, we found that training with only the $K=5$ images that are given
within the tests is sufficient for solving easy tests, as well as for
achieving a substantially higher than chance performance in the more difficult
tests (Fig. 3).

###  4.2 Meta-Learning Sample Complexity

To evaluate the potential benefit of prior training and obtain the sample
complexity of M-CPC${}_{\text{1D}}$, we tested the performance of models whose
parameters are learned, rather than chosen randomly. Specifically, we
performed an episode-based meta-learning, in which prior to performing a
certain few-shot learning task, a model is trained on episodes (one
optimization step per episode) that are random realizations of the same few-
shot learning task Thrun & Pratt (1998); Sung et al. (2018). This was done for
the 16 test conditions of Fig. 3.

Figure 4: Meta-Learning Sample Complexity. For each of the 16 test conditions
in Fig. 3, we trained the model using 1000 training episodes. Performance was
evaluated every 50 training episodes. Accuracy measure corresponds to the
average performance of 50 randomly-initialized models, each evaluated using
500 intelligence tests that were randomly generated from the same test
condition. For clarity, performance is also averaged over test conditions that
share the same number of distractors (thus, for example, 0 distractors
correspond to a single test condition whereas 2 distractors depict the average
over 6 test conditions).

As depicted in Fig. 4, meta-learning improved the performance of
M-CPC${}_{\text{1D}}$ to above 85% accuracy in all test conditions within
several hundreds of training episodes.

###  4.3 Cross-Domain Generalization

Meta-learning algorithms often overfit to the tasks they were trained on,
impairing their cross-domain generalization Li et al. (2017); Yin et al.
(2020); Rajendran et al. (2020). To evaluate the cross-domain generalization
properties of M-CPC${}_{\text{1D}}$ we extensively trained (1000 episodes)
networks on intelligence tests with certain feature rules, and then tested
them with intelligence tests that were characterized by other feature rules.
Specifically, we trained and tested the networks using intelligence tests in
which the predictive feature was either the size or the color of the objects
(increasing monotonically), and the rest of the features could either be all
constant (easy condition), or all distracting (hard condition). In total, we
considered 4 types of intelligence tests: size-easy, size-hard, color-easy,
color-hard; and we crossed between them in the training and final-evaluation
stages of the model.

Figure 5: Cross-Domain Generalization. Randomly initiated networks $Z$ and $T$
were trained for 1000 episodes on a certain intelligence test type, and then
evaluated over the 4 intelligence test types with 500 tests each. The right
column presents the performance without prior training for comparison.
Accuracies presented are averaged over 50 experiments and shown with 95%
confidence intervals.

The emerging picture is interesting (Fig. 5). Within the same predictive
feature, we find that training using easy episodes is more effective than
training with hard ones. Interestingly, after training with the easy episodes,
networks’ performance in the hard tests was comparable to their performance in
the easy ones. These results suggest that the difference in the asymptote
performances in Fig. 4 in the different conditions may reflects differences in
the training episodes rather than differences in the difficulty of the tests.
In other words, after training, the networks can solve even the most difficult
tests if these tests are preceded by training using easy episodes.

When considering training and testing on different predictive features, the
picture is more complex. Training with easy episodes of one feature rule
improved performance in hard tests of the other feature rule, but was
detrimental to performance when the tests of the other rule were easy.
Training with hard episodes, on the other hand, was catastrophic, bringing
performance in both easy and hard tests of the other predictive feature to
chance levels.

###  4.4 Conclusion

Our main result is that M-CPC${}_{\text{1D}}$ can successfully solve
intelligence tests without any prior training. Moreover, training the model by
meta-learning can either improve or impair the performance of the model,
depending on the feature alignment between the training and testing domains.
These results indicate that zero-episode few-shot training can outperform
trained models in environments in which domain shifts are expected,
demonstrating the potential benefit of few-shot learning without any prior
training.

##  5 Applications

###  5.1 Stochastic Movement Prediction

To demonstrate the usefulness of M-CPC${}_{\text{1D}}$, we applied it to a
video prediction task. The video we chose to predict is similar to those of
the Stochastic Movement (SM) dataset Babaeizadeh et al. (2017), in which a
random shape moves in a random direction from the middle of the frame to one
of its sides. It is difficult to train effective deterministic video
prediction models in stochastic environments such as the SM. Therefore,
stochastic video prediction models have been used to make predictions in such
settings Babaeizadeh et al. (2017); Kumar et al. (2019). Our approach, by
contrast, was to use the data-efficiency of M-CPC${}_{\text{1D}}$ to predict a
video with only the first five frames of that video, without training on other
stochastic videos.

Figure 6: Stochastic Movement as Intelligence Tests. A video of a 10x10 pixel
square moving upwards is predicted by treating it as an “intelligence test”.
Five consecutive frames of the video are used to predict the sixth frame,
whose identity is chosen out of 4 images that are shifted by one pixel in one
of the four directions relative to the last frame.

Without loss of generality, we evaluated the video prediction ability of
M-CPC${}_{\text{1D}}$ with one video in which a square moved upwards (Fig. 6).
We optimized M-CPC${}_{\text{1D}}$ on the first $K=5$ frames
$\\{\mathbf{x}_{i}\\}_{i=1}^{K}$ of the video and used the latent variable to
select the sixth frame $\mathbf{\tilde{x}}_{K+1}$ out of four possible motion
directions, similar to the way we solved the intelligence tests. We found that
based on five frames, the model can correctly predict the $K+1$ frame with a
probability of $97\%\pm 1\%$.

To predict the next frame ($K+2$), we used the last $K-1$ frames of the
original video and the predicted frame $\mathbf{\tilde{x}}_{K+1}$ to create a
new sequence of length $K$
($\\{\mathbf{x}_{i}\\}_{i=2}^{K}\cup\\{\mathbf{\tilde{x}}_{K+1}\\}$). We then
trained new, randomly-initialized networks $Z$ and $T$, on the new sequence
and used it to predict frame $K+2$, and so on. We iterated this process for 45
frames and the results are depicted in Fig. 7.

Figure 7: Predicted Videos. 500 videos obtained by video prediction with
M-CPC${}_{\text{1D}}$, conditioned on the $5$ initial example frames. A
pixel’s shade corresponds to the ratio of predicted videos, in logarithmic
scale, that visited that pixel out of the 500 videos.

As demonstrated in Fig. 7, the first $K=5$ frames and the iterative process
are enough for M-CPC${}_{\text{1D}}$ to predict the video correctly with a
high probability.

###  5.2 Anomaly Detection

To solve intelligence tests, we used a predictive latent variable to _select_
a congruent image from a set of alternative choice images. Predictive latent
variables can also be used for anomaly detection in tasks that do not entail a
selection between alternatives. Specifically, given a sequence of images, the
task is to determine whether the last image is congruent or incongruent with
its preceding images.

Consider the two sequences depicted in Fig. 8. The task is to determine,
without prior training, that there is no anomaly in the top sequence of images
(tile 6 is congruent with its preceding tiles) while there is anomaly in the
bottom sequence (tile 6 is incongruent with its preceding tiles). To classify
the congruency of a candidate image $\mathbf{x}_{c}$ with a given sequence, we
performed a single optimization step on the five sequence images with our loss
function (Eq. 2). We then compared the prediction error of the candidate image
$\epsilon_{K,c}$ to the average of the prediction errors of the sequence’s
consecutive images:

$\epsilon_{th}=\frac{1}{K-1}\sum_{j=1}^{K-1}\epsilon_{j,j+1}$ (3)

We use a threshold parameter $\alpha$ to classify candidate images: when
$\epsilon_{K,c}>\alpha\cdot\epsilon_{th}$ we classify the candidate image as
anomalous; when $\epsilon_{K,c}<\alpha\cdot\epsilon_{th}$ we classify it as
congruent.

Figure 8: Anomaly Detection Tests. We provide our model a sequences of five
images. Additionally, we generated two candidate images: one congruent and one
incongruent with the sequence. The task of the model is to classify whether a
candidate image is congruent or not, based only on the sequence, without
comparing the two candidates.

Image sequences of length $K=5$ were created such that the size, number and
shade of the objects in the images increased monotonically along the
sequences, while the objects’ shape and the order of grid placement were
constants. Two images were candidates for anomaly in each sequence: a
congruent image with features that changed according to the sequence rules,
and an incongruent image in which the size, number and color of the shapes did
not follow the sequence rules. The shape and order of grid placement was also
constant in the incongruent image. To evaluate the anomaly detection
performance, we generated 500 such tests. With this classification criterion,
we achieved a success rate of $85\%\pm 3\%$ (taking into account false
positive and misses results; chance accuracy is $50\%$). Remarkably, this
result is achieved without any prior training, using only the five images of
each sequence, and without relying on selection from alternatives.

##  6 Conclusion

We showed that an easy-to-train latent prediction model M-CPC${}_{\text{1D}}$
can successfully solve prediction tasks. Specifically, we used the predictive
latent variable to evaluate the consistency of an image with a sequence of
preceding images by training on those preceding sequence images alone. This
consistency evaluation allowed us to solve three tasks without prior training:
1) Intelligence tests, in which the task is to select the image that is most
consistent with its preceding images. 2) Video prediction in which we showed
that a small number of video frames, together with an iterative process, are
sufficient to make some predictions in a simple video. 3) And an anomaly
detection task. Common to all these examples is that reasonable performance is
achieved with an extremely small number of examples. This highlights the data
efficiency of latent predictions.

It is generally argued that one main difference between human and machine
learning is the amount of data required for the learning. For example, GPT-3
has been trained on hundreds of billions of words, more than three orders of
magnitude more words than humans use when learning to speak Hart & Risley
(1995). Our results demonstrate that also in machine learning, data efficiency
can go a long way.

##  7 Relation to Previous Works

Intelligence tests solvers \- Human intelligence is often measured using
intelligence tests that are similar to ours. Solving intelligence tests relies
on finding relevant features in the provided examples and the rules that
govern these features Blum & Blum (1975); Siebers et al. (2015). Traditional
computational models that solved intelligence tests utilized either prior
knowledge of the relevant features Rasmussen & Eliasmith (2011), knowledge
about the rules that govern these features Sun & Dai (2018), or both Carpenter
et al. (1990). Today, machine learning models are able to learn the relevant
features and rules using deep artificial neural networks Barrett et al.
(2018); Hill et al. (2019). However, they relied on both supervised learning
and large datasets, unlike humans that seem to be able to solve such tests
without prior training. Other models solved intelligence tests with
unsupervised learning Zhuo & Kankanhalli (2020) and by meta-learning Santoro
et al. (2017); Kim et al. (2020), but they also relied on extensive prior
training before solving the tests. Our work presents an ability to solve
intelligence tests without any prior training.

Contrastive learning \- Contrastive loss function are widely used for self-
supervised learning Chen et al. (2020); Le-Khac et al. (2020). One contrastive
algorithm, the CPC algorithm, Oord et al. (2018) is used for finding
predictive latent representations, which is useful for data-efficient image
recognition Henaff (2020) and learning world-models that supports robotic
object manipulation Yan et al. (2020) and playing Atari games Anand et al.
(2019). Compared with these works, which relied on extensive training, we
trained M-CPC${}_{\text{1D}}$ with only five images. This was possible because
we used a 1D latent variable feed-forward encoder rather than a higher-
dimensional recurrent network encoder.

Video prediction models \- The video prediction task is useful for various
down-stream applications such as representational learning and model-based
reinforcement learning. Therefore, many deep learning models were developed to
solve this task Oprea et al. (2020). State-of-the-art models utilize a latent
prediction models to solved the task Minderer et al. (2019); Kim et al.
(2019); Yang et al. (2018); Lee et al. (2021). These models also required
generative models, which needed training as well. A stochastic model for the
latent variable has been shown useful for video generation Kumar et al.
(2019); Franceschi et al. (2020); Babaeizadeh et al. (2017). However, as we
show here, a deterministic latent model is sufficient for the selection task
even in stochastic environments, which allows for better data efficiency.

Relation networks and meta-learning \- M-CPC${}_{\text{1D}}$ is similar to
Relation Network (RN) Sung et al. (2018), a model which utilized episode based
meta-learning for classifying examples by the abstract relation between them.
While RN can learn general abstract relations between inputs, our model is
inductively biased for learning a future-directed residual relation between
consecutive latent representations. As a result of this and the fact that the
task is a selection task, RN models require a large number of episodes for
training while the intelligence tests can be solved without any prior
training.

## References

  * Anand et al. (2019) Anand, A., Racah, E., Ozair, S., Bengio, Y., Côté, M.-A., and Hjelm, R. D.  Unsupervised State Representation Learning in Atari.  (NeurIPS), 2019.  URL http://arxiv.org/abs/1906.08226.
  * Babaeizadeh et al. (2017) Babaeizadeh, M., Finn, C., Erhan, D., Campbell, R. H., and Levine, S.  Stochastic Variational Video Prediction.  October 2017.  URL http://arxiv.org/abs/1710.11252.
  * Barrett et al. (2018) Barrett, D. G. T., Hill, F., Santoro, A., Morcos, A. S., and Lillicrap, T.  Measuring abstract reasoning in neural networks.  2018\.  ISSN 17740746.  doi: 10.1051/agro/2009059.  URL http://arxiv.org/abs/1807.04225.  ISBN: 1807.04225v1.
  * Blum & Blum (1975) Blum, L. and Blum, M.  Toward a mathematical theory of inductive inference.  _Information and Control_ , 28(2):125–155, 1975\.  ISSN 00199958.  doi: 10.1016/S0019-9958(75)90261-2.
  * Carpenter et al. (1990) Carpenter, P. A., Just, M. A., and Shell, P.  What One Intelligence Test Measures: A Theoretical Account of the Processing in the Raven Progressive Matrices Test.  (3):28, 1990.
  * Chen et al. (2020) Chen, T., Kornblith, S., Norouzi, M., and Hinton, G.  A Simple Framework for Contrastive Learning of Visual Representations.  In _Proceedings of the 37th International Conference on Machine Learning_ , pp. 1597–1607. PMLR, November 2020.  URL https://proceedings.mlr.press/v119/chen20j.html.  ISSN: 2640-3498.
  * Franceschi et al. (2020) Franceschi, J.-Y., Delasalles, E., Chen, M., Lamprier, S., and Gallinari, P.  Stochastic Latent Residual Video Prediction.  _arXiv:2002.09219 [cs, stat]_ , August 2020.  URL http://arxiv.org/abs/2002.09219.  arXiv: 2002.09219 version: 4.
  * Hart & Risley (1995) Hart, B. and Risley, T. R.  The early catastrophe: The 30 million word gap by age 3.  _American Educator_ , 27(1):4–9, 1995.
  * Henaff (2020) Henaff, O.  Data-Efficient Image Recognition with Contrastive Predictive Coding.  In _Proceedings of the 37th International Conference on Machine Learning_ , pp. 4182–4192. PMLR, November 2020.  URL https://proceedings.mlr.press/v119/henaff20a.html.  ISSN: 2640-3498.
  * Hill et al. (2019) Hill, F., Santoro, A., Barrett, D. G. T., Morcos, A. S., and Lillicrap, T.  Learning to Make Analogies by Contrasting Abstract Relational Structure.  2019\.  URL http://arxiv.org/abs/1902.00120.
  * Kaplan & Saccuzzo (2009) Kaplan, R. M. and Saccuzzo, D. P.  _Psychological Testing: Principles, Applications, and Issues_.  2009\.  ISBN 0-495-09555-9.
  * Kim et al. (2019) Kim, Y., Nam, S., Cho, I., and Kim, S. J.  Unsupervised Keypoint Learning for Guiding Class-Conditional Video Prediction.  October 2019.  URL https://arxiv.org/abs/1910.02027v1.
  * Kim et al. (2020) Kim, Y., Shin, J., Yang, E., and Hwang, S. J.  Few-shot Visual Reasoning with Meta-analogical Contrastive Learning.  _arXiv:2007.12020 [cs, stat]_ , July 2020.  URL http://arxiv.org/abs/2007.12020.  arXiv: 2007.12020.
  * Kumar et al. (2019) Kumar, M., Babaeizadeh, M., Erhan, D., Finn, C., Levine, S., Dinh, L., and Kingma, D.  VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation.  March 2019.  URL http://arxiv.org/abs/1903.01434.
  * Le-Khac et al. (2020) Le-Khac, P. H., Healy, G., and Smeaton, A. F.  Contrastive Representation Learning: A Framework and Review.  _IEEE Access_ , 8:193907–193934, 2020.  ISSN 2169-3536.  doi: 10.1109/ACCESS.2020.3031549.  Conference Name: IEEE Access.
  * Lee et al. (2021) Lee, W., Jung, W., Zhang, H., Chen, T., Koh, J. Y., Huang, T., Yoon, H., Lee, H., and Hong, S.  Revisiting Hierarchical Approach for Persistent Long-Term Video Prediction.  April 2021.  URL http://arxiv.org/abs/2104.06697.
  * Li et al. (2017) Li, D., Yang, Y., Song, Y.-Z., and Hospedales, T. M.  Learning to Generalize: Meta-Learning for Domain Generalization.  _arXiv:1710.03463 [cs]_ , October 2017.  URL http://arxiv.org/abs/1710.03463.  arXiv: 1710.03463.
  * Liu et al. (2021) Liu, B., Chen, Y., Liu, S., and Kim, H.-S.  Deep Learning in Latent Space for Video Prediction and Compression.  In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 701–710, June 2021.  doi: 10.1109/CVPR46437.2021.00076.  ISSN: 2575-7075.
  * Lohman (2000) Lohman, D. F.  Complex Information Processing and Intelligence.  In Sternberg, R. J. (ed.), _Handbook of Intelligence_ , pp. 285–340. Cambridge University Press, Cambridge, 2000.  ISBN 978-0-521-59648-0.  doi: 10.1017/CBO9780511807947.015.
  * Minderer et al. (2019) Minderer, M., Sun, C., Villegas, R., Cole, F., Murphy, K., and Lee, H.  Unsupervised Learning of Object Structure and Dynamics from Videos.  June 2019.  URL http://arxiv.org/abs/1906.07889.
  * Oord et al. (2018) Oord, A. v. d., Li, Y., and Vinyals, O.  Representation Learning with Contrastive Predictive Coding.  2018\.  URL http://arxiv.org/abs/1807.03748.
  * Oprea et al. (2020) Oprea, S., Martinez-Gonzalez, P., Garcia-Garcia, A., Castro-Vargas, J. A., Orts-Escolano, S., Garcia-Rodriguez, J., and Argyros, A.  A Review on Deep Learning Techniques for Video Prediction.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pp. 1–1, 2020.  ISSN 1939-3539.  doi: 10.1109/TPAMI.2020.3045007.  Conference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence.
  * Rajendran et al. (2020) Rajendran, J., Irpan, A., and Jang, E.  Meta-Learning Requires Meta-Augmentation.  In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H. (eds.), _Advances in Neural Information Processing Systems_ , volume 33, pp. 5705–5715. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper/2020/file/3e5190eeb51ebe6c5bbc54ee8950c548-Paper.pdf.
  * Rasmussen & Eliasmith (2011) Rasmussen, D. and Eliasmith, C.  A neural model of rule generation in inductive reasoning.  _Topics in Cognitive Science_ , 3(1):140–153, 2011.  ISSN 17568757.  doi: 10.1111/j.1756-8765.2010.01127.x.
  * Raven et al. (1998) Raven, J., Raven, J. C., and Court, J. H.  _Manual for Raven’s progressive matrices and vocabulary scales_.  Pearson, San Antonio, TX, 1998.  ISBN 978-0-15-868643-1 978-0-15-468622-0 978-0-15-468623-7 978-1-85639-022-4.  OCLC: 697438611.
  * Santoro et al. (2017) Santoro, A., Raposo, D., Barrett, D. G. T., Malinowski, M., Pascanu, R., Battaglia, P., and Lillicrap, T.  A simple neural network module for relational reasoning.  _Advances in Neural Information Processing Systems_ , 2017-Decem:4968–4977, 2017.  ISSN 10495258.
  * Siebers et al. (2015) Siebers, M., Dowe, D. L., Schmid, U., Hernández-Orallo, J., and Martínez-Plumed, F.  Computer models solving intelligence test problems: Progress and implications.  _Artificial Intelligence_ , 230:74–107, 2015.  ISSN 00043702.  doi: 10.1016/j.artint.2015.09.011.  URL http://dx.doi.org/10.1016/j.artint.2015.09.011.  ISBN: 9780999241103 Publisher: Elsevier B.V.
  * Sternberg (1977) Sternberg, R. J.  Component processes in analogical reasoning.  _Psychological Review_ , 84(4):353–378, 1977\.  ISSN 0033295X.  doi: 10.1037/0033-295X.84.4.353.
  * Sun & Dai (2018) Sun, R. and Dai, D. Y.  Deep Learning of Raven’s Matrices.  _Advances in Cognitive Systems_ , pp. 1–6, 2018.  URL http://www.cogsys.org/papers/ACS2017/ACS_2017_paper_23_Mekik.pdf.
  * Sung et al. (2018) Sung, F., Yang, Y., Zhang, L., Xiang, T., Torr, P. H. S., and Hospedales, T. M.  Learning to Compare: Relation Network for Few-Shot Learning.  pp. 1199–1208, 2018.  URL https://openaccess.thecvf.com/content_cvpr_2018/html/Sung_Learning_to_Compare_CVPR_2018_paper.html.
  * Thrun & Pratt (1998) Thrun, S. and Pratt, L.  Learning to Learn: Introduction and Overview.  In Thrun, S. and Pratt, L. (eds.), _Learning to Learn_ , pp. 3–17. Springer US, Boston, MA, 1998.  ISBN 978-1-4615-5529-2.  doi: 10.1007/978-1-4615-5529-2˙1.  URL https://doi.org/10.1007/978-1-4615-5529-2_1.
  * Wang & Su (2015) Wang, K. and Su, Z.  Automatic generation of Raven’s progressive Matrices.  _IJCAI International Joint Conference on Artificial Intelligence_ , 2015-Janua(Ijcai):903–909, 2015.  ISSN 10450823.  ISBN: 9781577357384.
  * Yan et al. (2020) Yan, W., Vangipuram, A., Abbeel, P., and Pinto, L.  Learning Predictive Representations for Deformable Objects Using Contrastive Estimation.  2020\.  URL http://arxiv.org/abs/2003.05436.
  * Yang et al. (2018) Yang, C., Wang, Z., Zhu, X., Huang, C., Shi, J., and Lin, D.  Pose Guided Human Video Generation.  July 2018.  URL https://arxiv.org/abs/1807.11152v1.
  * Yin et al. (2020) Yin, M., Tucker, G., Zhou, M., Levine, S., and Finn, C.  Meta-Learning without Memorization.  _arXiv:1912.03820 [cs, stat]_ , April 2020.  URL http://arxiv.org/abs/1912.03820.  arXiv: 1912.03820.
  * Zhuo & Kankanhalli (2020) Zhuo, T. and Kankanhalli, M.  Solving Raven’s Progressive Matrices with Neural Networks.  _arXiv:2002.01646 [cs]_ , February 2020.  URL http://arxiv.org/abs/2002.01646.  arXiv: 2002.01646.

##  S8 Results of other predictive features

Figure S9: Predictive Feature: color. Zero-episodes performance when the color
of the shapes became darker along the sequences.

Figure S10: Predictive Feature: number. Zero-episodes performance when the
number of shapes increased monotonically along the sequences.

Figure S11: Predictive Feature: shape. Zero-episodes performance when the
shapes alternated between a triangle and a square along the sequences.


# Study of Optical Networks, 5G, Artificial Intelligence and Their
Applications

Quanda Zhang  School of Telecommunication
Daqing Normal College  Qi Zhang  School of Telecommunication
Daqing Normal College

###### Abstract

This paper discusses the application of artificial intelligence (AI)
technology in optical communication networks and 5G. It primarily introduces
representative applications of AI technology and potential risks of AI
technology failure caused by the openness of optical communication networks,
and proposes some coping strategies, mainly including modeling AI systems
through modularization and miniaturization, combining with traditional
classical network modeling and planning methods, and improving the
effectiveness and interpretability of AI technology. At the same time, it
proposes response strategies based on network protection for the possible
failure and attack of AI technology.

###### Index Terms:

AI, 5G, Optical Networks

##  I Introduction

Artificial intelligence (AI, artificial intelligence) technology is very early
has been used in many fields, but for many years this technology has not
gained high attention until AlphaGo defeated Chinese and Korean Go players
After the hand, it began to become a research hotspot, and researchers tried
to AI technology is applied in different fields, including optical
communication network network. In the past two years, the United States
Optical Communications Conference (OFC, optical fiber communication) and the
European Conference of Optical Communications (ECOC, European conference of
optical communication), at least 16 conference topics focused on AI or machine
learning (ML, machine learning) technology. This paper combines AI technology
and ML technology AI technologies are regarded as the same class of
technologies, and at the same time, although AI technologies cover a wide
range, The AI technology referred to in this article is mainly neural network
technology.

AI technology has received widespread attention mainly due to the following
two reasons. First, AI technology is relatively easy to get started and use.
it comes in black Model the system in a box way, through a large number of
samples Learning, let the black box connect neurons by itself, and distribute
neurons Connect weights without requiring the user to understand why neurons
behave the way they do connections and are assigned current weights. Users
only need to provide enough learning samples, increase the number of neurons
and the number of hidden layers, It can improve the prediction accuracy of AI
technology. Second, AI technology is After the AlphaGo incident, it has almost
been deified, and almost everyone knows that ”people ”artificial
intelligence”, and in the academic circle, the paper labeled AI Papers also
seem to be easier to publish, so this also leads to a current phenomenon Like,
that is, for almost all problems, regardless of whether it is suitable or not,
the use of AI technology for modeling and solving.

AI technology is very successful in solving some problems, as before Go and
some image-to-speech recognition scenarios mentioned above, but cannot Due to
the successful solution of a certain field or certain problems, AI is regarded
as a ”universal method”. This paper aims at the current AI technology in
optical communication Discuss the application of AI technology in the network,
including the application of AI technology in optical communication network
applicability in the network, and raises the potential risk of using AI
technology Some coping strategies.

##  II AI Applications in Optical Networks

AI technology has been widely used in the literature of optical communication
networks [1, 2, 3, 4, 5, 6, 7]. A great deal of research can be found in this
area. This paper introduces several representative applications of AI
technology in optical communication networks. 1) On receiving At the end,
using the digital signal processing method combined with AI technology, it can
effectively improve the detection sensitivity of optical signals and improve
the optical fiber transmission system performance and improve the spectrum
utilization efficiency of the network [8, 7]. 2) In the optical network, there
are a large number of end-to-end optical channels, and these optical channels
are respectively related parameters (including transmission rate, modulation
format, number of optical fiber links, number of optical amplifiers and gain,
etc.) and their receiving the signal transmission quality (QoT, quality of
transmission) detected by the end is used as input and output, and through a
lot of learning, it can be realized Prediction of QoT for different end-to-end
optical channels in optical networks; where QoT often expressed as the signal-
to-noise ratio of the optical channel (OSNR, optical signal to noise ratio),
its accurate prediction can reduce the optical channel OSNR margin
configuration, thereby improving the spectrum utilization efficiency of the
network [9, 10]. 3) pass Continuously learn the fault events in the optical
network, and use the fault and fault cause Because it is used as input and
output, it can accurately analyze and diagnose the cause of the fault early
warning of future failures [3]. 4) Combined with the need for network
security, AI technology can also be used for early warning and identification
of network attacks on the optical layer [11].

##  III AI Applications in 5G Communication

AI and ML are being utilized in 5G and mmWave communication [12, 13, 14, 15,
16, 17, 18] to enhance performance, reduce costs, and boost efficiency.
Applications of AI in this field include network optimization, predictive
maintenance, self-organizing networks, traffic prediction, security, resource
allocation, network slicing, edge computing, interference management, and
spectrum management. These technologies are still in the early stages of
development but have the potential to significantly improve the performance,
efficiency, and cost-effectiveness of these networks. As AI technologies
continue to evolve and become more widely adopted, they will likely play an
increasingly important role in the development and deployment of 5G and mmWave
communication systems. However, it is also crucial to consider potential risks
and challenges such as ensuring privacy, security, and ethical considerations.

AI techniques apply the same “black box” approach to different application
scenarios leading to method innovation and analysis of the underlying
mechanism slack. A very typical example is as follows. Thanks to AI
technologies such as deep learning can effectively recognize some image
patterns, there are studies that the researchers applied this technology to
the identification of lesions in different parts of the human body [19, 20].
Based on the same method and process, different human parts are continuously
used bit pictures, which can form a large number of so-called ”research
results” and thesis. Obviously, from the perspective of cultivating students
and scientific research, students The development of research skills and
professionalism acquired practically in the project are very few, and the
actual work is only to collect relevant image data and Write a small amount of
Python code, and finally hand over the training task to the graph Processor
(GPU, graphics processing unit) to complete, no Carry out in-depth thinking on
the method and mechanism of specific research questions It is obviously not
conducive to innovation and effective innovation, and it is impossible to
grasp (in fact, it is currently impossible to grasp) what is going on in the
black box.

## References

  * [1] M. Zhang, C. You, H. Jiang, and Z. Zhu, “Dynamic and adaptive bandwidth defragmentation in spectrum-sliced elastic optical networks with time-varying traffic,” _Journal of Lightwave Technology_ , vol. 32, no. 5, pp. 1014–1023, 2014.
  * [2] W. Lu, X. Jin, and Z. Zhu, “Game theoretical flexible service provisioning in ip over elastic optical networks,” in _2017 16th International Conference on Optical Communications and Networks (ICOCN)_. IEEE, 2017, pp. 1–3.
  * [3] X. Jin, W. Lu, S. Liu, and Z. Zhu, “On multi-layer restoration in optical networks with encryption solution deployment,” in _2018 Optical Fiber Communications Conference and Exposition (OFC)_. IEEE, 2018, pp. 1–3.
  * [4] L. Gong and Z. Zhu, “Virtual optical network embedding (vone) over elastic optical networks,” _Journal of Lightwave Technology_ , vol. 32, no. 3, pp. 450–460, 2013.
  * [5] Y. Yin, H. Zhang, M. Zhang, M. Xia, Z. Zhu, S. Dahlfort, and S. B. Yoo, “Spectral and spatial 2d fragmentation-aware routing and spectrum assignment algorithms in elastic optical networks,” _Journal of Optical Communications and Networking_ , vol. 5, no. 10, pp. A100–A106, 2013.
  * [6] O. J. Ciceri, C. A. Astudillo, Z. Zhu, and N. L. da Fonseca, “Federated learning over next-generation ethernet passive optical networks,” _arXiv preprint arXiv:2109.14593_ , 2021.
  * [7] X. Chen, B. Li, R. Proietti, H. Lu, Z. Zhu, and S. B. Yoo, “Deeprmsa: a deep reinforcement learning framework for routing, modulation and spectrum assignment in elastic optical networks,” _Journal of Lightwave Technology_ , vol. 37, no. 16, pp. 4155–4163, 2019.
  * [8] X. Chen, B. Li, R. Proietti, Z. Zhu, and S. B. Yoo, “Multi-agent deep reinforcement learning in cognitive inter-domain networking with multi-broker orchestration,” in _2019 Optical Fiber Communications Conference and Exhibition (OFC)_. IEEE, 2019, pp. 1–3.
  * [9] X. Chen, B. Li, R. Proietti, C.-Y. Liu, Z. Zhu, and S. B. Yoo, “Demonstration of distributed collaborative learning with end-to-end qot estimation in multi-domain elastic optical networks,” _Optics Express_ , vol. 27, no. 24, pp. 35 700–35 709, 2019.
  * [10] S. Liu, B. Li, and Z. Zhu, “Realizing ai-assisted multi-layer restoration in a software-defined ip-over-eon with deep learning: An experimental study,” in _Optical Fiber Communication Conference_. Optical Society of America, 2018, pp. W4F–1.
  * [11] X. Tian, B. Li, R. Gu, and Z. Zhu, “Reconfiguring multicast sessions in elastic optical networks adaptively with graph-aware deep reinforcement learning,” _Journal of Optical Communications and Networking_ , vol. 13, no. 11, pp. 253–265, 2021.
  * [12] X. Lu, R. Zhang, Y. Zhou, J. Liu, X. Jin, Q. Guo, and C. Cao, “Convolutional modeling and antenna de-embedding for wideband spatial mmwave channel measurement,” in _2017 IEEE Wireless Communications and Networking Conference (WCNC)_. IEEE, 2017, pp. 1–6.
  * [13] K. Shafique, B. A. Khawaja, F. Sabir, S. Qazi, and M. Mustaqim, “Internet of things (iot) for next-generation smart systems: A review of current challenges, future trends and prospects for emerging 5g-iot scenarios,” _Ieee Access_ , vol. 8, pp. 23 022–23 040, 2020.
  * [14] A. Alkhateeb, O. El Ayach, G. Leus, and R. W. Heath, “Channel estimation and hybrid precoding for millimeter wave cellular systems,” _IEEE journal of selected topics in signal processing_ , vol. 8, no. 5, pp. 831–846, 2014.
  * [15] P. Bhartia and I. J. Bahl, “Millimeter wave engineering and applications,” 1984\.
  * [16] T. S. Rappaport, S. Sun, R. Mayzus, H. Zhao, Y. Azar, K. Wang, G. N. Wong, J. K. Schulz, M. Samimi, and F. Gutierrez, “Millimeter wave mobile communications for 5g cellular: It will work!” _IEEE access_ , vol. 1, pp. 335–349, 2013.
  * [17] Z. Pi and F. Khan, “An introduction to millimeter-wave mobile broadband systems,” _IEEE communications magazine_ , vol. 49, no. 6, pp. 101–107, 2011\.
  * [18] M. Marcus and B. Pattan, “Millimeter wave propagation: spectrum management implications,” _IEEE Microwave Magazine_ , vol. 6, no. 2, pp. 54–62, 2005\.
  * [19] A. Shrikumar, P. Greenside, A. Shcherbina, and A. Kundaje, “Not just a black box: Learning important features through propagating activation differences,” _arXiv preprint arXiv:1605.01713_ , 2016.
  * [20] M. Sendak, M. C. Elish, M. Gao, J. Futoma, W. Ratliff, M. Nichols, A. Bedoya, S. Balu, and C. O’Brien, “” the human body is a black box” supporting clinical decision-making with deep learning,” in _Proceedings of the 2020 conference on fairness, accountability, and transparency_ , 2020, pp. 99–109.


11institutetext: Institute of Computational Intelligence, Czȩstochowa
University of Technology
Al. Armii Krajowej 36, 42-200 Czȩstochowa, Poland
11email:  {rafal.grycuk, marcin.gabryel<EMAIL_ADDRESS>
http://iisi.pcz.pl 22institutetext: University of Geneva, Computer Science
Department,
7 Route de Drize, Geneva, Switzerland
group web page: http://sip.unige.ch

# Multi-layer Architecture For Storing Visual Data Based on WCF and Microsoft
SQL Server Database

Rafał Grycuk 11 Marcin Gabryel 11 Rafał Scherer 11 Sviatoslav Voloshynovskiy
22

###### Abstract

In this paper we present a novel architecture for storing visual data.
Effective storing, browsing and searching collections of images is one of the
most important challenges of computer science. The design of architecture for
storing such data requires a set of tools and frameworks such as SQL database
management systems and service-oriented frameworks. The proposed solution is
based on a multi-layer architecture, which allows to replace any component
without recompilation of other components. The approach contains five
components, i.e. Model, Base Engine, Concrete Engine, CBIR service and
Presentation. They were based on two well-known design patterns: Dependency
Injection and Inverse of Control. For experimental purposes we implemented the
SURF local interest point detector as a feature extractor and $K$-means
clustering as indexer. The presented architecture is intended for content-
based retrieval systems simulation purposes as well as for real-world CBIR
tasks.

###### Keywords:

WCF, Microsoft SQL Server, Dependency Injection, Inversion of Control, Entity
Framework, Multi-layer Architecture, $k$-means, SURF, Content-Based Image
Retrieval

##  1 Introduction

Images are created everyday in tremendous amount and there is ongoing research
to make it possible to efficiently search these vast collections by their
content. Generally, this work can be divided into image classification [4][18]
and image retrieval [17]. Recognizing images and objects on images relies on
suitable feature extraction which can be basically divided into several
groups, i.e. based on color representation [28], textures [48], shape
[50][52], edge detectors [42][43][53] or local invariant features, e.g. SURF
[2], SIFT [34] or ORB [46]. Matching features can be also performed by several
methods, e.g. clustering, nearest neighbour, bag of features [21] or soft
computing [15][30].

There are many content-based image processing systems developed so far. A good
review of such systems is provided in [51]. To the best of our knowledge no
other system uses similar set of tools to the system proposed in the paper.
Now we describe briefly the most important tools used to design the proposed
framework.

###  1.1 Windows Communication Foundation

Windows Communication Foundation (WCF) is a framework based on Service-
Oriented Architecture [14][27]. WCF allows to send data asynchronously between
two service and client endpoints. Service endpoints can be deployed on IIS
server or be hosted locally. The messages can be send as XML (value types) or
binary file (complex types) [9][29][35][39]. WCF consist of following
features:[45][49] Service Orientation, Interoperability, Multiple Message
Patterns, Service Metadata, Data Contracts, Security, Multiple Transports and
Encodings, Reliable and Queued Messages, Durable Messages, Transactions, AJAX
and REST Support, Extensibility.

###  1.2 SQL Sever

MS SQL Server is a database management system for storing various types of
data, fully supporting cloud computing technologies [5][11][12][16]. It
provides a set of tools to extract data from various devices or sources, even
at datacenters. MS SQL query language, T-SQL (Transact-SQL), allows for both
structural or procedural queries [6][10][13][31][37][38]. The DBMS (Database
Management System) is based on a client-server architecture. The platform is
composed of the following services:[33][41]

  * •

Database engine - allows to execute queries and is necessary to run the
server,

  * •

Integration Services (SSIS, SQL Server Integration Services) - ETL
(Extraction, Transformation and Loading) platform responsible for data
migration from the heterogeneous data sources,

  * •

SQL Agent - answerable for performing tasks according to the specified
schedule,

  * •

Full-text Filter Daemon Launcher - allow to perform full-text searches on text
columns,

  * •

Reporting Services (SSIS, SQL Server Reporting Services) - responsible for
designing and deploying reports,

  * •

Analysis Services (SSAS, SQL Server Analysis Services) - allows to create
multidimensional cubes and executing MDX (Multidimensional Expressions)
queries [25],

DBMS store data in a relational form (tables and their relations) and allows
to select information by executing queries. Many frameworks perform object
relational mapping. In this paper we use Entity Framework (EF) with Code First
approach [1][8][32].

###  1.3 Dependency Injection and Inverse of Control

Re-usability of existing components is crucial in modern software engineering.
The aim of this approach is to combine separate layers into one application.
This is a challenging task, because as the application complexity increases,
so do dependencies [7]. The best practice for tone down proliferation of
dependencies is by using Dependency Injection (DI) design pattern allowing to
inject objects into a class constructor. Thus, the creation of the object does
not rely on a class. The initialization logic is rarely reusable outside of
created component. That pattern provides a layer of abstraction for the
injected object, thus we can implement the concrete logic in the other
component and inject it in the class constructor by the interface. DI [39] is
an implementation of Inverse of Control (IoC)[20][44][47]. Figure 1 shows the
typical class dependencies. Such a scheme entails the following problems [36]:

  1. 1.

Any code changes of $ServiceA$ forces changes in $ClassA$, thus the
recompilation of all components is required,

  2. 2.

All classes must be implemented and available at the compile time,

  3. 3.

Classes are difficult to test and to achieve components isolation,

  4. 4.

Contradicts the (DRY) Don’t Repeat Yourself principle.

To resolve this issue we used the Dependency Injection [7][20]. A Conceptual
view of DI is presented in Figure 2.

Figure 1: Problem illustration of tightly coupled dependencies. $ClassA$ uses
$ServiceA$ and $ServiceB$. This is a simple representation of $ClassA$
dependencies on $ServiceA$ and $ServiceB$.  Figure 2: Solution of the problem
formulated in Fig. 1. The Inverse of Control pattern can be implemented by
Dependency Injection. $Builder$ creates $ClassA$ which uses abstract interface
$IServiceA$. Dependencies of $ServiceA$ are injected to $ClassA$ by
inheritance.

##  2 Proposed Architecture For Storing Visual Data

Proposed architecture is based on two main components. The first one is SQL
database which is Microsoft SQL Server, and the second one is Windows
Communication Foundation (WCF). Our approach consists of five main layers
(tiers):

  1. 1.

Model - which contains data model generated by the Entity Framework (6.1) with
Code First approach,

  2. 2.

Base Engine - consist of several abstract classes or interfaces which can be
used to implement user solutions, but they provide appropriate business logic,

  3. 3.

Concrete Engine - implements user logic based on previous the layer (in this
paper we implemented the SURF descriptors for feature extraction and $k$-means
clustering [26] for indexing),

  4. 4.

CBIR Service (for more about CBIR see [23][22]) - which is WCF service that
allows to invoke engine methods as Service Oriented Applications (SOA).

  5. 5.

Endpoint (Client) - presentation layer for invoking service methods, it can be
desktop, web or mobile application. All the user needs to do, is to add
service reference and invoke methods.

The agility applied in the presented approach is important, because it is not
restricted to any particular implementation and it can be applied in various
solutions. The architecture is presented in Fig. 3a.

(a) Multi-tier architecture for storing visual data. Final components are
compiled release of dll’s and open components are user implementation.

(b) Entity diagram based on database tables.

Figure 3: Multi-layer architecture and entity diagram of the proposed
approach.

The Base Engine and Concrete Engine layers are based on two design patterns,
i.e. Dependency Injection (DI) and Inverse of Control (IoC) described in
Section 1.3. These patterns allowed to separate the containers and maintain
the S.O.L.I.D. principles [39]. Components are composed of a set of classes or
interfaces. Each component has a representation in UML (Unified Modeling
Language) diagram. Figure 5a shows the model layer, which contains five
classes and two interfaces. Classes: $Bin$, $Histogram$ and $Image$ were
generated by the Entity Framework and they correspond with the database tables
presented in Fig. 3b. Interface $IRepositoryBase$ is a generic interface which
provides the basic C.R.U.D. (create, read, update and delete) operations. The
methods allow to operate on any types of objects. The $GenericRepository$
implements $IRepositoryBase$ interface. In addition, the $dataContext$ field
is generic, thus the concrete implementation does not contains any
dependencies. The $GenericRepository$ class is based on the Singleton pattern,
to create instance which user needs to use static method $GetInstance$. A very
interesting interface is $IFeature$, which allows to implement any type of
image features (an image descriptor).

(a) Class diagram for Concrete Engine layer.

(b) Class diagram for Base Engine layer.

Figure 4: Class diagrams for Concrete Engine and Base Engine layers.

(a) Class diagram for Model layer.

(b) Class diagram for CBIR Service layer.

Figure 5: Class diagrams for Model and CBIR Service layers.

Fig. 4b represents a class diagram for the Base Engine layer, which contains
items for feature extraction ($IFeatureExtractor$), feature indexation
($Feature$ $Indexer$), operations executions ($IExecutor$, $ExecutorBase$) and
simulation ($SimulationEvaluator$, $RetrievalFactors$). $IExecutor$ provides
abstract methods for following operations: index creation and deletion, query
execution and image insertion. It contains one property: repository.
$IFeatureExtractor$ is responsible for feature extraction and contains:
$Features$ property and ExtractFeatures method. The first one stores
previously extracted features, the second one extracts features from image
passed as a parameter. The method is only a definition, thus the class that
will implement that interface must contain its own version. $FeatureIndexer$
consists of one field (histograms) which is a composition relation with the
$Histogram$ class. Methods are abstract, thus they require concrete
implementation in the inherited class. $ExecutorBase$ is a base class that
uses the dependency injection to initiate object, that allows to inject logic.
The $RetrievalFactors$ contains fields that describes the query results. The
Concrete Engine layer (see Fig. 4a) is designed and presented for simulation
purposes. We implemented SURF [3][19][24] as a feature extractor and $k$-means
method as a indexer. Figure 5b shows diagram for the CBIR Service layer. Each
WCF service consists of the following items:

  * •

Interface - defines the method e.g. $IImage$ $Retrival$ $Service$,

  * •

Implementation class - implements the method’s body e.g. $Image$ $Retrieval$
$Service$,

  * •

Contract (optional) - required to retrieve data from the service $Image$
$Contract$.

CBIR Service component contains two services: simulation service is used to
perform simulations on the created index. The most interesting methods are
SimulateMultiQuery and SimulateSingleQuery. The first one returns a list of
$RetrievalFactors$ and performs multi query. The second one executes a single
query. The image retrieval service allows to execute queries. The difference
between SimulateSingleQuery and ExecuteQuery methods are the following:
ExecuteQuery returns a list of retrieved images, SimulateSingleQuery returns a
list of factors ($precision$, $recall$) which allows to evaluate index
efficiency. The proposed architecture was designed in .NET framework and
written in C#.

##  3 Experimental Results

Experiments were carried out using the designed architecture. The created
endpoint was a desktop application with service reference to CBIR Service.
Research includes experiments on various objects with background described by
the SURF local interest point descriptor [34]. Test images were taken from the
Corel database. We chose images with various types of objects. In experiments
we used 90% of each class for index creating and 10% as query images. In Tab.2
we presented the retrieved factors for each query image. Tab. 2 shows
retrieved images for a query image (the image with border).

For the purposes of the performance evaluation we use two measures; precision
and recall [40]. Fig. 6 shows the performance measures of the image retrieval.
The $AI$ is a set of appropriate images, that should be returned as being
similar to the query image. The $RI$ represents a set of returned images by
the system. $Rai$ is a group of properly returned images. $Iri$ represents
improperly returned images, $anr$ proper not returned and $inr$ improper not
returned images. The presented measure allows to define $precision$ and
$recall$ by the following formulas [40]:

$precision=\frac{|rai|}{|rai+iri|},$ (1) $recall=\frac{|rai|}{|rai+anr|}.$ (2)

Figure 6:  Performance measures diagram.

Image Id | RI | AI | rai | iri | anr |  Precision |  Recall
---|---|---|---|---|---|---|---
1 (1).jpg | 19 | 33 | 21 | 12 | 7 | 13 | 7
1 (2).jpg | 21 | 33 | 31 | 3 | 18 | 3 | 19
1 (20).jpg | 19 | 33 | 21 | 12 | 7 | 13 | 7
1 (21).jpg | 19 | 33 | 21 | 12 | 7 | 13 | 7
2 (1).jpg | 14 | 33 | 27 | 7 | 7 | 7 | 8
2 (10).jpg | 12 | 33 | 28 | 6 | 6 | 6 | 7
2 (11).jpg | 19 | 33 | 24 | 9 | 9 | 10 | 10
2 (17).jpg | 12 | 33 | 28 | 6 | 6 | 6 | 7
3 (1).jpg | 22 | 33 | 23 | 11 | 12 | 11 | 12
3 (10).jpg | 21 | 33 | 23 | 10 | 11 | 11 | 12
3 (11).jpg | 18 | 33 | 27 | 7 | 11 | 7 | 12
3 (15).jpg | 21 | 33 | 29 | 4 | 17 | 4 | 17
Table 1: Simulation results for multi query. Measures were normalized and
presented as percentage value [%].

Table 2: Query results. Eighteen example images from the experiment. The image
with border is the query image.

##  4 Final Remarks

The presented approach is a novel architecture for storing visual data. We
used SQL Server and WCF services as a core of our method. The proposed
solution for storing visual data has no dependencies with concrete
implementation, thus we can simulate any CBIR method. Our approach allows
creating any type of endpoint: desktop, web or mobile application. It can be
used as a core back-end solution. The performed experiments proved
effectiveness of our method. Our paper present a part of a larger system that
allows to search and identify specific classes of objects.

## Acknowledgments

The work presented in this paper was supported by a grant BS/MN-1-109-301/14/P
”Clustering algorithms for data stream - in reference to the Content-Based
Image Retrieval methods (CBIR)”. The work presented in this paper was
supported partially by a grant from Switzerland through the Swiss Contribution
to the enlarged European Union.

## References

  * [1] Adya, A., Blakeley, J.A., Melnik, S., Muralidhar, S.:  Anatomy of the ado. net entity framework.  In: Proceedings of the 2007 ACM SIGMOD international conference on Management of data, ACM (2007) 877–888
  * [2] Bay, H., Ess, A., Tuytelaars, T., Van Gool, L.:  Speeded-up robust features (surf).  Comput. Vis. Image Underst. 110(3) (June 2008) 346–359
  * [3] Bay, H., Tuytelaars, T., Van Gool, L.:  Surf: Speeded up robust features.  In: Computer Vision–ECCV 2006.  Springer (2006) 404–417
  * [4] Bazarganigilani, M.:  Optimized image feature selection using pairwise classifiers.  Journal of Artificial Intelligence and Soft Computing Research 1(2) (2011) 147–153
  * [5] Ben-Gan, I.:  Microsoft SQL Server 2012 T-SQL Fundamentals.  Pearson Education (2012)
  * [6] Biniaz, A., Abbasi, A.:  Segmentation and edge detection based on modified ant colony optimization for iris image processing.  Journal of Artificial Intelligence and Soft Computing Research 3(2) (2013) 133–141
  * [7] Caprio, G.:  Design patterns-dependency injection.  MSDN Magazine (2005) 103–110
  * [8] Castro, P., Melnik, S., Adya, A.:  Ado. net entity framework: raising the level of abstraction in data programming.  In: Proceedings of the 2007 ACM SIGMOD international conference on Management of data, ACM (2007) 1070–1072
  * [9] Chappell, D.:  Introducing windows communication foundation in .net framework 4.  Retrieved May 11 (2010) 2011
  * [10] Chen, M., Ludwig, S.A.:  Particle swarm optimization based fuzzy clustering approach to identify optimal number of clusters.  Journal of Artificial Intelligence and Soft Computing Research 4(1) (2014) 43–56
  * [11] Chromiak, M., Stencel, K.:  The linkup data structure for heterogeneous data integration platform.  In: Future Generation Information Technology - 4th International Conference, FGIT 2012, Gangneug, Korea, December 16-19, 2012. Proceedings. (2012) 263–274
  * [12] Chromiak, M., Stencel, K.:  A data model for heterogeneous data integration architecture.  In: Beyond Databases, Architectures, and Structures - 10th International Conference, BDAS 2014, Ustron, Poland, May 27-30, 2014. Proceedings. (2014) 547–556
  * [13] Chu, J.L., Krzyźak, A.:  The recognition of partially occluded objects with support vector machines, convolutional neural networks and deep belief networks.  Journal of Artificial Intelligence and Soft Computing Research 4(1) (2014) 5–19
  * [14] Cibraro, P., Claeys, K., Cozzolino, F., Grabner, J.:  Professional WCF 4: Windows Communication Foundation with. NET 4.  John Wiley & Sons (2010)
  * [15] Cpalka, K.:  On evolutionary designing and learning of flexible neuro-fuzzy structures for nonlinear classification.  Nonlinear Analysis: Theory, Methods & Applications 71(12) (2009) e1659 – e1672
  * [16] Delaney, K., Freeman, C.:  Microsoft SQL Server 2012 Internals.  Microsoft Press (2013)
  * [17] Drozda, P., Sopyla, K., Górecki, P.:  Online crowdsource system supporting ground truth datasets creation.  In: Artificial Intelligence and Soft Computing - 12th International Conference, ICAISC 2013, Zakopane, Poland, June 9-13, 2013, Proceedings, Part I. (2013) 532–539
  * [18] Drozda, P., Sopyla, K., Górecki, P.:  Different orderings and visual sequence alignment algorithms for image classification.  In: Artificial Intelligence and Soft Computing - 13th International Conference, ICAISC 2014, Zakopane, Poland, June 1-5, 2014, Proceedings, Part I. (2014) 693–702
  * [19] Evans, C.:  Notes on the opensurf library.  University of Bristol, Tech. Rep. CSTR-09-001, January (2009)
  * [20] Fowler, M.:  Inversion of control containers and the dependency injection pattern (2004)
  * [21] Grauman, K., Darrell, T.:  Efficient image matching with distributions of local invariant features.  In: Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on. Volume 2. (June 2005) 627–634 vol. 2
  * [22] Grycuk, R., Gabryel, M., Korytkowski, M., Romanowski, J., Scherer, R.:  Improved digital image segmentation based on stereo vision and mean shift algorithm.  In Wyrzykowski, R., Dongarra, J., Karczewski, K., Waśniewski, J., eds.: Parallel Processing and Applied Mathematics. Volume 8384 of Lecture Notes in Computer Science.  Springer Berlin Heidelberg (2014) 433–443
  * [23] Grycuk, R., Gabryel, M., Korytkowski, M., Scherer, R.:  Content-based image indexing by data clustering and inverse document frequency.  In Kozielski, S., Mrozek, D., Kasprowski, P., Małysiak-Mrozek, B., Kostrzewa, D., eds.: Beyond Databases, Architectures, and Structures. Volume 424 of Communications in Computer and Information Science.  Springer International Publishing (2014) 374–383
  * [24] Grycuk, R., Gabryel, M., Korytkowski, M., Scherer, R., Voloshynovskiy, S.:  From single image to list of objects based on edge and blob detection.  In Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L., Zurada, J., eds.: Artificial Intelligence and Soft Computing. Volume 8468 of Lecture Notes in Computer Science.  Springer International Publishing (2014) 605–615
  * [25] Harinath, S., Pihlgren, R., Lee, D.G.Y., Sirmon, J., Bruckner, R.M.:  Professional Microsoft SQL Server 2012 Analysis Services with MDX and DAX.  John Wiley & Sons (2012)
  * [26] Hartigan, J.A., Wong, M.A.:  Algorithm as 136: A k-means clustering algorithm.  Applied statistics (1979) 100–108
  * [27] Hirschheim, R., Welke, R., Schwarz, A.:  Service-oriented architecture: Myths, realities, and a maturity model.  MIS Quarterly Executive 9(1) (2010) 37–48
  * [28] Huang, J., Kumar, S., Mitra, M., Zhu, W.J., Zabih, R.:  Image indexing using color correlograms.  In: Computer Vision and Pattern Recognition, 1997. Proceedings., 1997 IEEE Computer Society Conference on. (Jun 1997) 762–768
  * [29] Klein, S.:  Professional WCF programming:. NET development with the Windows communication foundation.  John Wiley & Sons (2007)
  * [30] Łapa, K., Zalasiński, M., Cpałka, K.:  A new method for designing and complexity reduction of neuro-fuzzy systems for nonlinear modelling.  In Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L., Zurada, J., eds.: Artificial Intelligence and Soft Computing. Volume 7894 of Lecture Notes in Computer Science.  Springer Berlin Heidelberg (2013) 329–344
  * [31] Leblanc, P.:  Microsoft SQL Server 2012 Step by Step.  Pearson Education (2013)
  * [32] Lerman, J., Miller, R.:  Programming Entity Framework: Code First.  ” O’Reilly Media, Inc.” (2011)
  * [33] Lobel, L., Brust, A.:  Programming Microsoft® SQL Server® 2012\.  ” O’Reilly Media, Inc.” (2012)
  * [34] Lowe, D.G.:  Distinctive image features from scale-invariant keypoints.  Int. J. Comput. Vision 60(2) (November 2004) 91–110
  * [35] Mackey, A.:  Windows communication foundation.  In: Introducing. NET 4.0.  Springer (2010) 159–173
  * [36] Magazine, M.:  Design patterns: Dependency injection (2005)
  * [37] Makinana, S., Malumedzha, T., Nelwamondo, F.V.:  Quality parameter assessment on iris images.  Journal of Artificial Intelligence and Soft Computing Research 4(1) (2014) 21–30
  * [38] Mallik, S., Mukhopadhyay, A., Maulik, U.:  Integrated statistical and rule-mining techniques for dna methylation and gene expression data analysis.  Journal of Artificial Intelligence and Soft Computing Research 3(2) (2013) 101–115
  * [39] Martin, M., Martin, R.C.:  Agile principles, patterns, and practices in C#.  Pearson Education (2006)
  * [40] Meskaldji, K., Boucherkha, S., Chikhi, S.:  Color quantization and its impact on color histogram based image retrieval.  (2009)
  * [41] Mistry, R., Misner, S.:  Introducing Microsoft® SQL Server® 2012\.  ” O’Reilly Media, Inc.” (2012)
  * [42] Ogiela, M.R., Tadeusiewicz, R.:  Syntactic reasoning and pattern recognition for analysis of coronary artery images.  Artificial Intelligence in Medicine 26(1) (2002) 145–159
  * [43] Ogiela, M.R., Tadeusiewicz, R.:  Nonlinear processing and semantic content analysis in medical imaging-a cognitive approach.  Instrumentation and Measurement, IEEE Transactions on 54(6) (2005) 2149–2155
  * [44] Prasanna, D.R.:  Dependency injection.  Manning Publications Co. (2009)
  * [45] Rosen, M., Lublinsky, B., Smith, K.T., Balcer, M.J.:  Applied SOA: service-oriented architecture and design strategies.  John Wiley & Sons (2012)
  * [46] Rublee, E., Rabaud, V., Konolige, K., Bradski, G.:  Orb: An efficient alternative to sift or surf.  In: Computer Vision (ICCV), 2011 IEEE International Conference on. (Nov 2011) 2564–2571
  * [47] Seemann, M.:  Dependency injection in. NET.  Manning (2012)
  * [48] Śmietański, J., Tadeusiewicz, R., Łuczyńska, E.:  Texture analysis in perfusion images of prostate cancer—a case study.  International Journal of Applied Mathematics and Computer Science 20(1) (2010) 149–156
  * [49] Tsai, W.T., Sun, X., Balasooriya, J.:  Service-oriented cloud computing architecture.  In: Information Technology: New Generations (ITNG), 2010 Seventh International Conference on, IEEE (2010) 684–689
  * [50] Veltkamp, R.C., Hagedoorn, M.:  State of the art in shape matching.  In Lew, M.S., ed.: Principles of Visual Information Retrieval.  Springer-Verlag, London, UK, UK (2001) 87–119
  * [51] Veltkamp, R.C., Tanase, M.:  Content-based image retrieval systems: A survey.  Department of Computing Science, Utrecht University (2002) 1–62
  * [52] Zalasinski, M., Cpalka, K.:  New approach for the on-line signature verification based on method of horizontal partitioning.  In Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L., Zurada, J., eds.: Artificial Intelligence and Soft Computing. Volume 7895 of Lecture Notes in Computer Science.  Springer Berlin Heidelberg (2013) 342–350
  * [53] Zitnick, C., Dollar, P.:  Edge boxes: Locating object proposals from edges.  In Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T., eds.: Computer Vision – ECCV 2014. Volume 8693 of Lecture Notes in Computer Science.  Springer International Publishing (2014) 391–405