can be developed in brain evolution or achieved by machine learning algorithm,
but for application of the correctors or for appearance of concept cells,
these three conditions should be fulfilled. The third condition can be
considered as a byproduct of the first two and the tendency to extend the
sensitivity to ‘vital differences’ as much as possible under limited
resources. If, after compression, the dimension remains sufficiently high then
the single linear elements become useful and blessing of dimensionality
appears.

The riddle of simplicity is split into the applications of the extended
background of statistical physics and into the engineering and reverse
engineering technology of lossy compression in preprocessing. We partially
understand now the revolution of simplicity (single-cell revolution) in
neuroscience and high-dimensional data processing but the design of
preprocessing of big data and unravelling the data preprocessing in biological
neural nets retain a lot of puzzles.

One can say that the preprocessing creates the inner world from the outer
world, and the concept cells (or small ensembles of cells) translate events in
this inner world into the concepts. This is not the mind yet but an important
precursor of the mind.

##  6 Acknowledgments

We are grateful to our coauthors B. Grechuck, E.M. Mirkes, D. Prokhorov, K.
Sofeykov, I. Romanenko, J. Makarova, C. Calvo, A. Golubkov, and R. Burton.
Without our joint work [35, 36, 39, 42, 95, 96], this review would not have
been possible. The work was supported by the Ministry of Education and Science
of the Russian Federation (Project No. 14.Y26.31.0022). Work of ANG and IVT
was also supported by Innovate UK (Knowledge Transfer Partnership grants
KTP009890; KTP010522) and University of Leicester. VAM acknowledges support
from the Spanish Ministry of Economy, Industry and Competitiveness (grant
FIS2017-82900-P).

## References

  * [1] Amaral DG, Witter MP.  The three-dimensional organization of the hippocampal formation: a review of anatomical data.  Neuroscience 1989; 31:571–591.
  * [2] Anderson J., Belkin M., Goyal N., Rademacher L., Voss J. The More, the Merrier: the Blessing of dimensionality for learning large Gaussian mixtures. Journal of Machine Learning Research: Workshop and Conference Proceedings 2014;35:1–30.
  * [3] Averbeck BB, Latham PE, Pouget A., . Neural correlations, population coding and computation. Nature reviews neuroscience 2006;7(5):358–366.
  * [4] Ball K. An Elementary Introduction to Modern Convex Geometry. Flavors of Geometry, Vol. 31. Cambridge, UK: MSRI Publications; 1997.
  * [5] Barlow HB. Single units and sensation: a neuron doctrine for perceptual psychology?. Perception 1972;1(4):371–394.
  * [6] Benito N, Fernandez-Ruiz A, Makarov VA, Makarova J, Korovaichuk A, Herreras O.  Spatial modules of coherent activity in pathway-specific LFPs in the hippocampus reflect topology and different modes of presynaptic synchronization.  Cerebral Cortex 2014; 24(7):1738–1752.
  * [7] Benito N, Martin-Vazquez G, Makarova J, Makarov VA, Herreras O.  The right hippocampus leads the bilateral integration of gamma-parsed lateralized information.  eLife, 5:e16658, doi: 10.7554/eLife.16658, 2016.
  * [8] Bowers JS. On the biological plausibility of grandmother cells: implications for neural network theories in psychology and neuroscience. Psychological review 2009;116(1):220–251.
  * [9] Bowers JS. More on Grandmother Cells and the Biological Implausibility of PDP Models of Cognition: A Reply to Plaut and McClelland (2010) and Quian Quiroga and Kreiman (2010), Psychological Review 2010;117(1):300–308.
  * [10] Brock W, Lakonishok J, LeBaron B. Simple technical trading rules and the stochastic properties of stock returns. The Journal of finance 1992;47(5):1731–1764.
  * [11] Burton R. NOTTINGHAM video, 2016. https://youtu.be/SJbhOJQCSuQ, notea test video for pedestrians detection taken from the streets of Nottingham by an action camera.
  * [12] Cangelosi R, Goriely A. Component retention in principal component analysis with application to cDNA microarray data, Biol. Direct 2007, 2:2. https://doi.org/10.1186/1745-6150-2-2
  * [13] Cerf M, Thiruvengadam N, Mormann F, Kraskov A, Quian Quiroga R, Koch C, Fried I. On-line, voluntary control of human temporal lobe neurons. Nature 2010;467(7319):1104–1108.
  * [14] Cucker F, Smale S.  On the mathematical foundations of learning.  Bulletin of the American mathematical society 2002;39(1):1–49.
  * [15] Dalal N, Triggs B. Histograms of oriented gradients for human detection. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2005). Vol. 1, pp. 886-893. IEEE press, 2005.
  * [16] Donoho DL. 2000 High-dimensional data analysis: The curses and blessings of dimensionality. AMS Math Challenges Lecture, 1, 33 pp. See https://pdfs.semanticscholar.org/63c6/8278418b69f60b4814fae8dd15b1b1854295.pdf.
  * [17] Donoho DL. Compressed sensing. IEEE Transactions on Information Theory 2006;52(4):1289–1306.
  * Donoho and Tanner [2009] Donoho D, Tanner J. Observed universality of phase transitions in high-dimensional geometry, with implications for modern data analysis and signal processing. Phil. Trans. R. Soc. A 2009;367:4273–4293.
  * [19] Dormann CF, Elith J, Bacher S, Buchmann C, Carl G, Carré G, Marquéz JR, Gruber B, Lafourcade B, Leitão PJ, Münkemüller T. Collinearity: a review of methods to deal with it and a simulation study evaluating their performance. Ecography 2013;36(1):27–46.
  * [20] Eichenbaum H.. Thinking about brain cell assemblies. Science 1993; 261(5124):993–995.
  * [21] Einstein A. Autobiographical Notes, in Albert Einstein, Philosopher-Scientist, PA Schilpp, ed., Evanston: Open Court, 1949.
  * [22] Eldar, Y.C. and Kutyniok, G. eds.,. Compressed sensing: theory and applications. Cambridge University Press, 2012.
  * [23] Face recognition systems and error rates - is this a concern? Biometrics Institute, Thu 24 May 2018, https://www.biometricsinstitute.org/blogs/face-recognition-systems-and-error-rates-is-this-a-concern-
  * [24] Fernandez-Ruiz A, Makarov VA, Herreras O.  Sustained increase of spontaneous input and spike transfer in the CA3-CA1 pathway following long term potentiation in vivo.  Frontiers in Neural Circuits 2012;6:71 https://doi.org/10.3389/fncir.2012.00071.
  * [25] Feynman R. The character of physical law. Boston: MIT press, 2017.
  * [26] Finnerty CT, Jefferys JGR.  Functional connectivity from CA3 to the ipsilateral and contralateral CA1 in the rat dorsal hippocampus.  Neuroscience 1993;56(1):101–108.
  * [27] Fontaine B, Peña JL, Brette R. Spike-threshold adaptation predicted by membrane potential dynamics in vivo. PLoS computational biology 2014;10(4):e1003560.
  * [28] Fried I, MacDonald KA, Wilson C. Single neuron activity in human hippocampus and amygdale during recognition of faces and objects. Neuron 1997;18:753–765.
  * Foxx [2018] Foxx C. Face recognition police tools ‘staggeringly inaccurate’, BBC News, Technology, 15 May 2018. http://www.bbc.co.uk/news/technology-44089161
  * [30] Giannopoulos AA, Milman VD. Concentration property on probability spaces. Adv. Math. 2000;156:77–106.
  * [31] Gibbs GW. Elementary Principles in Statistical Mechanics, Developed With Especial Reference to the Rational Foundation of Thermodynamics. New York: Dover Publications, 1960 [1902]
  * [32] Gorban AN. Looking for Tomorrows Mainstream, An inaugural lecture given at the University of Leicester in February, 2006, https://doi.org/10.13140/RG.2.2.34822.47682.
  * [33] Gorban AN. Model reduction in chemical dynamics: slow invariant manifolds, singular perturbations, thermodynamic estimates, and analysis of reaction graph. Current Opinion in Chemical Engineering 2018;21:48–59.
  * [34] Gorban AN. Hilbert’s Sixth Problem: the endless road to rigour. Phil. Trans. R. Soc. A 2018;376(2118):20170238.
  * [35] Gorban AN, Golubkov A, Grechuk B, Mirkes EM, Tyukin IY. Correction of AI systems by linear discriminants: Probabilistic foundations. Inf. Sci. 2018; https://doi.org/10.1016/j.ins.2018.07.040.
  * [36] Gorban AN, Grechuk B, Tyukin IY. Augmented Artificial Intelligence: a Conceptual Framework, arXiv preprint 2018, arXiv:1802.02172 [cs.AI].
  * [37] Gorban AN, Kégl B, Wunsch D, Zinovyev A. (eds.) Principal Manifolds for Data Visualisation and Dimension Reduction. Lecture Notes in Computational Science and Engineering, Vol. 58. Berlin – Heidelberg – New York: Springer, 2008.
  * [38] Gorban AN, Kazantzis N, Kevrekidis IG, Öttinger HC, Theodoropoulos C. (eds.) Model Reduction and Coarse–Graining Approaches for Multiscale Phenomena, Berlin-Heidelberg-New York: Springer, 2006.
  * Gorban, Romanenko et al. [2016] Gorban AN, Romanenko I, Burton R, Tyukin IY, One-trial correction of legacy AI systems and stochastic separation theorems, arXiv preprint 2016, arXiv:1610.00494 [stat.ML]. https://arxiv.org/abs/1610.00494
  * [40] Gorban AN, Tyukin IY. Stochastic separation theorems. Neural Netw. 2017;94:255–259.
  * Gorban and Tyukin [2018] Gorban AN, Tyukin IY, Blessing of dimensionality: mathematical foundations of the statistical physics of data, Phil. Trans. R. Soc. A 2018;376:20170237.
  * [42] Gorban AN, Tyukin I, Prokhorov D, Sofeikov K. Approximation with random bases: Pro et contra. Inf. Sci. 2016:364–365;129–145.
  * [43] Gorban AN, Tyukin IY, Romanenko I. The blessing of dimensionality: Separation theorems in the thermodynamic limit. IFAC-PapersOnLine 2016;49:64–69.
  * [44] Gorban AN, Yablonsky GS. Grasping Complexity, Computers and Mathematics with Applications 2013;65(10):1421–1426.
  * [45] Gorban AN, Zinovyev A.Principal manifolds and graphs in practice: from molecular biology to dynamical systems. International Journal of Neural Systems 2010;20(03):219–232.
  * [46] Gromov M. Isoperimetry of waists and concentration of maps. Geom. Funct. Anal. 2003;13:178–215.
  * [47] Gross CG. Genealogy of the “grandmother cell”. The Neuroscientist 2002;8(5):512–518.
  * [48] He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition, in: Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, 770–778. arXiv:1512.03385
  * [49] Hecht-Nielsen R. Context vectors: General-purpose approximate meaning representations self-organized from raw data. In Zurada J, Marks R, Robinson C, eds. Computational Intelligence: Imitating Life. New York: IEEE Press, 1994, 43–56.
  * [50] Hintikka J. Individuals, possible worlds, and epistemic logic. Nous. 1967;1(1): 33–62.
  * [51] Hintikka J. Impossible possible worlds vindicated, Journal of Philosophical Logic 1975;4(4):475–484
  * [52] Huang C, Resnik A, Celikel T, Englitz B. Adaptive spike threshold enables robust and temporally precise neuronal Encoding. PLoS computational biology 2016;12(6):e1004984.
  * [53] Iandola FN, Han S, Moskewicz MW, Ashraf K, Dally WJ, Keutzer K. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and $<$ 0.5 mb model size, arXiv preprint 2016, arXiv:1602.07360.
  * [54] Ishizuka N, Weber J, Amaral DG.  Organization of intrahippocampal projections riginating from CA3 pyramidal cells in the rat.  J. Comp. Neurol. 1990;295:580–623.
  * [55] Ison MJ, Quian Quiroga R, Fried I. Rapid encoding of new memories by individual neurons in the human brain. Neuron 2015;87(1):220–230.
  * [56] Izhikevich EM, Edelman GM. Large-scale model of mammalian thalamocortical systems, PNAS 2008;105:3593–3598.
  * [57] James W. The Principles of Psychology, Vol. 1. NY: Henry Holt and Co, 1890.
  * [58] Kainen PC. Utilizing geometric anomalies of high dimension: when complexity makes computation easier. In Computer-Intensive Methods in Control and Signal Processing: The Curse of Dimensionality. New York, Springer, 1997, 283–294.
  * [59] Kainen P, Kůrková V. Quasiorthogonal dimension of Euclidean spaces. Appl. Math. Lett. 1993;6:7–10.
  * [60] Kaski S. 1998, May. Dimensionality reduction by random mapping: Fast similarity computation for clustering. In Neural networks proceedings, the 1998 IEEE international joint conference on (Vol. 1, pp. 413-418).1998. IEEE world congress on computational intelligence. IEEE.
  * [61] Khinchin AY. Mathematical Foundations of Statistical Mechanics. New York: Courier Corporation; 1949. (English translation from the Russian edition, Moscow – Leningrad, 1943.)
  * [62] Konorski J. Integrative activity of the brain; an interdisciplinary approach. Chicago: University of Chicago Press; 1967.
  * [63] Kramer MA. Nonlinear principal component analysis using autoassociative neural networks. AIChE journal 1991;37(2);233–243.
  * [64] Kreiman G, Koch C, Fried I. Category-specific visual responses of single neurons in the human medial temporal lobe. Nature Neurosci. 2000;3:946–-953.
  * Kůrková and Sanguineti [2017] Kůrková V, Sanguineti M. Probabilistic lower bounds for approximation by shallow perceptron networks, Neural Netw. 2017;91:34–41.
  * [66] Learned-Miller E, Huang GB, RoyChowdhury A, Li H, Hua G. Labeled Faces in the Wild: A Survey. In Advances in Face Detection and Facial Image Analysis, ed. by M. Kawulok, M.E. Celebi, B. Smolka, Springer, 2016, pp. 189–248.
  * [67] Ledoux M. 2001 The Concentration of Measure Phenomenon. (Mathematical Surveys & Monographs No. 89). Providence: AMS; 2005.
  * [68] Lévy P. Problèmes concrets d’analyse fonctionnelle. Paris: Gauthier-Villars, 1951.
  * [69] LFW Database: Main http://vis-www.cs.umass.edu/lfw/, (accessed 29 July 2018).
  * [70] Li XG, Somogyi P, Ylinen A, Buzsaki G.  The hippocampal CA3 network: an in vivo intracellular labeling study.  J Comp Neurol 1994;339:181–208.
  * [71] Minsky M. The society of mind. Cambridge, MA: MIT Press; 1988.
  * [72] E. Oja.  A simplified neuron model as a principal component analyzer.  Journal of Mathematical Biology 1982;15:267–273.
  * [73] Olshausen BA, Field DJ. Sparse coding of sensory inputs. Current opinion in neurobiology 2004;14(4):481–487.
  * [74] Peierls R. Model-making in physics. Contemporary Physics 1980;21(1):3–17.
  * [75] Penrose R. Shadows of the mind. Oxford: Oxford University Press; 1994.
  * [76] Perlovsky LI, Toward physics of the mind: Concepts, emotions, consciousness, and symbols, Physics of Life Reviews 2006;3:23–55.
  * [77] Plaut DC, McClelland JL. Locating object knowledge in the brain: Comment on Bowers’s (2009) attempt to revive the grandmother cell hypothesis. Psychological Review 2010;117:284–290.
  * [78] Pouget A, Dayan P, Zemel R. Information processing with population codes. Nature Rev. Neurosci. 2000;1:125–132.
  * [79] Quian Quiroga R, Fried I, Koch C. Brain cells for grandmother, Scientific American 2013;308(2):31–35.
  * [80] Quian Quiroga R, Kreiman G. Measuring sparseness in the brain: Comment on Bowers (2009). Psychological Review 2010;117(1):291–297.
  * [81] Quian Quiroga, R., Kreiman, G. Postscript: About grandmother cells and Jennifer Aniston neurons. Psychological Review 2010;117(1):297–299.
  * [82] Quian Quiroga R, Kreiman G, Koch C, Fried I. Sparse but not ‘grandmother-cell’ coding in the medial temporal lobe, Trends in Cognitive Sciences 2008;12(3): 87–91.
  * [83] Quian Quiroga R, Panzeri S. Extracting information from neuronal populations: information theory, and decoding approaches, Nature Reviews Neuroscience 2009;10:173–185.
  * [84] Quian Quiroga R, Reddy L, Kreiman G, Koch C, Fried I. Invariant visual representation by single neurons in the human brain. Nature 2005;435(7045):1102–1107.
  * [85] Ritter H, Kohonen T. Self-organizing semantic maps. Biological Cybernetics 1989;61(4):241–254.
  * [86] Roy A. An extension of the localist representation theory: grandmother cells are also widely used in the brain. Frontiers in Psychology 2013;4:300. https://doi.org/10.3389/fpsyg.2013.00300
  * [87] Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, Fei-Fei L. ImageNet Large Scale Visual Recognition Challenge, Int. J. Comput. Vis. 2015;115(3):211-252.
  * [88] F. Schroff, D. Kalenichenko, J. Philbin, FaceNet: A Unified Embedding for Face Recognition and Clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition CVPR2015, IEEE,2015, 815–823,
  * [89] Sherrington CS. Man on His Nature. Cambridge: Cambridge University Press; 1941.
  * [90] Sidorov S. On the 1-convexity of random points in the d-dimensional spherical layer, arXiv preprint 2018, arXiv:1806.04732 [math.CO].
  * [91] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition, in: International Conference on Learning Representations, arXiv preprint 2015, arXiv:1409.1556.
  * [92] Slemrod M. Hilbert’s sixth problem and the failure of the Boltzmann to Euler limit. Phil. Trans. R. Soc. A 2018;376:20170222.
  * [93] Talagrand M. Concentration of measure and isoperimetric inequalities in product spaces. Publications Mathematiques de l’IHES 1995;81:73–205.
  * [94] Tomlin AS, Turányi T, Pilling MJ. Mathematical tools for the construction, investigation and reduction of combustion mechanisms, In Low-Temperature Combustion and Autoignition, M.J. Pilling (ed.), Series Comprehensive Chemical Kinetics, Volume 35, Elsevier, 1997, 293–437.
  * [95] Tyukin IY, Gorban AN, Calvo C, Makarova J, Makarov VA. High-dimensional brain. A tool for encoding and rapid learning of memories by single neurons, Bull. Math. Biol. 2018 https://doi.org/10.1007/s11538-018-0415-5.
  * [96] Tyukin IY, Gorban AN, Sofeykov KI, Romanenko I. Knowledge Transfer Between Artificial Intelligence Systems. Frontiers in Neurorobotics 2018 https://doi.org/10.3389/fnbot.2018.00049, arXiv:1709.01547.
  * [97] Tyutchev F. Selected Poems, Translated with an Introduction and Notes by J. Dewey, The Mount, Buckhorn Weston Gillingham, Dorset: Brimstone Press, 2014.
  * Vapnik [2000] Vapnik V. The Nature of Statistical Learning Theory. Springer; 2000.
  * [99] Valdez AB, Papesh MH, Treiman DM, Smith KA, Goldinger SD, Steinmetz PN. Distributed representation of visual objects by single neurons in the human brain. Journal of Neuroscience 2015;35(13):5180–5186.
  * [100] Vincent P, Larochelle H, Bengio Y, Manzagol PA. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning. ACM. 2008, 1096–1103.
  * [101] Wigner EP. The unreasonable effectiveness of mathematics in the natural sciences. Communications in Pure and Applied Mathematics 1960;13(1):1–14.
  * [102] Wilson MA, McNaughton BL. Dynamics of the hippocampal ensemble code for space. Science 1993;261(5124):1055—1058.
  * [103] Wittner L, Henze DA, Zaborszky L, Buzsaki G. Three-dimensional reconstruction of the axon arbor of a CA3 pyramidal cell recorded and filled in vivo. Brain Struct Funct 2007;212(1):75–83.
  * [104] Zinovyev A, Mirkes E. Data complexity measured by principal graphs. Computers and Mathematics with Applications 2013;65(10);1471–1482.
  * [105] Zolotarev VM. Modern Theory of Summation of Random Variables. Utrecht, The Netherlands: Walter de Gruyter; 2011.


# An Efficient Image-to-Image Translation HourGlass-based Architecture
for Object Pushing Policy Learning

Marco Ewerton, Angel Martínez-González, Jean-Marc Odobez The authors are with
the Idiap Research Institute, CH-1920, Martigny, Switzerland {marco.ewerton,
angel.martinez<EMAIL_ADDRESS>research leading to these
results has received funding from the Swiss National Science Foundation
through the HEAP project (Human-Guided Learning and Benchmarking of Robotic
Heap Sorting, ERA-net CHIST-ERA).

###### Abstract

Humans effortlessly solve pushing tasks in everyday life but unlocking these
capabilities remains a challenge in robotics because physics models of these
tasks are often inaccurate or unattainable. State-of-the-art data-driven
approaches learn to compensate for these inaccuracies or replace the
approximated physics models altogether. Nevertheless, approaches like Deep
Q-Networks (DQNs) suffer from local optima in large state-action spaces.
Furthermore, they rely on well-chosen deep learning architectures and learning
paradigms. In this paper, we propose to frame the learning of pushing policies
(where to push and how) by DQNs as an image-to-image translation problem and
exploit an Hourglass-based architecture. We present an architecture combining
a predictor of which pushes lead to changes in the environment with a state-
action value predictor dedicated to the pushing task. Moreover, we investigate
positional information encoding to learn position-dependent policy behaviors.
We demonstrate in simulation experiments with a UR5 robot arm that our overall
architecture helps the DQN learn faster and achieve higher performance in a
pushing task involving objects with unknown dynamics. Code available at
https://github.com/idiap/hourglass_push

##  I Introduction

Pushing is an important motor skill that can serve different purposes like
moving objects that are too large, too heavy, too distant to be grasped, or in
heavily cluttered scenarios. As such, several types of pushes can help to
solve many manipulation tasks. For example, pushes can be used to isolate
objects, reorient them to improve the chances of grasping, bring them closer
or put them into a container. Pushing can also aid perception by improving
object segmentation. It is interesting to note that 25 out of the 50
manipulation tasks of the Meta-World benchmark involve pushing [1]. Numerous
works have investigated pushing tasks from several perspectives, e.g.,
creating a dataset of pushes [2] or studying and modeling their dynamics [3,
4, 5]. However, deciding whether to push or not, which push to perform, where,
and how to do it to help manipulation remains an open problem.

Figure 1: Push into Box task. The robot receives depth images as inputs and
has to decide where to push so that all the objects land inside the box. The
input depth images are orthographic projections of the images captured by the
camera on the left. Training a Deep Neural Network to process the scene and
decide which push to perform presents challenges, since the best pushes depend
on object and box positions, as well as on the physical properties of the
objects and of the table surface, which are not known by the learning agent.

To address this later problem, Zeng et al. [6] proposed an interesting
approach relying on a model-free deep reinforcement learning (RL) method (DQN)
to synergistically perform pushing and grasping. In their learning approach,
RGB-D images are mapped using two Fully Convolutional Networks (FCNs) to a
discretized action space. The first FCN maps RGB-D images to the utility of
pushing 10 cm to the right at each pixel. The second FCN maps the images to
the utility of grasping at each pixel. By presenting these networks with the
input RGB-D image rotated at 16 different angles, the networks can compute the
utilities of pushes and grasps with different orientations. For training,
Q-learning [7] is used, relying on positive rewards for successful grasps and
for pushes that produce detectable changes to the environment, resulting in
the synergistic selection of pushes and grasps. This seminal work and idea can
also be found in other papers studying skill synergies, e.g., between grasping
and throwing [8], or suction and grasping [9].

Despite their successes, these approaches present several shortcomings in
their network architectures. First, they all rely on image classification
network architectures (DenseNet [6], ResNet [8, 9]) which progressively and
drastically downscale input images before classification. In these works,
bilinear upscaling is used in the last layers of their proposed networks,
after the DenseNet or ResNet modules, considerably reducing the sampling
accuracy of the action space when applying the policy. Even if in theory every
pixel position could be sampled, selecting the action corresponding to the
maximum state-action value leads to only selecting a position at the lowest
resolution, as illustrated in Fig. 2b.

Secondly, as the same network is applied to rotated images, the network is
agnostic about orientation-specific information (it does not know which
rotated image it is processing). Although this is not a problem when selecting
which push better breaks objects apart for grasping [6], it can be problematic
when handling task direction asymmetries, as is the case in the task we are
interested in, namely, pushing objects into a box, as illustrated in Fig. 1.

Thirdly, as the chosen networks are fully convolutional, they are translation
invariant, meaning that, for instance, when processing a single object on a
table, they will likely predict very similar state-action values around this
object whatever its position on the table. Such networks cannot handle task-
dependent position asymmetries, as in our task. See Fig. 2c.

Finally, in [6], one network has to learn at the same time which pushes are
valid (i.e., alter the scene) as well as those which are effective for the
specific manipulation task (in their case, for making grasping possible).
However, given the same input image and the same action parameterization, the
set of valid pushes should be common to all tasks involving pushing (such as
isolating objects, aligning them, bringing them closer). There is no need to
learn from scratch to detect valid pushes whenever a new task involving
pushing is learned.

a)

b)  c)

Figure 2: a) Architecture used in [6] (similar in [8, 9]), in which the same
image classification network followed by upsampling is applied to 16 rotations
of the same input image. b) Sampling locations of pushes for one direction
when applying a policy learned with this architecture, showing the important
loss in action space accuracy when applying the learned policy. c) Fully
convolutional networks are translation (position) invariant. Tasks, however,
may require different outputs depending on object positions. In the example,
the cube in A should be pushed vertically, whereas the cube in B should be
pushed horizontally.

Approach and contributions.

In this paper, we address the task of pushing objects into a box whose
environment setup is shown in Fig. 1. To address this task, we adopted a
similar DQN learning framework as in [6] to learn a pushing policy using deep
convolutional neural network (CNN) agents, which entails several challenges,
as discussed above, due to orientation and position asymmetries. In this
context, we first argue that this is essentially an image-to-image translation
problem where deep CNN agents predict the state-action value for all possible
pushes given an input depth image. In this view, we propose to rely on an
Hourglass architecture that is much more suitable than image classification
networks to address such problems and is also computationally more efficient
than common U-Networks [10] used for image-to-image translation tasks. We also
solve the orientation asymmetry challenge by predicting state-action values
for all orientations in one pass. Moreover, we propose to provide as input to
the network the position of each pixel with respect to the box in addition its
depth, which could better allow the network to make decisions based on the
object location.

Secondly, in contrast to [6], we introduce a masking method that improves the
sampling efficiency by constraining the set of possible actions to those
effectively leading to changes in the environment. Our driving motivation is
to exploit the collaboration between two different CNN agents, each focusing
on different aspects of the task: pushing spot detection and task-specific
pushing state-action value prediction.

Indeed, decoupling the task with two different agents brings several benefits.
First, as our pushing spot detection agent has been trained to identify valid
regions for pushing, it increases the sampling efficiency of the state-action
value prediction agent since action sampling only needs to be conducted in a
much smaller set. Second, training each agent independently allows for
exploiting prior information regarding the task at hand and potentially
prevents them from getting trapped in local minima as often observed with end-
to-end agents. Finally, this formulation allows us to use different
architectures for the decoupled tasks, possibly increasing the complexity for
the more difficult one. In that context, our contributions are as follows:

  * •

We propose to exploit the Hourglass CNN architecture in DQNs to learn object
pushing policies, which leads to more efficient and more accurate high-
resolution state-action value prediction and push selection.

  * •

We investigate the use of positional information as network input to allow
position-dependent prediction behavior.

  * •

We propose a decoupled approach for solving the pushing task, introducing a
pushing spot detection agent that works as a masking mechanism by identifying
pushing actions leading to effective changes. Such an agent, once trained, can
be exploited in any other manipulation task requiring pushes for other
objectives.

We demonstrate in simulated experiments with a UR5 robot arm that our approach
helps our DQN learn a pushing task faster and achieve better performance than
the state-of-the-art.

##  II Related work

For pushing, robots often rely on physics models. These models are computable
but are only approximations of physical phenomena. For instance, Yu et al. [2]
present a dataset of planar pushing experiments to study how reliable these
models are, benchmark motion prediction methods, or for model learning. It
shows that pushing can be seen as a stochastic process even if a highly
precise manipulator is used to perform the pushes, highlighting the importance
of learning to tackle such tasks. This is investigated in [4] which, as in
[8], uses the concept of Residual Physics, i.e., augments analytical models
with data-driven techniques to compensate for the imperfections of the models.
In [4], the trained neural networks not only correct the model predictions but
also provide distributions over possible outcomes of actions. Hogan et al. [3]
further address the problem of pushing an object on a plane along a desired
trajectory or to pass through a sequence of via points. The proposed method
consists of using Model Predictive Control and a family of mode sequences that
have been designed by the authors. The main limitation of [3] is the necessity
of hand-designing a specific family of mode sequences for each task the robot
has to solve. In addition, all the above works only deal with single objects.

As mentioned in the introduction, Zeng et al. [6] use DQNs to synergistically
perform pushing and grasping, an idea which has also been employed to explore
synergies between other movements such as grasping and throwing of arbitrary
objects [8]. In the latter, a perception module computes a feature
representation of an RGB-D image, and a simple physics model is used to
estimate the release velocity necessary to throw an object at a certain target
position. Both pieces of information (feature representation and estimated
release velocity) are fed into the grasping and throwing modules. The grasping
module computes the utility of a grasp at each pixel. The throwing module
computes, for each possible grasp, the residual on top of the estimated
velocity to account for phenomena not accounted for by the simple physics
model. The perception, grasping, and throwing modules are fully convolutional
networks, and rewards are defined based on whether or not the thrown object
landed in the correct box. Both papers [6, 8] suffer from limitations in their
network architectures.

The methods proposed in [6] and [8] succeed in learning to push, grasp and
throw objects in part due to the discretization of the action space, which
simplifies the reinforcement learning problem. In contrast, other methods do
not require this assumption [11, 12]. For instance, [11] uses Logic-Geometric
Programming [13] and Multi-Bound Tree Search [14] to find and optimize a
sequence of continuous actions to solve tasks involving the usage of tools and
the manipulation of objects. Notwithstanding its success, some assumptions
have been made (e.g. that each object has a sphere-swept convex geometry) to
make this optimization problem numerically solvable, which may be difficult to
hold in real-world applications. We thus resorted to the DQN framework of [6]
to solve our task.

Finally, Suh and Tedrake [5] propose a switch-linear model to deal with the
problem of pushing little carrot pieces towards a certain region. They do not
deal with variable height and their images are binary, which is limiting when
working with arbitrary heaps of objects. Nevertheless, they show that a
switch-linear model can surpass the performance of deep learning models for
this task because the tested deep models can get stuck in a loop, predicting
actions that do not cause any change in the actual scene. The method we
propose allows us to handle this issue by introducing a module specialized in
identifying pushes that lead to changes in the environment. Once trained, this
module can be used in conjunction with other networks to address any pushing
task.

##  III Approach

###  III-A Overview

The proposed system is described in Fig. 3. It takes as input a depth image,
complements this information with positional information, and processes the
result with the _Push-into-Box_ architecture (termed PushInBox), which
delivers an estimate of the state-action value for each possible robot pushing
action, as parameterized by its position and orientation. At run time, the
action having the highest state-action value is executed, and the new scene is
observed.

Looking more in detail, the PushInBox network itself is composed of two
HourGlass [15] subnetworks: the PushMask subnetwork, whose aim is to spot all
the valid pushes, i.e., the pushes which result in scene changes; and the
PushReward subnetwork, whose aim is to predict the state-action value $Q$ that
results from the reward function that we have defined to tackle the pushing
task at hand.

First, we provide a more formal description of the different components of our
framework, then introduce the different rewards involved in the learning, and
finally the approach required for training the global network.

Figure 3:  Proposed push into box agent collaboration pipeline. (a) The
environment state $s_{t}$ (a depth image of the scene) is complemented with
positional information and used as input to the PushInBox network. This input
is processed by a PushReward network predicting the state-action value Q for
each possible push, and by the PushMask predicting whether a push may lead to
a scene change. (b) The pushing state-action values are masked by the outputs
of the PushMask through elementwise multiplication to only select relevant
pushes. (c) The environment is updated by applying the push that maximizes the
masked state-action value maps and produces a new environment state $s_{t+1}$.

###  III-B Modeling

State-space and network input. We represent the state $s_{t}$ of the
environment at time step $t$ by its sensed depth image ${\mathbf{I}}_{t}$ with
shape $224\times 224$. Additionally, the input image can be complemented with
2D positional information of each point on the surface of the table with
respect to the box. The image ${\mathbf{I}}_{t}$ is given as input to both
PushMask and PushReward subnetworks.

Action-space. The robot can perform a pushing action of length
$$\mathrm{1}\mathrm{0}${cm}$ at any $(x,y)$ position in the image and in any
one of $16$ directions $o$. Hence, the action set ${\cal A}$ is the set of all
the possible $(x,y,o)$ pushes and is of dimension $224\times 224\times 16$.
Each subnetwork outputs one scalar for each action. Therefore, the output of
each subnetwork is composed of $16$ layers of size $224\times 224$.

Network architecture. We use an Hourglass architecture for each of our
subnetworks to produce accurate and high-resolution predictions. Each of them
comprises a U-Net-like network [10] that processes image features in an
encoding-decoding fashion, from high-resolution to low-resolution and back to
high-resolution. The image is first processed by a set of convolution filters
to extract a feature representation of the image. In the downward part of the
network, a set of residual blocks $r_{d}$ progressively filters the features
from the previous resolution and downsamples the result, until it reaches the
bottleneck layers, which correspond to an encoding of the image at a small
resolution (128 layers at a $14\times 14$ resolution in our case). The upward
part of the network comprises a set of residual blocks $r_{u}$, each of them
processing the output from the previous (lower) resolution followed by
upsampling. Each $r_{u}$ also integrates information from the same resolution
thanks to skip connections coming from the same resolution blocks in the
downward part, allowing to combine the different semantic levels while
maintaining the localization accuracy of the network prediction.

###  III-C Rewards

We introduce reward functions specific to each subtask, i.e., valid pushing
spot detection and pushing objects into the box.

Pushing spot detection reward. We design this reward under the premise that
pushing actions should produce _effective changes_. More precisely, given the
environment state at time $t$, $s_{t}$ (the depth image ${\mathbf{I}}_{t}$),
and the executed action $a_{t}$ leading to the next state $s_{t+1}$ (image
${\mathbf{I}}_{t+1}$), we define a notion of change by counting the amount of
local pixel changes

${c}_{t}=\sum_{p}I_{(|{\mathbf{I}}(p)_{t}-{\mathbf{I}}(p)_{t+1}|>=$\mathrm{1}${cm})},$
(1)

where $I_{v}$ is the indicator function, equal to 1 if $v$ is true, and 0
otherwise. Accordingly, we define the PushMask change reward

$R^{c}\left(a_{t},s_{t},s_{t+1}\right)=\begin{cases}0&\text{ if
}{c}_{t}<{\tau_{mask}}\\\ 1&\text{otherwise}\end{cases},$ (2)

where ${\tau_{mask}}$ is a threshold which we set to 1000 in practice.

Push-into-box reward. We define an effective push as that which moves the
objects towards the box, and hence will minimize the average distance between
object pixels and the box. Accordingly, our measure of pushing effectiveness
is defined as:

$\Delta d^{M}_{t}=d^{M}_{t}-d^{M}_{t+1}\mbox{ with }d^{M}_{t}=\frac{1}{|{\cal
O}_{t}|}\sum_{p\in{\cal O}_{t}}d_{t}(p),$ (3)

where $p$ is a pixel position in the image, $d_{t}(p)=\|p-p_{box}\|$ denotes
the distance between pixel $p$ and the center of the edge of the box in
contact with the table $p_{box}$, ${\cal O}_{t}$ is the set of objects’ pixels
obtained by thresholding the scene depth image, and $|{\cal O}_{t}|$ denotes
its cardinality. Finally we define the push-into box reward $R^{p}$ as the sum
of a distance reward $R^{d}_{t}=\max\left(0,\Delta d^{M}_{t}\right)$ measuring
how much objects move closer to the box, as well as a reward
$R^{ib}_{t}=10\times N^{box}_{t}$ proportional to the number of objects
$N^{box}_{t}$ falling into the box, unless one or more objects fall to the
ground ($N^{gro}_{t}>0$) where $N^{box}_{t}$ and $N^{gro}_{t}$ are provided by
the environment simulator. More formally,

$R^{p}\left(a_{t},s_{t},s_{t+1}\right)=\begin{cases}0&\text{if
}N^{gro}_{t}>0\\\ R^{d}_{t}+R^{ib}_{t}&\text{otherwise}\end{cases}.$ (4)

###  III-D Training protocol

In this section, we indicate the protocol followed to train our architecture,
which involves both supervised training and reinforcement learning steps. The
main steps are as follows:

  1. 1.

Train PushMask using appropriate data. The parameters of this network are
assumed to be frozen afterward.

  2. 2.

Train PushReward using an offline dataset $D_{rew}$.

  3. 3.

Train the whole PushInBox using online reinforcement learning, taking
advantage of the PushMask to improve training. In this step, the parameters of
PushMask are frozen.

Before describing the specific aspects of how the PushMask, PushReward, and
PushInBox models were trained, we introduce the basic elements of
reinforcement learning.

Reinforcement learning (RL). During RL, the system simulates a set of $B$
experiences, which can be expressed as triplets $(s_{t},a_{t},s_{t+1})$. For
each experience, we can compute the expected state-action value
$Q(a_{t},s_{t})$ as predicted by the network. The output $Q(a_{t},s_{t})$
should be as close as possible to the target value

$\displaystyle y\left(a_{t},s_{t},s_{t+1}\right)=$ $\displaystyle
R\left(a_{t},s_{t},s_{t+1}\right)+\gamma
Q\left(s_{t+1},\underset{a^{\prime}}{\arg\max}\left(Q\left(a^{\prime},s_{t+1}\right)\right)\right),$
(5)

which can be computed from the definition of the reward and
$\underset{a^{\prime}}{\arg\max}\left(Q\left(a^{\prime},s_{t+1}\right)\right)$.
The latter can be computed by applying the network to the state $s_{t+1}$ of
the current experience. In other words, for an experience $i$, the temporal
difference error
$\delta^{i}_{t}=L_{e}(Q\left(a^{i}_{t},s^{i}_{t}\right),y\left(a^{i}_{t},s^{i}_{t},s^{i}_{t+1}\right))$
must be minimized, where $L_{e}$ is the loss used to measure the discrepancy
between the prediction and the target. The loss for each mini-batch of size
$B$ is $Loss=\sum_{i=1}^{B}\delta^{i}_{t}$, which can be optimized via
gradient descent optimization algorithms and using backpropagation to compute
the gradients. We relied on the Adam optimizer.

Training PushMask. To train the PushMask we first generate a training set
$D_{m}$ using a heuristics that identifies pixels close to the borders of
objects as promising or not to push according to edge and orientation
analysis. More precisely, we first generate a set of scenarios by tossing 10
randomly selected objects onto the table and generating their corresponding
depth images. To compute the ground truth, we start by detecting edges using
the Canny edge detector. Then, we label an edge pixel $p$ to be a valid
pushing position if ${\mathbf{I}}(p_{\theta})-{\mathbf{I}}(p)>\delta$, where
$p_{\theta}$ is the pixel $p$ moved in a line along the pushing orientation
possibility $\theta$. The reasoning behind this procedure is that a push
starting at $p$ and passing through $p_{\theta}$ will probably meet an object
border if $p_{\theta}$ is higher than $p$ by a certain threshold $\delta$,
meaning that this push will probably produce some significant change. Fig.
4(a) shows an example of the ground truth generated with this heuristics for a
given image.

Our goal is to train a network, PushMask, which identifies whether a given
push $a$ is leading to a change or not. In this regard, we have a binary
classification problem for each $a$, and we thus rely on the binary cross-
entropy loss $L_{e}$ for training. The network is trained in two stages. In
the first stage, we train PushMask using the dataset $D_{m}$, i.e., the ground
truth masks generated by the described heuristics. However, as this heuristics
does not account well for setup characteristics (gripper shape and size,
friction, noise), we run a refinement training stage (in an RL fashion) that
uses actual object pushes and Eq.(2) as ground truth for supervised learning.
Fig. 4(b) illustrates our results on a sample image.

Figure 4:  (a) Example of generating ground truth with our proposed
heuristics. Top: Input depth image; Bottom: generated ground truth push mask
for pushes pointing to the right. (b) Example of generated valid pushing
probability maps predicted by our PushMask for different pushing orientation
possibilities on a sample image. The color red indicates start positions of
pushes with a higher probability of leading to changes, while blue indicates a
lower probability of leading to changes.

Training PushReward and PushInBox.

We divide the PushInBox training into two stages. In each stage, the network
is trained to minimize the temporal difference errors $\delta_{t}^{i}$.

In the first stage (offline RL) we train PushReward from scratch from a
dataset $D_{rew}$ of experiences obtained as follows: scenarios are generated
at random, and the pushing actions are sampled uniformly at random 50% of the
time, or using a trained PushMask otherwise. Specifically, in the latter case,
we compute the forward pass of the PushMask on the current depth image and
generate a binary mask of pushing actions. Then, we select pushing actions
that lead to a change in the environment with probability greater than a
certain threshold. These actions are executed in simulation and the
corresponding experiences are recorded. We have chosen the probability
threshold to select the actions based on the F-score obtained by different
thresholds in a test set, as detailed in Section IV. Subsequently, PushReward
is trained to minimize the temporal difference errors associated with the
recorded experiences.

In the second stage (online RL) we further train PushReward in simulation,
using the previously trained PushReward model as the starting point and
exploring new experiences with an epsilon-greedy approach. We proceed as
follows: given each new depth image experienced in simulation, we compute the
forward pass of PushReward and PushMask. These predictions are then combined
via the Hadamard product. After each new batch of experiences, the parameters
of PushReward are optimized to minimize the squared error loss $L_{e}$
associated with the experiences in this new batch.

The offline RL stage allows for an effective initialization of PushReward with
pre-recorded experiences while the online RL stage further improves the
network by exploring new experiences. In this work, for simplicity, PushMask
is trained and its parameters are frozen before training PushReward. One could
also consider training PushMask further together with PushReward.

##  IV Experiments

###  IV-A Experimental protocol

Experimental protocol for PushMask. As this network is trained independently,
we generated a test dataset to evaluate the performance of the different
models. To that end, we generated a test set by randomly sampling 10240
scenarios involving 1 to 10 objects, along with random pushes. The ground-
truth (GT) regarding these pushes has been determined as during training
(i.e., thresholding the number of pixels that have changed after the push).
This test set was used to produce precision-recall curves. The precision is
defined as: out of the set of pushes identified by the system as valid (true),
what is the proportion of pushes that are true (according to the GT). The
recall corresponds to the proportion of true pushes correctly identified by
the system.

Push-into-box experimental protocol. We created 100 random scenarios
comprising exactly 10 objects. For each model under evaluation, we applied the
corresponding policy, until either there was no more object on the table, or
the policy did not produce any change 10 times in a row (implicitly implying
that the model is pushing in the void).

Evaluation Metrics. We quantify the performance of our models by the mean and
standard deviation of the number of objects successfully pushed into the box
$N_{ObjB}$, number of objects that fell on the ground $N_{ObjG}$, and number
of objects left on the table $N_{ObjT}$. Additionally, we measure the
efficiency of our system by computing the ratio between the average number of
objects successfully pushed into the box $N_{ObjB}$ and the average number of
executed actions $N_{Act}$.

Tested models. To evaluate the benefits of the different parts of our
framework, we tested different algorithmic configurations. These elements are
specified when reporting results in Table I.

  * •

DenseNet architecture [6]: in this case, we used the architecture proposed by
[6] as PushReward network, which is based on a DenseNet image recognition
network pre-trained on ImageNet [16], with an improvement. Rather than
predicting the reward for the 16 pushing directions by rotating the image 16
times and applying the DensetNet architecture, we modified the network so that
it can predict the reward for all directions in the last stage of the network,
as is done with our architecture.

  * •

HGlassN architecture: for the PushReward network.

  * •

HGlassN-Pos architecture: the same as HGlassN, but using an additional
position encoding for the network input.

  * •

Mask option: this indicates whether the PushMask network was exploited by the
system, both during training in the online RL step and in testing.

  * •

OnlineRL: whether the PushReward network was only trained with the offline
dataset $D_{rew}$ or if an additional RL fine-tuning with an epsilon-greedy
exploration strategy was conducted (OnlineRL = Yes).

###  IV-B Results

Figure 5: Precision vs. recall curves of PushMask before fine-tuning (trained
only based on a heuristics) vs. after fine-tuning with actual pushes. The
model with probability threshold $0.14$ obtained the best F-score.

Figure 6:  Comparison of output state-action value maps for (a) Densenet, and
(b) Hourglass architectures. The smoothing effects in (a) appear by producing
the maps at very low resolution. A more precise localization of pushing action
can be identified in the maps produced by the Hourglass network in higher
resolution.

PushMask results. As shown in Fig. 5, training based only on the masks
generated by the heuristics led to high precision but resulted in missing some
of the pushes which produced changes (low recall). In this regard, the
PushMask benefited from the fine-tuning with actual pushes. The fine-tuning
led to a higher recall and a higher F-Score. The probability threshold of 0.14
to detect changes has achieved the highest F-Score and has been used in our
further experiments involving the PushMask.

PushInBox results. Our main quantitative results are summarized in Table I.
The best result in each column is bold, the second-best, underlined. All the
HourGlass-based models have surpassed the DenseNet-based ones in terms of the
average number of objects correctly pushed into the box. Figs. 6 and 7 show
state-action value predictions made by different models.

Comparing the HGlassN models with and without the mask after online RL, we can
observe that the mask only slightly improved the performance in terms of the
average number of objects in the box and efficiency (fewer actions).
Nevertheless, the mask has given the HGlassN-Pos model a significant boost in
performance before online RL ($8.05$ objects on average in the box with the
mask versus $6.71$ without). We hypothesize that the mask helps to prevent
models from sliding on the top of objects or pushing where there is no object.

The model with the highest performance overall has been the HGlassN with mask
and after online RL with discount factor $\gamma=0$. By observing the behavior
of this model during tests, we noticed that the main sources of problems
preventing the model from pushing all the objects into the box are objects
very close to the borders of the table, where the robot is unable to move them
back to the center due to its limited workspace, and cylinders, which easily
roll out of the table.

We had expected models with discount factor $\gamma=0.4$ to perform better
than the models with $\gamma=0$, but this has not been the case. A possible
explanation is that this task does not require learning complex successions of
pushes to obtain higher expected future rewards.

Finally, while the model with linear position encoding performed best in
offline learning, exhibiting a better behavior for carefully bringing objects
in front of the box before pushing them into the box, its performance after
online RL decreased. A closer look at the learning curve showed a relatively
chaotic behavior beyond a certain number of epochs, suggesting that the
selected optimizer and parameters are not optimal. This will be investigated
in future work.

TABLE I:  Push into the box performance results obtained by testing our model in scenarios with 10 objects  Architecture | OnlineRL | Mask |  $N_{ObjB}$ (std) |  $N_{ObjG}$ (std) |  $N_{ObjT}$ (std) | $N_{Act}$ | $\frac{\mbox{avg.}N_{ObjB}}{\text{avg.}N_{Act}}$
---|---|---|---|---|---|---|---
DenseNet | No | No | $4.56(2.1)$ | $4.38(1.7)$ | $1.06(2.2)$ | $\mathbf{18.0}(8.4)$ | $0.25$
DenseNet | Yes | No | $5.38(3.2)$ | $1.17(1.2)$ | $3.45(3.9)$ | $21.5(8.4)$ | $0.25$
HGlassN | No | Yes | $7.80(1.2)$ | $2.15(1.1)$ | $0.05(0.4)$ | $23.7(4.8)$ | $0.33$
HGlassN | No | No | $7.76(1.4)$ | $1.90(1.3)$ | $0.34(1.2)$ | $26.6(9.6)$ | $0.29$
HGlassN-Pos | No | Yes | $8.05(1.7)$ | $1.48(1.2)$ | $0.47(1.6)$ | $28.2(10.8)$ | $0.29$
HGlassN-Pos | No | No | $6.71(2.6)$ | $1.21(1.3)$ | $2.08(3.0)$ | $30.0(12.6)$ | $0.22$
HGlassN, $\gamma=0$ | Yes | Yes | $\mathbf{8.96}(0.9)$ | $\underline{1.01}(0.9)$ | $\underline{0.03}(0.2)$ | $21.5(4.6)$ | $\mathbf{0.42}$
HGlassN, $\gamma=0$ | Yes | No | $\underline{8.94}(1.0)$ | $\mathbf{1.00}(1.0)$ | $0.06(0.4)$ | $23.2(6.7)$ | $\underline{0.39}$
HGlassN, $\gamma=0.4$ | Yes | Yes | $8.00(1.4)$ | $1.98(1.4)$ | $\mathbf{0.02}(0.1)$ | $\underline{20.6}(6.9)$ | $\underline{0.39}$
HGlassN, $\gamma=0.4$ | Yes | No | $7.61(1.6)$ | $2.11(1.2)$ | $0.28(1.4)$ | $21.1(5.3)$ | $0.36$
HGlassN-Pos, $\gamma=0$ | Yes | Yes | $6.66(1.8)$ | $2.89(1.4)$ | $0.45(1.8)$ | $24.2(7.0)$ | $0.28$
HGlassN-Pos, $\gamma=0.4$ | Yes | Yes | $6.59(1.7)$ | $3.16(1.5)$ | $0.25(1.3)$ | $24.0(8.8)$ | $0.27$

a)

b)

Figure 7:  Examples of state-action value map predictions for a set of pushing
directions. a) HGlassN architecture, using the PushMask network, and OnlineRL
with $\gamma=0$. b) Same as a), but with $\gamma=0.4$. A white arrow shows the
direction of the pushing action, the magenta circle center indicates the
location of the maximum for that state-action value map and the golden star
indicates the position of the center of the edge of the box in contact with
the table.

##  V Conclusion

In this paper, we presented an approach for object pushing policy learning
from visual data. We have addressed the task as an image-to-image translation
and proposed to use Hourglass architectures to produce high-resolution state-
action value maps given an input depth image. We divided the task into valid
pushing spot detection and pushing state-action value prediction subtasks, and
designed their rewards so that pushing actions effectively produce changes in
the environment and solve the task at hand, in this case pushing objects into
a box. Our results suggest that the use of the Hourglass architecture is
better suitable for predicting more accurate pushing actions and is more
efficient than other classification-based architectures such as DenseNet.

## References

  * [1] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine, “Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning,” PMLR, pp. 1094–1100, 2020.
  * [2] K.-T. Yu, M. Bauza, N. Fazeli, and A. Rodriguez, “More than a million ways to be pushed. A high-fidelity experimental dataset of planar pushing,” in _IROS_. IEEE, 2016, pp. 30–37.
  * [3] F. R. Hogan and A. Rodriguez, “Feedback control of the pusher-slider system: A story of hybrid and underactuated contact dynamics,” _Algorithmic Foundations of Robotics XII_ , pp. 800–815, 2020.
  * [4] A. Ajay, J. Wu, N. Fazeli, M. Bauza, L. P. Kaelbling, J. B. Tenenbaum, and A. Rodriguez, “Augmenting physical simulators with stochastic neural networks: Case study of planar pushing and bouncing,” in _IROS_. IEEE, 2018, pp. 3066–3073.
  * [5] H. Suh and R. Tedrake, “The surprising effectiveness of linear models for visual foresight in object pile manipulation,” _arXiv preprint arXiv:2002.09093_ , 2020.
  * [6] A. Zeng, S. Song, S. Welker, J. Lee, A. Rodriguez, and T. Funkhouser, “Learning synergies between pushing and grasping with self-supervised deep reinforcement learning,” in _IROS_. IEEE, 2018, pp. 4238–4245.
  * [7] C. J. Watkins and P. Dayan, “Q-learning,” _Machine learning_ , vol. 8, no. 3-4, pp. 279–292, 1992.
  * [8] A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. Funkhouser, “Tossingbot: Learning to throw arbitrary objects with residual physics,” _T-RO_ , vol. 36, no. 4, pp. 1307–1319, 2020.
  * [9] A. Zeng, S. Song, K.-T. Yu, E. Donlon, F. R. Hogan, M. Bauza, D. Ma, O. Taylor, M. Liu, E. Romo, _et al._ , “Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching,” in _ICRA_. IEEE, 2018, pp. 3750–3757.
  * [10] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [11] M. Toussaint, K. Allen, K. A. Smith, and J. B. Tenenbaum, “Differentiable physics and stable modes for tool-use and manipulation planning.” in _RSS_ , 2018.
  * [12] R. Strudel, A. Pashevich, I. Kalevatykh, I. Laptev, J. Sivic, and C. Schmid, “Learning to combine primitive skills: A step towards versatile robotic manipulation,” _ICRA_ , pp. 4637–4643, 2020.
  * [13] M. Toussaint, “Logic-geometric programming: An optimization-based approach to combined task and motion planning,” in _Twenty-Fourth International Joint Conference on Artificial Intelligence_ , 2015.
  * [14] M. Toussaint and M. Lopes, “Multi-bound tree search for logic-geometric programming in cooperative manipulation domains,” in _ICRA_. IEEE, 2017, pp. 4044–4051.
  * [15] A. Newell, K. Yang, and J. Deng, “Stacked hourglass networks for human pose estimation,” in _ECCV_. Springer, 2016, pp. 483–499.
  * [16] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _CVPR_. IEEE, 2009, pp. 248–255.


# Inflationary resolution of the initial singularity

Damien A<EMAIL_ADDRESS>Joseph E<EMAIL_ADDRESS>Department of Physics, Arizona State University, Tempe, AZ 85287-1504, USA

(August 28, 2024)

###### Abstract

We construct geodesically complete inflationary models, addressing and
overcoming existing no-go theorems related to past eternal inflation. We show
that a period of accelerated expansion is indispensable for achieving
nontrivial, geodesically complete (Generalized) Friedmann-Robertson-Walker
spacetimes. We show within the framework of General Relativity, the
persistence of eternal inflation comes at the expense of violating the null
energy condition.

_Introduction_ –The inflationary universe paradigm is a cornerstone of modern
cosmology Guth (1981); Linde (1982); Albrecht and Steinhardt (1982). A
prevailing view asserts that inflationary scenarios cannot be past-eternal
even at the classical level, a conclusion drawn independent of the energy
conditions involved. Specifically, the general notion stems from the now
renowned work of Borde, Guth and Vilenkin (BGV) Borde  _et al._ (2003), who
stated that a cosmological model which is inflating – or just expanding
sufficiently fast – must be incomplete in null and timelike past directions.
This belief has led to the strong statement that inflationary models
necessitate physics other than inflation to describe the past boundary of the
inflating region of a spacetime, and even to the perspective that, not only
inflation must have had a beginning, but the universe itself must have
originated from a definite beginning.

In this Letter, we challenge this perspective by explicitly constructing
geodesically-complete eternal inflationary models. Furthermore, we show that
all non-trivial geodesically-complete Friedmann Robertson Walker (FRW)
spacetimes necessarily require an epoch of accelerated expansion. This finding
underscores the critical role of acceleration and inflation in the formation
and evolution of a geodesically-complete universe. 333In this context, non-
trivial implies an evolving non-constant scale factor. Spacetimes homothetical
to Minkowski spacetime form a cohort of counter-examples to many propositions
discussed in this Letter.

We show the price of this eternal inflation in a universe with non-trivial
evolution in standard General Relativity (GR), is a required period in which
the null energy condition is violated. Further details concerning inflationary
as well as geodesically-complete bouncing and loitering models, which may
require only an arbitrarily short period of accelerated expansion, are
discussed in a companion paper Easson and Lesnefsky (2024).

_Eternal inflating universe_ –We begin with a detailed analysis of the model
introduced in Lesnefsky  _et al._ (2023), having scale factor:

$\displaystyle a(t)=a_{0}\exp[2t/\alpha]+c\,,$ (1)

for constants $a_{0}$, $\alpha$ and $c$. 444An earlier model which shares some
feature of the above was discussed in Ellis and Maartens (2004). For this
scale factor, $2\alpha^{-1}\neq H=\dot{a}/a$; and $c>0$ is required to
construct a geodesically complete spacetime. In what follows, including plot
parameters, we take $a_{0}=c=1/2$. With these values it is easy to show Eq. 1
may equivalently be expressed as

$a(t)=\exp[t/\alpha]\cosh[t/\alpha]\,.$ (2)

The Hubble parameter $H=\dot{a}/a$ is given by:

$H=\frac{2a_{0}e^{\frac{2t}{\alpha}}}{(c+a_{0}e^{\frac{2t}{\alpha}})\alpha}\,.$
(3)

This is an example of an eternally inflating spacetime, as is easily seen from
the calculation of

$\frac{\ddot{a}}{a}=4\alpha^{-2}\left(1-\frac{c}{c+a_{0}e^{\frac{2t}{\alpha}}}\right)\,.$
(4)

Figure 1: Evolutionary behavior for Eq. 1 with $\alpha=1$. The scale factor is
plotted in red. The Hubble parameter (multiplied by 100) is in green. The co-
moving Hubble radius is in blue. The Ricci scalar is plotted dashed and the
Kretchmann scalar is dotted.

The acceleration is positive for all $t\in(-\infty,+\infty)$ for all non-zero
values of $\alpha$ and for $c>0$. For $\alpha>0$, the universe is expanding
and inflating for all $t$, asympotically approaching Minkowski spacetime as
$t\rightarrow-\infty$. 555We have previously suggested a canonical definition
for inflation in Lesnefsky  _et al._ (2023), see Defn. 1. This definition
includes arbitrarily short periods of accelerated expansion. For $\alpha<0$
the universe is contracting and inflating for all time, approaching zero
acceleration as $t\rightarrow\infty$.

The cosmological evolution is depicted in Fig. 1. Shown are the scale factor
$a(t)$, Hubble parameter $H$ and co-moving Hubble radius $H^{-1}/a$. When the
co-moving Hubble radius is decreasing the spacetime is inflating–in this case
it is decreasing for all $t\in(-\infty,+\infty)$.

The absence of curvature singularities is confirmed by the finite curvature
invariants, as shown in Fig. 1. We plot the Ricci scalar $R$ and the
Kretchmann scalar $K=R_{\mu\nu\rho\sigma}R^{\mu\nu\rho\sigma}$ built from the
Riemann tensor. Both are observed to be finite for all $t$

$\displaystyle R$ $\displaystyle=$
$\displaystyle\frac{24a_{0}e^{\frac{2t}{\alpha}}(c+2a_{0}e^{\frac{2t}{\alpha}})}{(c+a_{0}e^{\frac{2t}{\alpha}})^{2}\alpha^{2}}\,,$
$\displaystyle K$ $\displaystyle=$
$\displaystyle\frac{192a_{0}^{2}e^{\frac{4t}{\alpha}}(c^{2}+2a_{0}ce^{\frac{2t}{\alpha}}+2a_{0}^{2}e^{\frac{4t}{\alpha}})}{(c+a_{0}e^{\frac{2t}{\alpha}})^{4}\alpha^{4}}\,.$
(5)

_Geodesic Completeness_ –The canonical mathematical objects to discuss the
geodesic completeness of inflationary spacetime models are generalized
Friedmann-Robertson-Walker (GFRW) spacetimes, of which FRW models are an
exceptional subset. A time dimension of $\mathbb{R}^{1}_{1}$ is warped with
smooth strictly positive scale factor $a>0$ to any geodesically complete
Riemannian manifold $\left(\Sigma,g_{\Sigma}\right)$ constituting the purely
spacelike foliation.666We use the hyperquadratic notation $\mathbb{R}^{n}_{k}$
of O’Neill (1983). The space $\mathbb{R}^{n}_{k}$ is manifold $\mathbb{R}^{n}$
imbued with the diagonal metric
$\\{-1,\overset{k-2}{\cdots},-1,1,\overset{n-k-2}{\cdots},1\\}$. The
Lorentzian manifold $\mathbb{R}^{1}_{1}=\left(\mathbb{R},-dt^{2}\right)$ and
de Sitter is the unit psuedosphere in $\mathbb{R}^{n+1}_{1}$. The warped
product formalism of Bishop and O’Neill (1969) is utilized. The FRW spacetime
has $\Sigma=S_{k}$ a spaceform of constant sectional curvature $k$. The
criteria for geodesic completeness of a GFRW appeared in Lesnefsky  _et al._
(2023):

######  Theorem 1.

(LED)–Let $\mathcal{M}=\mathbb{R}^{1}_{1}\times_{a}\Sigma$ be a GFRW
spacetime.

  1. 1.

The spacetime $\mathcal{M}$ is future timelike complete iff
$\int_{t_{0}}^{\infty}\frac{a\left(t\right)dt}{\sqrt{\left(a\left(t\right)\right)^{2}+1}}$
diverges for all $t_{0}\in\mathbb{R}$.

  2. 2.

The spacetime $\mathcal{M}$ is future null complete iff
$\int_{t_{0}}^{\infty}a\left(t\right)dt$ diverges for all
$t_{0}\in\mathbb{R}$.

  3. 3.

The spacetime $\mathcal{M}$ is future spacelike complete iff $\mathcal{M}$ is
future null complete or if $\int_{t_{0}}^{\infty}a\left(t\right)dt<\infty$
then $a$ is unbounded; for all $t_{0}\in\mathbb{R}$.

  4. 4.

The GFRW is past timelike / null / spacelike complete if, for items 1-3 above,
upon reversing the limits of integration from $\int_{t_{0}}^{\infty}$ to
$\int_{-\infty}^{t_{0}}$ the word “future” is replaced by “past”.

  5. 5.

The spacetime $\mathcal{M}$ is _geodesically complete_ iff it is both future
and past timelike, null, and spacelike geodesically complete.

Unlike other theorems such as the BGV theorem Borde  _et al._ (2003) to be
discussed below, which report only to show geodesic incompleteness, Thm. 1
represents a significant advancement, offering a concrete method for
ascertaining the geodesic completeness, or incompleteness, of a specific
Generalized Friedmann-Robertson-Walker (GFRW) spacetime.

For the scale factor of Eq. 1, it is possible to explicitly calculate the
integrals of Thm. 1. Assuming $c>0$ and $a_{0}>0$, we find for the indefinite
integrals:

$\displaystyle\int^{t}\frac{a(\zeta)}{\sqrt{(a(\zeta))^{2}+1}}\,d\zeta=\frac{1}{2}\alpha\arcsin\left(c+a_{0}e^{\frac{2t}{\alpha}}\right)$
$\displaystyle+\frac{\alpha\,c}{\sqrt{1+c^{2}}}\arctan\left(\frac{a_{0}e^{\frac{2t}{\alpha}}-\sqrt{1+\left(c+a_{0}e^{\frac{2t}{\alpha}}\right)^{2}}}{\sqrt{1+c^{2}}}\right)$
(6)

and

$\int^{t}a\left(\zeta\right)d\zeta=ct+a_{0}\frac{\alpha}{2}e^{\frac{2t}{\alpha}}\,.$
(7)

It is easy to show the above integrals diverge over the full set of conditions
discussed in Thm. 1 for all (non-zero) values of $\alpha$; hence, the
spacetime with scale factor Eq. 1 is geodesically complete.

We have thus established that the FRW spacetime, characterized by scale factor
Eq. 1, is geodesically complete, eternally inflating, and directly challenges
the prevailing view highlighted in our introduction; albeit, at the expense of
energy condition violation. In fact, such models are not difficult to
construct and further eternal models are discussed in our companion paper
Easson and Lesnefsky (2024).

Figure 2: Energy conditions from Eq. Inflationary resolution of the initial
singularity. Plot of energy density $\rho$ (red), $\rho+p$ (yellow), $|p|$
(blue) and $\rho+3p$ (green).

_Energy conditions_ –While both Thm. 1 and the BGV theorem report to hold
independent of the energy conditions, such an analysis is informative and we
now exam the energy conditions for the model given by Eq. 1. Calculation of
the Einstein tensor yields non-vanishing components:

$\displaystyle G_{tt}$ $\displaystyle=$
$\displaystyle\frac{12a_{0}^{2}e^{\frac{4t}{\alpha}}}{(c+a_{0}e^{\frac{2t}{\alpha}})^{2}\alpha^{2}}\,,$
$\displaystyle G_{ii}$ $\displaystyle=$
$\displaystyle-\frac{4a_{0}e^{\frac{2t}{\alpha}}(2c+3a_{0}e^{\frac{2t}{\alpha}})}{\alpha^{2}}\,.$
(8)

The energy density is given by $\rho=-G^{t}{}_{t}$ and the pressure is
$p=G^{i}{}_{i}$. A plot elucidating the energy conditions is given in Fig. 2.

To interpret this plot we recall that in the standard perfect-fluid treatment
of matter where $p$ and $\rho$ are the pressure and energy density of the
fluid respectively, the _energy conditions_ in a cosmological setting are as
follows S.W.Hawking and G.F.R.Ellis (1973): Weak energy condition (WEC):
$\rho\geq 0$ and $\rho+p\geq 0$, Null energy condition (NEC): $\rho+p\geq 0$,
Dominant energy condition (DEC): $\rho\geq|p|$, and Strong energy condition
(SEC): $\rho+p\geq 0$ and $\rho+3p\geq 0$. Note that the DEC$\implies$WEC,
WEC$\implies$NEC, SEC$\implies$NEC; and SEC$\hbox
to0.0pt{$\quad\not$\hss}\implies$WEC.

Our stance on employing energy conditions as a stringent basis for critiquing
models is marked by ambivalence. Notably, all classical energy conditions are
unequivocally breached by quantum effects, a fact supported both
experimentally and theoretically, as exemplified by the Casimir Effect Casimir
(1948) and outlined in Epstein  _et al._ (1965). Consequently, we entrust the
assessment of solution viability to the discernment of our readers, refraining
from making definitive judgments Barcelo and Visser (2002); Chatterjee  _et
al._ (2013).

From Fig. 2 we see that all of the above energy conditions are violated for
the eternal inflationary model given by Eq. 1. Violation of the NEC is
affirmed by the yellow curve dipping below the horizontal axis. The NEC is
saturated at early and late times since, with our plot parameters,
$\rho+p=-2\operatorname{sech}^{2}{t}\rightarrow 0$ as $t\rightarrow\pm\infty$.
As the model is eternally inflating, $\ddot{a}>0$ $\forall$
$t\in(-\infty,\infty)$, the SEC is violated as seen from (eternal negativity
of) the green curve. The model approaches flat Minkowski spacetime at
$t=-\infty$.

We have determined that the price one must pay for this eternal, non-singular,
inflating universe in the context of General Relativity is violation of the
NEC. While such violation is typically associated with model instabilities, in
some instances (arguably outside the context of conventional GR) the NEC may
be violated in a stable way Dubovsky  _et al._ (2006); Nicolis  _et al._
(2010); Kobayashi  _et al._ (2010); Easson  _et al._ (2011); Easson and Manton
(2019).

Our presented spacetime is both geodesically complete and eternally inflating
and directly contradicts the notion that a cosmological model which is
inflating must be incomplete (regardless of energy condition violation).
Hence, certain models featuring an inflationary phase possess the ability to
bypass the initial cosmological singularity.

_BGV Theorem_ –As the above findings may come as a surprise to some readers
given the famous no-go theorem of BGV, we now explain how the theorem is
obviated. The theorem may be quantified by Eq. 5 of Borde  _et al._ (2003),
which we formalize as in Lesnefsky  _et al._ (2023):

######  Theorem 2.

_(BGV)_ –Let $\left(\mathcal{M},g\right)$ be a spacetime, and
$\gamma\,:\,\left[\lambda_{i},\lambda_{f}\right]\rightarrow\mathcal{M}$ be
some causal geodesic. If the quantity

$H_{avg}^{\gamma}=\frac{1}{\lambda_{f}-\lambda_{i}}\int_{\lambda_{i}}^{\lambda_{f}}H^{\gamma}\left(\zeta\right)d\zeta$
(9)

is strictly positive along the image of $\gamma$, the spacetime is
geodesically-incomplete.

Let us apply the BGV theorem to the eternal inflationary model given by Eq. 1.
Without loss of generality we take $a_{0}=1$, $t_{i}<0$, and $t_{f}=0$, and
select any connected interval $\left[t_{i},0\right]$ where the boundary is
actually realized:

$H_{avg}^{\gamma}=\frac{1}{-t_{i}}\int_{t_{i}}^{0}Hdt=\frac{1}{-t_{i}}\ln\left(\frac{1+c}{e^{\nicefrac{{t_{i}}}{{\alpha}}}+c}\right)>0$
(10)

because, $1+c>e^{\nicefrac{{t_{i}}}{{\alpha}}}+c$. Thus, direct calculation of
Eq. 9, yields $H_{avg}>0$, yet despite Thm.2, the model is geodesically
complete per Thm. 1. What is further troubling for Thm. 2 is that Eq. 9 is
computed over compact intervals. A proper discussion of geodesic completeness
should involve maximal geodesics or maximal geodesic rays, as we have shown
above. A geodesic defined over a compact interval is inherently incomplete, as
it must inevitably encounter a singularity or boundary at its endpoint,
rendering it categorically incomplete yet inextendable, or it is
straightforwardly extendable in a (possibly small but non-empty) open
neighborhood of the endpoint by the exponential map Lesnefsky  _et al._
(2023).

One may attempt to salvage Thm. 2 by considering the case where a limit is
taken; for example, the limit as $t_{i}\rightarrow-\infty$, although, such
limits were not explicitly discussed in Borde  _et al._ (2003). This would
include the case where geodesics are maximal into the past, like a past
directed geodesic ray or a full maximal geodesic, maximally extended into both
the past and future. In such case Eq. 10 yields $H_{avg}=0$, and one may then
argue that the BGV theorem does not apply. This result is due to the cofinite
interval suppression of $1/(-t_{i})$ in the integral. We find this conclusion
dissatisfying, given that over any finite interval, $H>0$. In fact, depending
on the behavior of the scale factor, how intervals are selected, and how the
limiting process is executed, one can calculate a _continuum_ of values for
$H_{avg}$ including both zero and positive values - thus Thm. 2 both
simultaneously applies and does not apply despite the geodesic completeness of
the spacetime being fully established by Thm. 1. Further concerns pertaining
to the above are detailed in Lesnefsky  _et al._ (2023). For recent
developments in this area see Kinney  _et al._ (2023); Geshnizjani  _et al._
(2023).

We now discuss two important Corollaries to Thm. 1, the proofs of which are
relegated to our companion paper Easson and Lesnefsky (2024):

######  Corollary 3.

In General Relativity, every smooth, geodesically complete, non-constant scale
factor $a(t)$ of a FRW spacetime must undergo accelerated expansion for at
least some period of time.

This finding is not entirely surprising given the well-known fact that the NEC
must be violated in order to achieve a cosmological bounce (in a flat $k=0$)
FRW spacetime Hawking and Penrose (1970). During a bounce, the universe
transitions from a contracting phase ($H<0$) to an expanding phase ($H>0$).
This transition inherently requires that at the point where the contraction
halts and expansion begins, the derivative of the Hubble parameter, $\dot{H}$,
must be positive. Since $\dot{H}=-4\pi G(\rho+p)$, we must have $\rho+p<0$,
signalling violation of the NEC. Thus, any such spacetime which exhibits a
bounce, or bounces, for any part of it’s history must violate NEC, and since
SEC implies the NEC, its infringement is inevitable, thereby permitting the
possibility of accelerated expansion. 777With non-zero curvature $k$, one can
produce a bounce without violating NEC since, $\dot{H}=-4\pi
G(\rho+p)+k/a^{2}$, can become positive at the bounce due to $k$. Since
$\ddot{a}/a=H^{2}+\dot{H}=-4\pi G(\rho/3+p)$, and $H=0$ at the bounce,
$\ddot{a}/a>0$ and the SEC is violated. Such curvature bounces are discussed
in Starobinskii (1978); Graham  _et al._ (2014).

Outside the context of General Relativity and for Generalized-FRW metrics, the
above corollary lacks a formally established proof, and we therefore propose:

######  Conjecture 4.

As a metrical theory of gravity, any geodesically complete GFRW must undergo
inflation.

In support of this conjecture, for a specific class of GFRW scale factors, a
proof is possible resulting in:

######  Corollary 5.

Let $\mathbb{R}^{1}_{1}\times_{p}\Sigma$ be a GFRW with scale factor
$p\left(t\right)\in\mathbb{R}\left[t\right]$, a polynomial of degree $2n$ with
$n\geq 1$ with no real roots and $p>0$, then
$\mathbb{R}^{1}_{1}\times_{p}\Sigma$ undergoes accelerated expansion.

_Conclusions_ –Thm. 1, derived from direct integration of the geodesic
equations, is definitive in assessing the geodesic completeness of GFRW
spacetimes, opposed to integrals of $H$, $H_{avg}$, or the BGV Thm. 2.

Considering the above Corollaries within the context of the incompleteness
arguments presented in Borde  _et al._ (2003), we arrive at a compelling
contrapositive: The issue is not that inflationary spacetimes are necessarily
incomplete; instead, we find that for spacetimes to be complete, they must
exhibit inflationary behavior. Within classical GR, and with the caveat with
respect to traditional energy conditions, accelerated expansion and inflation
can play a critical role in resolving the initial singularity problem and,
overlooking potentially significant quantum effects, inflation can be eternal
into the past.

###### Acknowledgements.

It is a pleasure to thank R. Brandenberger, P. Davies, G. Ellis, G.
Geshnizjani, W. Kinney, B. Kotschwar, A. Vikman and S. Watson for useful
discussions and correspondence. DAE is supported in part by the U.S.
Department of Energy, Office of High Energy Physics, under Award Number DE-
SC0019470.

## References

  * Guth (1981) A. H. Guth, Phys. Rev. D 23, 347 (1981).
  * Linde (1982) A. D. Linde, Phys. Lett. B 108, 389 (1982).
  * Albrecht and Steinhardt (1982) A. Albrecht and P. J. Steinhardt, Phys. Rev. Lett. 48, 1220 (1982).
  * Borde  _et al._ (2003) A. Borde, A. H. Guth, and A. Vilenkin, Phys. Rev. Lett. 90, 151301 (2003), arXiv:gr-qc/0110012 .
  * Easson and Lesnefsky (2024) D. Easson and J. Lesnefsky, (to appear, 2024).
  * Lesnefsky  _et al._ (2023) J. E. Lesnefsky, D. A. Easson, and P. C. W. Davies, Phys. Rev. D 107, 044024 (2023), arXiv:2207.00955 [gr-qc] .
  * Ellis and Maartens (2004) G. F. R. Ellis and R. Maartens, Class. Quant. Grav. 21, 223 (2004), arXiv:gr-qc/0211082 .
  * O’Neill (1983) B. O’Neill,  _Semi-Riemannian Geometry with Applications to Relativity_, edited by S. Eilenberg and H. Bass (Academic Press, New York, NY, 1983) p. 468.
  * Bishop and O’Neill (1969) B. L. Bishop and B. O’Neill, Trans. Amer. Math. Soc. 145, 1 (1969).
  * S.W.Hawking and G.F.R.Ellis (1973) S.W.Hawking and G.F.R.Ellis,  _The large scale structure_ (1973).
  * Casimir (1948) H. B. G. Casimir, Indag. Math. 10, 261 (1948).
  * Epstein  _et al._ (1965) H. Epstein, V. Glaser, and A. Jaffe, Nuovo Cim. 36, 1016 (1965).
  * Barcelo and Visser (2002) C. Barcelo and M. Visser, Int. J. Mod. Phys. D 11, 1553 (2002), arXiv:gr-qc/0205066 .
  * Chatterjee  _et al._ (2013) S. Chatterjee, D. A. Easson, and M. Parikh, Class. Quant. Grav. 30, 235031 (2013), arXiv:1212.6430 [gr-qc] .
  * Dubovsky  _et al._ (2006) S. Dubovsky, T. Gregoire, A. Nicolis, and R. Rattazzi, JHEP 03, 025 (2006), arXiv:hep-th/0512260 .
  * Nicolis  _et al._ (2010) A. Nicolis, R. Rattazzi, and E. Trincherini, JHEP 05, 095 (2010), [Erratum: JHEP 11, 128 (2011)], arXiv:0912.4258 [hep-th] .
  * Kobayashi  _et al._ (2010) T. Kobayashi, M. Yamaguchi, and J. Yokoyama, Phys. Rev. Lett. 105, 231302 (2010), arXiv:1008.0603 [hep-th] .
  * Easson  _et al._ (2011) D. A. Easson, I. Sawicki, and A. Vikman, JCAP 11, 021 (2011), arXiv:1109.1047 [hep-th] .
  * Easson and Manton (2019) D. A. Easson and T. Manton, Phys. Rev. D 99, 043507 (2019), arXiv:1802.03693 [hep-th] .
  * Kinney  _et al._ (2023) W. H. Kinney, S. Maity, and L. Sriramkumar, (2023), arXiv:2307.10958 [gr-qc] .
  * Geshnizjani  _et al._ (2023) G. Geshnizjani, E. Ling, and J. Quintin, JHEP 10, 182 (2023), arXiv:2305.01676 [gr-qc] .
  * Hawking and Penrose (1970) S. W. Hawking and R. Penrose, Proc. Roy. Soc. Lond. A 314, 529 (1970).
  * Starobinskii (1978) A. A. Starobinskii, Soviet Astronomy Letters 4, 82 (1978).
  * Graham  _et al._ (2014) P. W. Graham, B. Horn, S. Kachru, S. Rajendran, and G. Torroba, JHEP 02, 029 (2014), arXiv:1109.0282 [hep-th] .