To solve a task with reinforcement learning (RL), it is necessary to formally specify the goal of that task. Although most RL algorithms require that the goal is formalised as a Markovian reward function, alternatives have been developed (such as Linear Temporal Logic and Multi-Objective Reinforcement Learning). Moreover, it is well known that some of these formalisms are able to express certain tasks that other formalisms cannot express. However, there has not yet been any thorough analysis of how these formalisms relate to each other in terms of expressivity. In this work, we fill this gap in the existing literature by providing a comprehensive comparison of the expressivities of 17 objective-specification formalisms in RL. We place these formalisms in a preorder based on their expressive power, and present this preorder as a Hasse diagram. We find a variety of limitations for the different formalisms, and that no formalism is both dominantly expressive and straightforward to optimise with current techniques. For example, we prove that each of Regularised RL, Outer Nonlinear Markov Rewards, Reward Machines, Linear Temporal Logic, and Limit Average Rewards can express an objective that the others cannot. Our findings have implications for both policy optimisation and reward learning. Firstly, we identify expressivity limitations which are important to consider when specifying objectives in practice. Secondly, our results highlight the need for future research which adapts reward learning to work with a variety of formalisms, since many existing reward learning methods  implicitly assume that desired objectives can be expressed with Markovian rewards. Our work contributes towards a more cohesive understanding of the costs and benefits of different RL objective-specification formalisms.

§ INTRODUCTION

There are many ways of specifying objectives in Reinforcement Learning (RL). The most common method is to maximise the expected time-discounted sum of scalar Markovian rewards.[A reward function is Markovian if it depends only on the most recent transition.]
While this method has achieved wide-ranging success, recent work has identified practical objectives that cannot be specified using standard Markov rewards ([Skalse & Abate, 2023, Abel et al., 2022, Bowling et al., 2022]).
Numerous other formalisms have been proposed and utilised for specifying objectives in practice, including Multi-Objective RL
([Hayes et al., 2022, Roijers et al., 2013, Coello Coello et al., 2002]), Maximum Entropy RL ([Hazan et al., 2019, Mutti et al., 2023, Ziebart et al., 2008]), Linear Temporal Logic ([Littman et al., 2017, Lahijanian et al., 2011, Ding et al., 2011]), and Reward Machines ([Icarte et al., 2018, Toro Icarte et al., 2022, Camacho et al., 2019]). Additionally, there are various abstract formalisms (such as arbitrary functions from trajectories to the real numbers), which are generally intractable to optimise but which capture intuitive classes of objectives.

In this paper, we comprehensively compare the expressivities of 17 formalisms, which are listed and defined in <Ref>. We say that a formalism A can express another formalism B if in all environments, A can express all objectives that B can express. We formalise this notion in <Ref>. Our main results are summarised in Figure <ref>, a Hasse diagram displaying the relative expressivities of all the formalisms we consider. We find that in many cases, there is no simple answer to the question of which of two formalisms is preferable to use for policy optimisation in practice because each can express objectives that the other cannot. While there are formalisms at the top of our diagram that can express every formalism below them, we suspect that none of these are tractable to optimise in general. Therefore, we advise RL practitioners to familiarise themselves with a variety of formalisms and think carefully about which to select depending on the desired use case. Our analysis describes some of the strengths and limitations of each formalism, which can be used to inform this selection.

Our results are relevant not only for policy optimisation but also for reward learning. Reward learning is often used when it is difficult to hardcode a reward function for a task but easy to evaluate attempts at that task. For instance, ([Christiano et al., 2017]) train a reward model which incentivises an RL agent to do backflips in a simulated environment. Notably, the structure of reward models in many prior works have implicitly assumed that the desired task is expressible with Markov rewards, since reward models output real numbers given a transition as input (<cit.>. Our findings highlight the limitations of Markov rewards and the importance of advancing reward learning methods for the other formalisms we discuss.

In Section <ref>, we briefly introduce core concepts from RL relevant to our paper, make the nature of our formalism comparisons precise, and define the objective-specification formalisms that we are comparing in this work. We provide our primary results in Section <ref>. These are summarised in Figure <ref>, a Hasse diagram that depicts all the expressivity relationships among the formalisms. We also present many of the theorems and propositions on which the diagram is based; the remaining statements and all proofs are found in <Ref>. In Section <ref>, we discuss the implications of our findings in the context of prior research, along with limitations and directions for future work.

§ PRELIMINARIES

§ RESULTS

This Hasse diagram displays all expressivity relationships between our formalisms. An arrow or chain of arrows from one formalism to another indicates that the first formalism can express all policy orderings that the second formalism can express, in all environments. Arrows going both directions mean that the formalisms have the same expressivity. If there is no chain of arrows from one formalism to another, there are policy orderings that the latter can express and the former cannot express.

In this section, we present our expressivity results. For each of the 289 ordered pairs of formalisms $(X,Y)$, we prove either that $X \exprsucceq Y$ or that $X \not \exprsucceq Y$.[We study 17 formalisms, so there are $17^2 = 289$ ordered pairs of formalisms. This includes the ordered pairs of a formalism with itself; trivially, any formalism can express itself. The expressivity results for some other ordered pairs of formalisms are also fairly straightforward, and many results follow from the transitivity of the expressivity relation. <Ref> in the appendix depicts all proof dependencies for these results.] <Ref> depicts the expressivity relationships between all formalisms. Note that the absence of a sequence of arrows from one formalism to another is as significant as the presence of a sequence of arrows: the presence of a sequence of arrows means that the first formalism can express all policy orderings that the second can express, and the absence of a sequence of arrows means that there exists a policy ordering that the second formalism can express and the first cannot express. A considerable part of our contribution in this work is to demonstrate the expressive limitations of these formalisms, and these limitations are represented by the absence of arrows. In this section we will state and provide intuition for many of the nontrivial positive and negative expressivity results that serve as the basis for the Hasse diagram. The remaining results, and all proofs, can be found in <Ref>.

While we do not include our trivial results in the main body of this paper, we encourage readers who are not already familiar with the formalisms in <Ref> to start by reading those trivial results in <Ref>. We believe those statements and proofs are easier to understand without prior knowledge of the formalisms and provide an excellent foundation for understanding the rest of the expressivity relationships. As such, we view these trivial results for our comprehensive list of formalisms as an important contribution of our work.

§.§ Positive Results

Several of the positive connections in the Hasse diagram are relatively trivial; these are discussed in <Ref>. Here, we focus on the most substantive results. Proofs for the results in this subsection are available in <Ref>.

Inner Nonlinear Markov Rewards (INMR), Inner Multi-Objective RL (IMORL), and Functions from Trajectories to Reals (FTR) are equally expressive.   Proof: <ref>

It is straightforward to see that IMORL can express all policy orderings that INMR can express and that FTR can express all policy orderings that IMORL can express (and we show these results explicitly in <Ref>). Therefore, the following proposition is sufficient to conclude that INMR, IMORL, and FTR are equally expressive:

Any policy ordering expressible with Functions from Trajectories to Reals (FTR) can be expressed with Inner Nonlinear Markov Rewards (INMR). Proof: <ref>

Briefly, our proof demonstrates that it is possible to express an arbitrary function from trajectories to reals $f_{FTR}$ with an INMR specification $(R,f_{INMR},\gamma)$ by selecting $R$ and $\gamma$ so that the trajectory return function $G$ is injective, and then setting $f_{INMR}$ equal $f_{FTR} \circ G^{-1}$. The full proof is in <Ref>. This proof notably relies on allowing $f_{INMR}$ to be an arbitrary function that need not satisfy properties such as differentiability, continuity, or monotonicity. Since many possible nonequivalent constraints might be of interest, we choose to define a very general version of INMR; we discuss the implications of this choice in the next section, and in more detail in <Ref>.

Outer Multi-Objective RL (OMORL), Functions from Occupancy Measures to Reals (FOMR), and Functions from Trajectory Lotteries to Reals (FTLR) are equally expressive. Proof: <ref>

We show that a) two policies have the same occupancy measure if and only if they generate the same trajectory lottery, and b) it is always possible to specify an OMORL objective such that two policies have the same policy evaluation vector ($\vec J(\pi) := \langle J_1(\pi),...,J_k(\pi)\rangle$) if and only if they have the same occupancy measure. Therefore, the expressivity of a function from policy evaluation vectors to reals is exactly the same as a function from occupancy measures to reals and a function from trajectory lotteries to reals. It is worth noting that in order to possess this expressive power, OMORL requires access to $|\SSpace||\ASpace||\SSpace|$ reward functions for any environment $E = (\SSpace,\ASpace,\Trans,\Init)$. It may be infeasible to specify and optimise an objective with so many reward functions in practice; in <Ref> and <Ref>, we discuss the possibility of restricting the number of reward functions a multi-reward objective can include.

Our third theorem is very similar to <Ref>:

Generalised Outer Multi-Objective RL (GOMORL), Occupancy Measures Orderings (OMO), and Trajectory Lottery Orderings (TLO) are equally expressive.  Proof: <ref>

The proof of this theorem utilises the same principles and lemmas as the proof of <Ref>. Total preorders over policy evaluation vectors, occupancy measure, and trajectory lotteries are all equally expressive, because two policies share a trajectory lottery if and only if they share an occupancy measure if and only if they share a policy evaluation vector (with appropriately selected reward functions).

§.§ Negative Results

All proofs for the results in this subsection can be found in <Ref>.

There exists an environment in which Linear Temporal Logic (LTL) and Limit Average Rewards (LAR) can express a policy ordering that Reward Machines (RM) and Markov Rewards (MR) cannot express. Proof: <ref>

RM specifications can only induce policy evaluation functions that are continuous (in the sense that infinitesimal changes to a policy can only lead to infinitesimal changes to the policy evaluation). Note that it must also be true that Markov Rewards (MR) can only induce continuous policy evaluation functions, because as shown in <Ref>, RM can express any policy evaluation function that MR can express. LTL and LAR can induce discontinuous policy evaluation functions.

There exists an environment in which Markov Rewards (MR) and Limit Average Rewards (LAR) can express a policy ordering that Linear Temporal Logic (LTL) cannot express. Proof: <ref>

An LTL specification can only assign a value of $\varphi(\xi)= 0$ or $\varphi(\xi)= 1$ to a trajectory, since an LTL formula $\varphi$ is either true or false for any given trajectory $\xi$. MR and LAR, on the other hand, can give any scalar-valued rewards. In certain cases, this prevents LTL from differentiating between trajectories — and lotteries over trajectories generated by policies — that MR and LAR can distinguish. For instance, LTL cannot induce any strict ordering of 3 deterministic policies in a deterministic environment.

There exists an environment in which Linear Temporal Logic (LTL) and Markov Rewards (MR) can express a policy ordering that Limit Average Rewards (LAR) cannot express. Proof: <ref>

LAR cannot distinguish between policies that generate the same trajectory lottery after a finite amount of time, even if the lotteries they generate over initial trajectory segments are different. LTL and MR can distinguish between policies based on differences that appear only early on in trajectories.

There exists an environment in which Outer Nonlinear Markov Rewards (ONMR) can express a policy ordering that Functions from Trajectories to Reals (FTR) cannot express. Proof: <ref>

ONMR can express the objective of making a trajectory occur with at least some desired threshold probability, while being indifferent to achieving probabilities higher than the threshold. For example, this can be achieved (for a particular environment and reward specification) with the wrapper function $f_{ONMR}(x) =
\begin{cases}
1 & \text{if } x \geq 0.5, \\
0 & \text{otherwise}
\end{cases}$. By contrast, for an FTR specification $(f_{FTR})$ to incentivise a trajectory, $f_{FTR}$ must assign a higher value to that trajectory than others. However, this would incentivise maximizing the probability of that trajectory, not merely meeting a threshold.

There exists an environment in which Functions from Policies to Reals (FPR) can express a policy ordering that Generalised Outer Multi-Objective RL (GOMORL) cannot express. Proof: <ref>

A function from policies to reals can assign different values to any two distinct policies, even if they only differ on states that both policies have probability zero of ever visiting. GOMORL cannot express preferences between policies that are identical on all states visited with nonzero probability.

There exists an environment in which Generalised Outer Multi-Objective RL (GOMORL) can express a policy ordering that Functions from Policies to Reals (FPR) cannot express. Proof: <ref>

There exist lexicographic preference orderings of GOMORL policy evaluation vectors that cannot be represented by any real-valued function ([Steen & Seebach, 2012]).  These orderings of policy evaluation vectors can be used to express tasks in GOMORL that are inexpressible in FPR.

§ DISCUSSION AND RELATED WORK

Previous work has attempted to settle the reward hypothesis, which states that "all of what we mean by goals and purposes can be well thought of as maximization of the expected value of the cumulative sum of a received scalar signal (called reward)" ([Sutton, 2004]). Related to this is the von Neumann-Morgenstern (VNM) utility theorem, which states that an agent's preferences between lotteries over a finite set of outcomes can be represented as ordered according to the expected value of a scalar utility function of the outcomes if and only if these preferences satisfy four axioms of rationality (the VNM axioms) ([Von Neumann & Morgenstern, 1944]). Consequently, if one's preferences cannot be represented by a utility function, the preferences must violate at least one of these axioms.

The VNM theorem is typically considered in the context of single-decision problems, while RL is applied to sequential decision-making. Without modification, the VNM theorem can be applied to RL settings by taking the outcome space to be a finite set of (full) trajectories. It then states that an agent's preferences about lotteries over a finite number of trajectories can be expressed as a function from trajectories to the reals if and only if those preferences satisfy the VNM axioms. Recent work has proven stronger statements about the form of this trajectory return function by supplementing the VNM axioms with additional axioms for preferences over temporally extended outcomes.

[Pitis, 2019], [Shakerinava & Ravanbakhsh, 2022], and [Bowling et al., 2022] all identify axioms from which it is possible to prove that preferences regarding temporally extended outcomes can be expressed using Markovian rewards. All three papers identify slightly different axioms that, when supplemented with the VNM axioms, enable this proof. One caveat is that all three papers show that preferences can be represented with a Markovian reward function along with a transition-dependent discount factor, rather than the traditional constant discount factor in MDPs.

The VNM theorem and the work that has adapted it to sequential decision-making settings present axioms that must be violated for an objective to be inexpressible with Markov rewards or a function from trajectories to reals. Our work demonstrates several objectives that Markov rewards and functions from trajectories to reals cannot express, which raises the question: are there reasonable objectives that violate the axioms, or are the objectives we present unreasonable? We elaborate on this tension in <Ref>, particularly for violations of the VNM axioms. In our work we also assume a conventional constant discount factor, but future work could compare the expressive power of formalisms that allow transition-dependent discount factors with the formalisms we consider.

[Abel et al., 2022] formalise a task in three different ways. One is a set of acceptable policies, another is an ordering over policies and the third is an ordering over trajectories. They find that for each notion of a task there exists an instance that no Markov reward function can capture. The second formalisation, policy orderings, is the one that we use, and we adapt their proof that Markov Rewards cannot express an "XOR" policy ordering to display the expressive limitations of several formalisms.

Multi-Objective RL (MORL) has received considerable attention in the literature. [Silver et al., 2021] argues that maximization of a scalar reward is enough to drive most behaviour in natural and artificial intelligence, while [Vamplew et al., 2022] argue that multi-objective rewards are necessary. In particular, [Vamplew et al., 2022] argue that scalar reward is insufficient to develop and align Artificial General Intelligence. [Miura, 2023] shows that multi-dimensional reward functions are also not universally expressive; they state necessary and sufficient conditions under which a task defined as a set of acceptable policies can be expressed with a multi-dimensional reward function. They use a definition of MORL where a policy is acceptable if each dimension of reward is above a given lower bound — we generalise this notion for one of our formulations of MORL . Algorithms for solving multi-objective problems have been developed and used in several applications ([Hayes et al., 2022, Jalalimanesh et al., 2017, Castelletti et al., 2013]).

[Skalse & Abate, 2023] demonstrate that some multi-objective, risk-sensitive, and modal tasks cannot be expressed with Markovian rewards. We analyse several formalisms related to this paper: Generalised Outer MORL (GOMORL) is based on this paper's definition of MORL, and our Inner Nonlinear Markov Rewards (INMR) is partially motivated as a generalisation for risk-sensitive tasks. We extend these definitions to study what we call Outer MORL (OMORL), Inner MORL (IMORL), and Outer Nonlinear Markov Rewards (ONMR), which enables a robust comparison of the expressive power of wrapper functions that are applied before and after taking an expected value over trajectories. [Skalse & Abate, 2023] also use occupancy measures in a proof to demonstrate limitations of Markov Rewards, and we find occupancy measures a sufficiently useful tool for reasoning about formalism expressivity that we include Functions from Occupancy Measures to Reals and Occupancy Measure Orderings as formalisms in our analysis.

Prior work has studied Reward Machines (RM), Linear Temporal Logic (LTL), and Limit Average Rewards (LAR) as well. [Icarte et al., 2018] point out that reward machines can trivially express all objectives that Markov Rewards can express and provide rewards that depend on a finite number of features of the history so far, rather than just the most recent transition. This paper also applies RM specifications to toy tasks. ([Littman et al., 2017]) study LTL, which was previously utilised in control theory, in an RL setting ([Bacchus et al., 1970, Puterman, 1994]). The authors mention that many common tasks can be expressed with LTL, and are sometimes easier to express with LTL than with Markov Rewards. They also develop practical methods for solving objectives specified in LTL. However, they do not prove any general statements about the expressivity or limitations of LTL. [Mahadevan, 1996] investigates algorithms for learning goals specified with Limit Average Rewards (LAR), but we have not found any work on the expressivity of LAR.

When attempting to reverse engineer an agent's utility function from its behavior, Maximum Entropy Inverse RL makes an assumption that an agent's distribution over actions has maximum entropy ([Ziebart et al., 2008]). Our formalism of Regularised RL relates to this concept by allowing any function of the distributions over actions to be included in the RL objective.

§.§ Limitations and Future Work

In this paper, we focus on stationary policies, and define objectives as orderings of stationary policies. In principle, however, policies can be nonstationary; that is, an agent can select actions based on the history rather than just the most recent state. We make the choice to focus on stationary policies because common RL algorithms (such as Q-learning ([Watkins & Dayan, 1992])) derive exclusively stationary policies from state-action value functions. Not all of our results carry over to the nonstationary setting. For instance, it is fairly straightforward to show that nonstationary policies can have the same occupancy measure but generate different trajectory lotteries. This indicates that future work extending our study to nonstationary policies may find substantially different results.

A few of our results rely heavily on the technical details of the formalisms in ways that may reduce their practical significance. For instance, <Ref> relies on the ability of Inner Nonlinear Markov Rewards to utilise an arbitrary wrapper function for the trajectory return, and Theorems <ref> and <ref> require allowing multi-objective formalisms access to an arbitrary number of reward functions. One direction for future work is to replicate our analysis under various restricting conditions; we discuss some conditions that may be of interest in <Ref>.

Another limitation of our results is that we largely ignore the important tradeoff between expressivity and tractability. In general, we would expect that more expressive formalisms are more difficult to implement in practice. An important area for future work might be to identify which of the more expressive formalisms considered here could be implemented in practice, and provide guidance for balancing expressivity and tractability in various practical settings. This might also involve considering restricting conditions for the formalisms, since the conditions required to make a formalism tractable to optimise could reduce its expressivity.

Until the tractability of these formalisms is studied more carefully, it is difficult to draw a clear conclusion about which formalisms are best to use in practice. A more immediate takeaway from our results is that many of the formalisms are expressively incommensurable; for example, Limit Average Reward, Linear Temporal Logic, Reward Machines, Regularised RL, and Outer Nonlinear Markov Rewards can each express policy orderings that none of the others can express. This highlights the importance of carefully selecting a formalism for any task of interest; our results elucidate strengths and limitations of different formalisms that can inform these selections.

Future research can also explore reward learning with formalisms other than Markov Rewards. There have been preliminary efforts in the direction of reward learning for Reward Machines ([Icarte et al., 2019]), Linear Temporal Logic ([Neider & Gavran, 2018]), and Limit Average Reward ([Wu et al., 2023]), but given the range of objectives that we show cannot be specified with Markov Rewards, it may be important to further develop reward learning alternatives that allow us to reliably express the objectives we desire across a variety of domains.

§ CONCLUSION

This paper provides a complete map of the relative expressivities of seventeen formalisms for specifying objectives in RL. We discovered meaningful expressive limitations to all of these formalisms,[With the exception of Policy Orderings, which by our definition can express all objectives.] including many of the alternatives and extensions to Markov rewards that have been discussed in the existing literature. We also related practical formalisms to a number of theoretical constructs, such as trajectory lotteries and occupancy measures, which help to fill out a richer intuitive picture of where each formalism stands in the expressivity hierarchy. We hope our work will serve as a reference point for future discussions of these methods for specifying temporally extended objectives, as well as provide an impetus for future work that contextualises these results in light of the tradeoff between expressivity and tractability that appears in both policy optimisation and reward learning.

§ REPRODUCIBILITY STATEMENT

All of our results are theoretical. All proofs can be found in Appendix <ref>. For our definitions of environments, objectives, total order and formalisms see Section <ref>. Any further assumptions have been stated in the proofs.

[Abel et al., 2022]
David Abel, Will Dabney, Anna Harutyunyan, Mark K. Ho, Michael L. Littman,
Doina Precup, and Satinder Singh.
On the Expressivity of Markov Reward, January 2022.
URL <http://arxiv.org/abs/2111.00876>.
arXiv:2111.00876 [cs].

[Bacchus et al., 1970]
Fahiem Bacchus, Craig Boutilier, and Adam Grove.
Rewarding Behaviors.
Proceedings of the National Conference on Artificial
Intelligence, 2, February 1970.

[Baier & Katoen, 2008]
Christel Baier and Joost-Pieter Katoen.
Principles of model checking.
MIT Press, Cambridge, Mass., 2008.
ISBN 978-0-262-02649-9.

[Bowling et al., 2022]
Michael Bowling, John D. Martin, David Abel, and Will Dabney.
Settling the Reward Hypothesis, December 2022.
URL <http://arxiv.org/abs/2212.10420>.
arXiv:2212.10420 [cs, math, stat].

[Camacho et al., 2019]
Alberto Camacho, Rodrigo Toro Icarte, Toryn Q. Klassen, Richard Valenzano, and
Sheila A. McIlraith.
LTL and Beyond: Formal Languages for Reward Function
Specification in Reinforcement Learning.
In Proceedings of the Twenty-Eighth International Joint
Conference on Artificial Intelligence, pp.  6065–6073, Macao, China,
August 2019. International Joint Conferences on Artificial Intelligence
ISBN 978-0-9992411-4-1.
URL <https://www.ijcai.org/proceedings/2019/840>.

[Castelletti et al., 2013]
Andrea Castelletti, F. Pianosi, and Marcello Restelli.
A multiobjective reinforcement learning approach to water resources
systems operation: Pareto frontier approximation in a single run.
Water Resources Research, 49:0 3476–3486, June 2013.

[Christiano et al., 2017]
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario
Deep reinforcement learning from human preferences, June 2017.
URL <http://arxiv.org/abs/1706.03741>.
arXiv:1706.03741 [cs, stat] version: 1.

[Coello Coello et al., 2002]
Carlos A. Coello Coello, David A. Van Veldhuizen, and Gary B. Lamont.
Evolutionary Algorithms for Solving Multi-Objective
volume 5 of Genetic Algorithms and Evolutionary
Computation, Boston, MA, 2002. Springer US.
ISBN 978-1-4757-5186-4 978-1-4757-5184-0.
URL <http://link.springer.com/10.1007/978-1-4757-5184-0>.

[Ding et al., 2011]
Xu Chu Ding, Stephen L. Smith, Calin Belta, and Daniela Rus.
LTL Control in Uncertain Environments with Probabilistic
Satisfaction Guarantees, April 2011.
URL <http://arxiv.org/abs/1104.1159>.
arXiv:1104.1159 [cs, math].

[Hadfield-Menell et al., 2016]
Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell.
Cooperative inverse reinforcement learning, 2016.
arXiv: 1606.03137 [cs.AI].

[Hayes et al., 2022]
Conor F. Hayes, Roxana Rădulescu, Eugenio Bargiacchi, Johan Källström,
Matthew Macfarlane, Mathieu Reymond, Timothy Verstraeten, Luisa M. Zintgraf,
Richard Dazeley, Fredrik Heintz, Enda Howley, Athirai A. Irissappane, Patrick
Mannion, Ann Nowé, Gabriel Ramos, Marcello Restelli, Peter Vamplew, and
Diederik M. Roijers.
A Practical Guide to Multi-Objective Reinforcement
Learning and Planning.
Autonomous Agents and Multi-Agent Systems, 360
(1):0 26, April 2022.
ISSN 1387-2532, 1573-7454.
URL <http://arxiv.org/abs/2103.09568>.
arXiv:2103.09568 [cs].

[Hazan et al., 2019]
Elad Hazan, Sham M. Kakade, Karan Singh, and Abby Van Soest.
Provably Efficient Maximum Entropy Exploration, January 2019.
URL <http://arxiv.org/abs/1812.02690>.
arXiv:1812.02690 [cs, stat].

[Icarte et al., 2018]
Rodrigo Toro Icarte, Toryn Klassen, Richard Valenzano, and Sheila McIlraith.
Using Reward Machines for High-Level Task Specification
and Decomposition in Reinforcement Learning.
In Proceedings of the 35th International Conference on
Machine Learning, pp.  2107–2116. PMLR, July 2018.
URL <https://proceedings.mlr.press/v80/icarte18a.html>.
ISSN: 2640-3498.

[Icarte et al., 2019]
Rodrigo Toro Icarte, Ethan Waldie, Toryn Q. Klassen, R. Valenzano, Margarita P.
Castro, and Sheila A. McIlraith.
Learning Reward Machines for Partially Observable
Reinforcement Learning.
September 2019.

[Jalalimanesh et al., 2017]
Ammar Jalalimanesh, Hamidreza Shahabi Haghighi, Abbas Ahmadi, Hossein Hejazian,
and Madjid Soltani.
Multi-objective optimization of radiotherapy: distributed
Q-learning and agent-based simulation.
Journal of Experimental & Theoretical Artificial
Intelligence, 290 (5):0 1071–1086, September 2017.
ISSN 0952-813X.
URL <https://doi.org/10.1080/0952813X.2017.1292319>.
Publisher: Taylor & Francis _eprint:

[Lahijanian et al., 2011]
M. Lahijanian, S. B. Andersson, and C. Belta.
Control of Markov decision processes from PCTL specifications.
In Proceedings of the 2011 American Control Conference,
pp.  311–316, San Francisco, CA, June 2011. IEEE.
ISBN 978-1-4577-0081-1 978-1-4577-0080-4 978-1-4577-0079-8.
URL <http://ieeexplore.ieee.org/document/5990952/>.

[Littman et al., 2017]
Michael L. Littman, Ufuk Topcu, Jie Fu, Charles Isbell, Min Wen, and James
Environment-Independent Task Specifications via GLTL, April
URL <http://arxiv.org/abs/1704.04341>.
arXiv:1704.04341 [cs].

[Mahadevan, 1996]
Sridhar Mahadevan.
Average reward reinforcement learning: Foundations, algorithms, and
empirical results.
Machine Learning, 220 (1):0 159–195, March
ISSN 1573-0565.
URL <https://doi.org/10.1007/BF00114727>.

[Manna & Pnueli, 1992]
Zohar Manna and Amir Pnueli.
The Temporal Logic of Reactive and Concurrent
Springer, New York, NY, 1992.
ISBN 978-1-4612-6950-2 978-1-4612-0931-7.
URL <http://link.springer.com/10.1007/978-1-4612-0931-7>.

[Miura, 2023]
Shuwa Miura.
On the Expressivity of Multidimensional Markov Reward, July
URL <http://arxiv.org/abs/2307.12184>.
arXiv:2307.12184 [cs].

[Mutti et al., 2023]
Mirco Mutti, Riccardo De Santi, Piersilvio De Bartolomeis, and Marcello
Challenging Common Assumptions in Convex Reinforcement
Learning, January 2023.
URL <http://arxiv.org/abs/2202.01511>.
arXiv:2202.01511 [cs].

[Neider & Gavran, 2018]
Daniel Neider and Ivan Gavran.
Learning Linear Temporal Properties, September 2018.
URL <http://arxiv.org/abs/1806.03953>.
arXiv:1806.03953 [cs].

[Pitis, 2019]
Silviu Pitis.
Rethinking the Discount Factor in Reinforcement Learning: A
Decision Theoretic Approach, February 2019.
URL <http://arxiv.org/abs/1902.02893>.
arXiv:1902.02893 [cs].

[Puterman, 1994]
Martin L. Puterman.
Markov decision processes: Discrete stochastic dynamic
John Wiley and Sons, Inc., USA, 1 edition, 1994.
ISBN 0-471-61977-9.

[Roijers et al., 2013]
D. M. Roijers, P. Vamplew, S. Whiteson, and R. Dazeley.
A Survey of Multi-Objective Sequential Decision-Making.
Journal of Artificial Intelligence Research, 48:0
67–113, October 2013.
ISSN 1076-9757.
URL <https://www.jair.org/index.php/jair/article/view/10836>.

[Shakerinava & Ravanbakhsh, 2022]
Mehran Shakerinava and Siamak Ravanbakhsh.
Utility Theory for Sequential Decision Making.
In Proceedings of the 39th International Conference on
Machine Learning, pp.  19616–19625. PMLR, June 2022.
URL <https://proceedings.mlr.press/v162/shakerinava22a.html>.
ISSN: 2640-3498.

[Silver et al., 2021]
David Silver, Satinder Singh, Doina Precup, and Richard S. Sutton.
Reward is enough.
Artificial Intelligence, 299:0 103535, October 2021.
ISSN 00043702.

[Skalse et al., 2023]
Joar Skalse, Matthew Farrugia-Roberts, Stuart Russell, Alessandro Abate, and
Adam Gleave.
Invariance in Policy Optimisation and Partial Identifiability
in Reward Learning, June 2023.
URL <http://arxiv.org/abs/2203.07475>.
arXiv:2203.07475 [cs, stat].

[Skalse & Abate, 2023]
Joar Max Viktor Skalse and Alessandro Abate.
On the Limitations of Markovian Rewards to Express
Multi-Objective, Risk-Sensitive, and Modal Tasks.
June 2023.
URL <https://openreview.net/forum?id=Vcvg76ZmcZt>.

[Steen & Seebach, 2012]
L.A. Steen and J.A.J. Seebach.
Counterexamples in Topology.
Springer New York, 2012.
ISBN 978-1-4612-6290-9.
URL <https://books.google.co.uk/books?id=UNbTBwAAQBAJ>.

[Sutton, 2004]
Richard S. Sutton.
The reward hypothesis, 2004.

[Sutton & Barto, 2018]
Richard S Sutton and Andrew G Barto.
Reinforcement learning: An introduction.
MIT Press, 2 edition, 2018.
ISBN 978-0-262-35270-3.

[Toro Icarte et al., 2022]
Rodrigo Toro Icarte, Toryn Q. Klassen, Richard Valenzano, and Sheila A.
Reward Machines: Exploiting Reward Function Structure in
Reinforcement Learning.
Journal of Artificial Intelligence Research, 73:0
173–208, January 2022.
ISSN 1076-9757.
URL <https://jair.org/index.php/jair/article/view/12440>.

[Vamplew et al., 2022]
Peter Vamplew, Benjamin J. Smith, Johan Källström, Gabriel Ramos, Roxana
Rădulescu, Diederik M. Roijers, Conor F. Hayes, Fredrik Heintz, Patrick
Mannion, Pieter J. K. Libin, Richard Dazeley, and Cameron Foale.
Scalar reward is not enough: a response to Silver, Singh,
Precup and Sutton (2021).
Autonomous Agents and Multi-Agent Systems, 360
(2):0 41, July 2022.
ISSN 1573-7454.
URL <https://doi.org/10.1007/s10458-022-09575-5>.

[Von Neumann & Morgenstern, 1944]
J. Von Neumann and O. Morgenstern.
Theory of games and economic behavior.
Theory of games and economic behavior. Princeton University Press,
Princeton, NJ, US, 1944.
Pages: xviii, 625.

[Watkins & Dayan, 1992]
Christopher J. C. H. Watkins and Peter Dayan.
Machine Learning, 80 (3):0 279–292, May 1992.
ISSN 1573-0565.
URL <https://doi.org/10.1007/BF00992698>.

[Wu et al., 2023]
Feiyang Wu, Jingyang Ke, and Anqi Wu.
Inverse Reinforcement Learning with the Average Reward
Criterion, May 2023.
URL <http://arxiv.org/abs/2305.14608>.
arXiv:2305.14608 [cs].

[Ziebart et al., 2008]
Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey.
Maximum entropy inverse reinforcement learning.
In Proceedings of the 23rd national conference on Artificial
intelligence - Volume 3, AAAI'08, pp.  1433–1438, Chicago, Illinois,
July 2008. AAAI Press.
ISBN 978-1-57735-368-3.


# Constraining the Densities of the Three Kepler-289 Planets with Transit
Timing Variations

Michael Greklek-McKeon Heather A. Knutson Shreyas Vissapragada Division of
Geological and Planetary Sciences, California Institute of Technology,
Pasadena, CA, 91125, USA  Daniel Jontof-Hutter Department of Physics,
University of the Pacific, 3601 Pacific Avenue, Stockton, CA 95211, USA
Yayaati Chachan Division of Geological and Planetary Sciences, California
Institute of Technology, Pasadena, CA, 91125, USA  Daniel Thorngren Department
of Physics, University of Montréal, 1375 Avenue Thérèse-Lavoie-Roux, Montréal,
QC, H2V 0B3, Canada  Gautam Vasisht Jet Propulsion Laboratory, California
Institute of Technology, 4800 Oak Grove Dr, Pasadena, CA 91109, USA

###### Abstract

Kepler-289 is a three-planet system containing two sub-Neptunes and one cool
giant planet orbiting a young, Sun-like star. All three planets exhibit
transit timing variations (TTVs), with both adjacent planet pairs having
orbital periods close to the 2:1 orbital resonance. We observe two transits of
Kepler-289c with the Wide-field InfraRed Camera (WIRC) on the 200” Hale
Telescope at Palomar Observatory, using diffuser-assisted photometry to
achieve space-like photometric precision from the ground. These new transit
observations extend the original four-year Kepler TTV baseline by an
additional 7.5 years. We re-reduce the archival Kepler data with an improved
stellar activity correction and carry out a joint fit with the Palomar data to
constrain the transit shapes and derive updated transit times. We then model
the TTVs to determine the masses of the three planets and constrain their
densities and bulk compositions. Our new analysis improves on previous mass
and density constraints by a factor of two or more for all three planets, with
the innermost planet showing the largest improvement. Our updated atmospheric
mass fractions for the inner two planets indicate that they likely have
hydrogen-rich envelopes, consistent with their location on the upper side of
the radius valley. We also constrain the heavy element composition of the
outer saturn-mass planet, Kepler-289c, for the first time, finding that it
contains 30.5 $\pm$ 6.9 $M_{\oplus}$ of metals. We use dust evolution models
to show that Kepler-289c must have formed beyond 1 au, and likely beyond 3 au,
and then migrated inward.

††facilities: Kepler, Palomar: Hale 200“ Telescope††software: astropy (Astropy
Collaboration et al., 2018b), scipy (sci, 2001), numpy (sci, 2001), matplotlib
(Hunter, 2007) rebound (Rein & Tamayo, 2015), BATMAN (Kreidberg, 2015), emcee
(Foreman-Mackey et al., 2013), corner (Foreman-Mackey, 2016), exoplanet
(Foreman-Mackey et al., 2021)

##  1 Introduction

The Kepler space telescope (Borucki et al., 2010) discovered thousands of new
transiting exoplanet systems, including hundreds of multi-planet systems.
Kepler was decommissioned in 2018, but the science value of the data it
provided has yet to be exhausted. Part of this legacy science includes follow-
up observations of the most interesting dynamically interacting multi-planet
systems. _Kepler_ transit photometry provides us with information about the
planetary radius, orbital period, inclination, and semi-major axis of
transiting planets. For the subset of dynamically interacting systems,
observations of transit timing variations (TTVs; Agol et al., 2005; Holman &
Murray, 2005) also provide complementary constraints on planetary masses,
eccentricities, and bulk densities. We can use this information to
characterize the bulk compositions and atmospheric mass fractions of these
planets, as well as their dynamical architectures, which in turn provide key
constraints on planet formation and evolution models.

For an isolated transiting planet on a stable circular orbit, transits occur
at uniform intervals. In multi-planet systems, however, gravitational
interactions between planets can cause the transit mid-times to deviate from a
linear ephemeris by amounts ranging from minutes to hours. The amplitude of
these TTVs depends on the masses of the planets and their orbital elements
(Agol & Fabrycky, 2018). Planets near first-order mean-motion resonances
(MMRs) typically have the largest TTV amplitudes (e.g. Deck & Agol, 2016) and
are therefore the most favorable for dynamical mass measurements.

Kepler discovered hundreds of multi-planet systems exhibiting detectable TTVs
(Holczer et al., 2016; Hadden & Lithwick, 2017). For many of these systems,
the data from the four-year Kepler prime mission was sufficient to obtain a
large sample of transits for each planet spanning the TTV “super-period", the
timescale on which the TTVs oscillate (Lithwick et al., 2012). However, the
_Kepler_ data are unable to provide strong constraints on the dynamical states
of some planets with orbital periods longer than $\sim$100 days, where
relatively few transits were observed and/or the baseline is shorter than the
predicted super-period. In cases where the _Kepler_ baseline is longer than
the TTV super-period and we can obtain good dynamical mass constraints, TTVs
provide us with a unique opportunity to measure the bulk densities of long-
period planets, without requiring an extensive multi-year radial velocity (RV)
follow-up campaign (e.g. Chachan et al., 2022). There are currently only 18
transiting planets with a fractional mass uncertainty less than 1/3 (e.g.,
$>3\sigma$ mass measurement) that have orbital periods greater than 100 days
listed in the Exoplanet Archive (Akeson et al., 2013). Of these 18, 11 were
characterized using TTVs.

In this study we focus on Kepler-289, a young ($\sim$1 Gyr) Sun-like
($T_{\mathrm{eff}}=5990\pm 38$ K) star that hosts three planets with orbital
periods of 34.5, 66.1, and 125.9 days, corresponding to period ratios of 1.9:1
for each adjacent planet pair (Schmitt et al., 2014). These period ratios are
reminiscent of those of the Galilean moons of Jupiter, which are in a Laplace
resonance. We calculate a normalized distance to resonance $\Delta$ of 0.04
for both planet pairs (Lithwick et al., 2012), indicating that they are likely
to exhibit TTVs. This is consistent with the population of near-resonant
Kepler systems identified in previous studies, which typically have
$\Delta<0.05$ (Holczer et al., 2016; Jontof-Hutter et al., 2016, 2021). Unlike
most of these sytems, the Kepler-289 planets have period ratios interior to
resonance, rather than the more common exterior configuration (Fabrycky et
al., 2014).

Although sub-Neptune planets frequently have outer Jovian companions (Bryan et
al., 2019; Zhu & Wu, 2018), these companions are typically located at much
larger orbital separations ($\gtrsim 1$ au). The relatively close spacing
between the outer gas giant and the inner sub-Neptunes in the Kepler-289
system, along with their near-resonant orbital configuration, suggests that
this system may have started with a much wider orbital spacing before
undergoing migration (Charalambous et al., 2022). The planets are not
currently in resonance, however, and by obtaining improved mass measurements
and bulk composition constraints for all three planets we can further examine
the unique history of this unusual system.

All three planets in the Kepler-289 system exhibit detectable TTVs in the
_Kepler_ data, with amplitudes ranging from $\sim 0.5-5$ hours (Schmitt et
al., 2014). Previous TTV studies of this system were complicated by the fact
that the light curve of Kepler-289 is significantly variable ($\sim$3%
amplitude in the _Kepler_ bandpass) due to rotational modulation from
starspots. This makes it difficult to obtain reliable transit mid-time
measurements for the inner and middle planets, which both have relatively
shallow transit depths of $\sim$400 ppm (Schmitt et al., 2014). In addition,
the outermost planet, Kepler-289c, only transited ten times during the
_Kepler_ observations, which cover only slightly more than one full TTV super-
period (Jontof-Hutter et al., 2021). This makes this planet an ideal target
for ground-based follow-up observations, which allow us to extend the TTV
baseline by many years and to obtain improved dynamical mass constraints for
all three planets.

In this study we re-analyze existing _Kepler_ photometry of this system and
combine it with two new observations of Kepler-289c obtained using the Wide
Field InfraRed Camera (WIRC) on the 200" Hale Telescope at Palomar
Observatory. We achieve space-quality infrared photometry of this system by
using a beam-shaping diffuser whose properties are described in more detail in
Stefansson et al. (2017) and Vissapragada et al. (2020). By observing in the
infrared, we mitigate the effects of stellar activity on the transit light
curve. In Section 2, we describe each observational data set. In Section 3, we
describe our stellar activity correction for the _Kepler_ photometry, our
reduction of the Palomar transit light curves, and the TTV analysis. In
Section 4 we discuss the results of our analysis, and in Section 5 we conclude
and discuss the implications for these new set of constraints on the
Kepler-289 system.

Figure 1: Representative photometry of Kepler-289 from Kepler quarter 5,
exhibiting significant rotational modulation. A deep transit of planet c, the
outermost planet, is easily visible around 614 (BJD-2454000). The right panel
shows a zoomed-in view of this transit with the masked region around the
transit indicated in blue and the baseline trend we use to remove the stellar
variability overplotted in red.

##  2 Observations

###  2.1 Space-Based Photometry

The Kepler-289 system was observed continuously during Kepler Quarters 1–16,
with only long-cadence (30 minute integrations) data available for the first
eleven quarters, and short-cadence (1 minute integrations) data in the
remaining five. We obtained the Pre-search Data Conditioning Simple Aperture
Photometry (PDCSAP) flux from the Kepler-289 postage stamp in Quarters 1–16
from the Mikulski Archive for Space Telescopes (MAST), and we perform our own
reduction of the Kepler photometry, described further in the following
section. The PDCSAP photometry already has long-term instrumental trends
removed from the data using co-trending basis vectors.

There are also _TESS_ (Ricker et al., 2014) data available in sectors 14, 15,
and 41 for Kepler-289, which we downloaded using the lightkurve package
(Lightkurve Collaboration et al., 2018) and checked for transits of the outer
planet, Kepler-289c. We find that _TESS_ observed the same two transits that
we observed with WIRC, but the _TESS_ photometric precision is significantly
worse. We therefore exclude the _TESS_ data from our subsequent analysis.

###  2.2 Ground-based WIRC Photometry

Kepler-289c’s 126-day orbital period makes it a challenging target to observe
from the ground, as the transit has a total duration of $\sim$8 hours and
there are typically no opportunities to observe full transits of this planet
at Palomar. We obtained two partial transits of Kepler-289c in J band using
the Wide-field InfraRed Camera (WIRC) instrument at the prime focus of the
200" Hale Telescope at Palomar Observatory. Our first observation was taken on
on UT 24 August 2019 with a baseline spanning slightly more than half of the
total transit duration, beginning approximately an hour before the transit
center and ending approximately an hour after the end of egress. We observed a
second partial transit on UT 17 September 2021, beginning $\sim$2.5 hours pre-
ingress and observing until just past the mid-transit time.

WIRC has an 8.7$\arcmin\times 8.7\arcmin$ field of view, ensuring that there
are at least 10 comparison stars with magnitudes comparable to that of
Kepler-289 visible in the same field. We utilized a custom near-infrared beam-
shaping diffuser for these observations, which creates a top-hat PSF with a
full width at half maximum of 3$\arcsec$. This diffuser improved our observing
efficiency and mitigated time-correlated noise from PSF variations, increasing
our observing precision to levels comparable with space-based infrared
photometry for a star of this magnitude ($J=12.9$; Stefansson et al., 2017;
Vissapragada et al., 2020). We minimized the time-correlated noise
contribution from flat-fielding errors by utilizing custom guiding software,
which limits the pointing drift over the night to a few pixels (Zhao et al.,
2014). This software guides on science images by fitting 2D Gaussian profiles
to comparison stars and determining guiding offsets on each image. We observed
on both nights with an exposure time of 25 seconds, which we co-added to a
total exposure time of 50 seconds. On both nights we achieved a guiding
stability of less than two pixels (0$\farcs 5$) throughout the night. On the
first night, our observations began at airmass 2.5, reached a minimum airmass
of 1.02, and continued until airmass 2.5. On the second night, our
observations began near airmass 1.0 and continued until airmass 2.5 when the
target set.

Figure 2: _Kepler_ transit photometry of Kepler-289c. Our initial masks from
visual inspection of the transits are shown as grey shading, the final best-
fit transit model from the stacked transit profile is overplotted as a blue
curve, and our final 3$\sigma$ outlier masks are shown as red points. We
binned the short-cadence data to match the long cadence data (30 minute bins)
for this figure.

##  3 Data Analysis

###  3.1 Kepler Transit Fits and Stellar Activity Correction

The Kepler simple aperture photometry already has instrumental variations
removed, but there is significant astrophysical variability in the light curve
of Kepler-289 due to rotational modulation from starspots. This variability is
on the scale of $\sim$3%, while the transits of the inner two planets have
depths of approximately 0.4%, and the outer planet $\sim$1%. To obtain the
highest possible precision on the individual _Kepler_ transit times, we first
need to remove the astrophysical variability.

We began by using the transit mid-times reported in Holczer et al. (2016), or
Rowe et al. (2014) where mid-times are not available in Holczer et al. (2016),
to mask all of the data within $\pm$1.5 transit durations (for the relevant
planet) of each transit mid-time. We then broke up the light curve into
segments, using any gap in the data longer than a day as a break point. For
each segment, we fit the long-term trends in flux using a cubic B-spline with
knots defined by a smoothing factor of $m+\sqrt{2m}$, with $m$ being the total
number of data points (Dierckx, 1993), and interpolated the trend over the
masked transit points. We then divided out this trend to flatten the light
curve. We show a representative section of the original Kepler light curve
from quarter 5, displaying significant astrophysical variability, along with a
close-up of the trend and mask we used, in Figure 1.

After removing the stellar variability, our next step was to fit the phased
transit profiles for each planet. We used the transit mid-times from Schmitt
et al. (2014) to phase up the individual transits for each planet. For the
inner and middle planets, which have more shallow transits, no additional
steps were required to correct for stellar activity in the light curve for
fitting. For the outer planet, which has a relatively deep transit, we noted
that there appear to be “bump"-like features consistent with starspot
crossings (e.g Wolter et al., 2009; Davenport et al., 2014) in many of the
individual transit light curves. These features were most readily evident in
the second, fourth, sixth, and tenth _Kepler_ transits, and we masked them in
our initial transit fits. The ninth transit was of generally poor quality due
to instrumental effects present in the _Kepler_ photometry and did not have a
well-constrained transit mid-time in the Holczer catalog, so we excluded it
from the stacked transit profile. The first half of the first transit also
appeared to be affected by uncorrected instrumental effects, so we masked it
as well. After defining this initial set of masks, we fit the phased transit
profile. We then removed our initial masks for the likely spot crossings and
instead flagged all points that were more than 3$\sigma$ away from the best-
fit stacked transit model. We found that the majority of the outliers flagged
were indeed in the same regions of the light curves as our original spot
masks. We use this updated 3$\sigma$ outlier mask in the final joint-fit
between the phased _Kepler_ data and Palomar data. Masking these spot
crossings and instrumental outliers allowed us to obtain an improved
constraint on the transit depth and ensured that the transit mid-time
measurements were not biased. In Figure 2 we plot all of the individual
_Kepler_ transit observations of Kepler-289c, with our initial and final spot
masks over-plotted for reference.

Figure 3: The left panel shows the first partial transit observation in $J$
band from Palomar/WIRC in 2019, middle panel shows the second partial transit
observation from WIRC in 2021, and right panel shows the phased _Kepler_
transit. The red lines in the left and center panels are the best-fit transit
models for the WIRC light curves, and the red shaded region is the 1$\sigma$
confidence interval on that model. The blue line in the right panel shows the
best-fit transit model for the _Kepler_ data, along with blue shading for the
1$\sigma$ confidence interval. All three of these transit data sets are fit
jointly with exoplanet, where the _Kepler_ transit profile has separate depth
and limb-darkening parameters, and all other parameters are shared aside from
transit mid-times.

We modeled the transit light curves for the inner two planets using the
exoplanet light curve modeling package BATMAN (Kreidberg, 2015). We fit for
the planet to star radius ratio, orbital inclination, semi-major axis to
stellar radius ratio, and quadratic stellar limb-darkening parameters. We used
the results reported in Schmitt et al. (2014) as initial guesses for the
planet parameters, and sampled their posterior distributions using the affine-
invariant ensemble Markov chain Monte Carlo fitting package emcee (Foreman-
Mackey et al., 2013). We run the sampler with 50 walkers for
$5.5\times\mathrm{10}^{5}$ iterations and discard the initial
$5\times\mathrm{10}^{4}$ steps as burn-in. We checked for convergence by
verifying that the number of iterations for each parameter in our MCMC chain
exceeded 50 autocorrelation lengths as calculated by emcee. After fitting the
transit shape parameters, we retrieved the “best-fit" values from a $\chi^{2}$
minimization on the 50th percentile parameter set in our posterior chain, and
then used the 16th and 84th percentile parameter values from the posterior
distribution as our 68% $\pm$1$\sigma$ confidence interval. We then fit each
transit of the inner and middle planets in the _Kepler_ photometry with each
parameter from the best-fit transit profile fixed except for the transit time,
to re-derive transit times for both planets. These times are listed in Table
1, and the stacked transit profiles for Kepler-289b and d are shown in Figure
8. The radii we retrieve for Kepler-289b and Kepler-289d are slightly
different than the radii reported in Schmitt et al. (2014) (within 2$\sigma$
for the inner planet, and 1$\sigma$ for the middle planet), which we attribute
to our different stellar activity removal process.

Figure 4: Observed TTVs from _Kepler_ (black points) and Palomar/WIRC (red
points) for each of the Kepler-289 planets, along with 100 random posterior
draws from our TTV model (blue curves). The inner, middle, and outer planets
of Kepler-289b, Kepler-289d, and Kepler-289c respectively have mean orbital
periods of 34.55, 66.06, and 125.85 days. The planets are labeled in
alphabetical order of their discovery in the _Kepler_ data. The Palomar
transit observations extend the _Kepler_ TTV baseline by more than 7.5 years
and help refine the TTV super-period originally inferred from the _Kepler_
data to $\sim$1344 days. The reduced BJD time is BJD-2454000.

###  3.2 Palomar Photometry and Transit Fits for Kepler-289c

We dark-subtracted, flat-fielded, and corrected our WIRC images for bad pixels
and hot pixels using the methods described in Vissapragada et al. (2020). We
then used aperture photometry with the photutils package (Bradley et al.,
2020) to determine the relative fluxes of Kepler-289 and a set of comparison
stars in each image. We tested every aperture size from 5-25 pixels and
compared the root-mean-square deviation of the final light curves for each
aperture radius. We found that the optimal aperture radius was 9 pixels
($2\farcs 25$) for the first night , and 11 pixels ($2\farcs 75$) for the
second night. We calculated light curves for every star in the image whose
cumulative PSF had a signal-to-noise ratio of 100 or more relative to the sky
background, and selected the ten stars with the lowest median absolute
deviation relative to Kepler-289’s flux as our final set of comparison stars.
The full set of comparison stars are not identical across our two WIRC
observation nights, due to slightly different detector positioning on the sky.
We estimated and subtracted the sky background from each star using an
uncontaminated sky annulus with inner and outer radii of 25 and 50 pixels,
respectively.

We then constructed a systematic noise model that accounts for changes in
Kepler-289’s flux due to changing airmass, atmospheric seeing, telescope
pointing, and other instrumental effects. This systematics model is a linear
function of the ten comparison star light curves, the airmass, the PSF width,
the distance moved on the detector relative to the initial centroid position,
the local sky background, and the mean-subtracted times. For more details on
the WIRC data reduction pipeline and instrumental noise model, see
Vissapragada et al. (2020). The WIRC data reduction and light curve modeling
software are publicly available online at readthedocs.

We jointly fit the _Kepler_ and Palomar transit light curves using exoplanet.
The exoplanet package (Foreman-Mackey et al., 2021) uses starry (Luger et al.,
2019) to quickly calculate accurate limb-darkened light curves and PyMC3
(Salvatier et al., 2016) to explore the posterior distribution of modeled
parameters with No-U-Turn Sampling (Hoffman & Gelman, 2011), an extension of
the Hamiltonian Monte Carlo algorithm that utilizes first order gradient
information to step through posterior distributions quickly and efficiently.
We allowed the mid-time of each transit to vary individually in this fit using
the TTVOrbit model, with the stacked _Kepler_ transit profile shifted to a
reference time one orbital period before the first WIRC transit observation.
In our joint _Kepler_ /WIRC transit fit, we use global parameters for the
planetary orbital period, impact parameter, and semi-major axis to stellar
radius ratio. We fit for separate planet to star radius ratios and quadratic
limb darkening parameters in each bandpass. We also fit for a jitter parameter
describing the excess noise added in quadrature to the photon noise in each
bandpass. In the left and middle panels of Figure 3, we show the final WIRC
transit light curves and their residuals. In the right panel, we show the
stacked Kepler light curve. Our WIRC light curves result in a transit mid-time
precision of $\sim$4 minutes, compared to a median mid-time precision of 0.55
minutes from the space-based Kepler photometry. This is because the long
duration of the transit limits us to partial transit baselines when observing
with WIRC, which also has a slightly lower photometric precision than
_Kepler_. In Figure 9, we plot the posterior probability distributions for the
physically meaningful transit shape parameters from the joint fit (all
parameters are global other than ${\frac{R_{p}}{R_{*}}}$ in the Kepler and
WIRC bandpasses, and the $t_{0}$ parameter on each WIRC night).

Figure 5: Updated mass and radius measurements for the two inner sub-Neptunes
(Kepler-289b and d) compared to the original values from Schmitt et al.
(2014). We overplot constant composition curves for 100% iron, an earth-like
rock-iron mix, irradiated hydrogen-rich atmospheres on top of an Earth-like
rocky core, and irradiated water worlds from Zeng et al. (2016)

###  3.3 TTV Analysis

We fit our updated set of transit mid-time measurements using the TTVfast
package (Deck et al., 2014). TTVfast is a computationally efficient $n$-body
code that computes transit times as a function of the planetary masses and
orbital elements at a reference epoch, which we chose to be $T_{0}=2455865$
(BJD). We fixed the orbital inclinations to 90∘ and the longitudes of the
ascending nodes to 0∘ because our transit profile fits indicate that the
inclination dispersion is small, and the TTV solution is second order in
mutual inclination (Nesvorný & Vokrouhlický, 2014; Hadden & Lithwick, 2016).
We used emcee to map the posterior probability distributions for the masses
and orbital elements for this three-planet system. We ran the sampler with 50
walkers for $3.5\times\mathrm{10}^{6}$ iterations and discard the initial
$5\times\mathrm{10}^{5}$ steps as burn-in. We checked for convergence by
verifying that the number of iterations for each parameter in our MCMC chain
exceeded 50 autocorrelation lengths as calculated by emcee. Our observed TTVs,
as well as 100 random draws of the TTV posterior distribution from our TTVfast
fit, are shown in Figure 4, while the corner plot showing the posterior
probability distributions for the planet masses and eccentricity vectors is
shown in Figure 10. We use our improved dynamical solution to predict the
transit mid-times for all three planets until 2032, which are listed along
with our observed transit times in Table 1.

Table 1: All observed transit epochs, mid-times, and uncertainties for the three Kepler-289 planets, as well as the predicted mid-times and associated 1$\sigma$ uncertainties from our best-fit TTV model, extending until January 1st, 2032. The table is abbreviated here, but available in full online. Planet |  Transit Number |  Observed Mid-time (JD-2454000) |  Error (days) |  Predicted Mid-time (JD-2454000) |  Error (days)
---|---|---|---|---|---
Kepler-289b |  0 |  965.7135 |  0.0110 |  965.6898 |  0.0031
Kepler-289b |  1 |  - |  - |  1000.2328 |  0.0036
Kepler-289b |  2 |  - |  - |  1034.7792 |  0.0041
Kepler-289b |  3 |  1069.3174 |  0.0036 |  1069.3195 |  0.0054
Kepler-289b |  4 |  - |  - |  1103.8616 |  0.0056
Figure 6: Posterior probability distributions for the bulk metallicity,
planetary mass, and age of the outermost planet (Kepler-289c). We use a
planetary evolution model as described in Thorngren et al. (2016) to compute
the planet’s bulk metal content as a function of its observed radius, mass,
and age, which are taken from published values in the literature.

##  4 Results

Our new analysis decreases the mass uncertainties for all three planets by
more than a factor of 2, with the innermost planet seeing the greatest
improvement relative to the original constraints found in Schmitt et al.
(2014). We summarize the final planetary masses, radii, densities, and orbital
properties from our photometric and TTV analyses in Table 2. In this section,
we use these values to constrain the bulk densities and possible compositions
of the three planets.

We find that Kepler-289b is a 4.1 $M_{\oplus}$ low-density sub-Neptune, as
illustrated in Figure 5. Both the inner and middle planets are significantly
larger than a pure-water planet of the same mass. This implies that both
planets likely possess a primordial hydrogen- and helium-rich atmosphere,
which increases their apparent radii. We use the models in Lopez & Fortney
(2014) to calculate the core-to-atmosphere mass fraction for a scenario in
which we assume that the planets consist of Earth-like rocky cores surrounded
by a solar metallicity gas envelope. We find that the measured densities of
the inner and middle planets can be matched by hydrogen-rich envelopes that
constitute 3% and 0.9% of their total planet masses, respectively. Both of
these values are consistent with the estimated atmospheric mass fractions of
the broader sub-Neptune population (e.g., Owen & Wu, 2013; Lopez & Fortney,
2014; Lee, 2019).

Table 2: Orbital and planetary parameters for Kepler-289b, d, and c. Each of
these parameters is retrieved from our transit and TTV fits, other than the
derived parameters of $M_{p}$, $R_{p}$, and $\rho$, which incorporate errors
on the stellar mass and radius of $M_{\star}$ = 1.08 $\pm$ 0.02 $M_{\odot}$
and $R_{\star}$ = 1.00 $\pm$ 0.02 $R_{\odot}$ from Schmitt et al. (2014).

Kepler-289b
---
|  Schmitt et al. 2014 |  Kepler Reanalysis |  Kepler Reanalysis + Palomar
$P$ (days) |  34.5450±0.0005 |  $34.5383\begin{subarray}{c}+0.0006\\\ -0.0006\end{subarray}$ |  $34.5383\begin{subarray}{c}+0.0006\\\ -0.0006\end{subarray}$
$T_{0}$ (JD-2454000) |  965.6404$\pm$0.0040 |  $965.6880\begin{subarray}{c}+0.0030\\\ -0.0032\end{subarray}$ |  $965.6879\begin{subarray}{c}+0.0006\\\ -0.0006\end{subarray}$
$M_{p}$ $(M_{\oplus})$ |  7.3$\pm$6.8 |  $4.39\begin{subarray}{c}+4.32\\\ -2.30\end{subarray}$ |  $3.70\begin{subarray}{c}+3.79\\\ -1.96\end{subarray}$
$R_{p}$ $(R_{\oplus})$ |  2.15$\pm$0.10 |  2.49$\pm$0.07 |  2.49$\pm$0.07
$\rho$ (g cm-3) |  4.1$\pm$3.9 |  $1.56\begin{subarray}{c}+1.54\\\ -0.82\end{subarray}$ |  $1.45\begin{subarray}{c}+1.5\\\ -0.77\end{subarray}$
$R_{p}$/$R_{*}$ |  $0.0197\begin{subarray}{c}+0.0011\\\ -0.0006\end{subarray}$ |  $0.0228\begin{subarray}{c}+0.0004\\\ -0.0004\end{subarray}$ |  $0.0228\begin{subarray}{c}+0.0004\\\ -0.0004\end{subarray}$
$M_{p}$/$M_{*}$(×10-5) |  2.0$\pm$1.9 |  $1.22\begin{subarray}{c}+1.2\\\ -0.64\end{subarray}$ |  $1.13\begin{subarray}{c}+1.17\\\ -0.6\end{subarray}$
$a/R_{*}$ |  $71.1\begin{subarray}{c}+10\\\ -20\end{subarray}$ |  $46.38\begin{subarray}{c}+2.1\\\ -2.0\end{subarray}$ |  $46.38\begin{subarray}{c}+2.1\\\ -2.0\end{subarray}$
$i(^{\circ}$) |  $89.59\begin{subarray}{c}+0.30\\\ -0.48\end{subarray}$ |  $88.98\begin{subarray}{c}+0.06\\\ -0.07\end{subarray}$ |  $88.98\begin{subarray}{c}+0.06\\\ -0.07\end{subarray}$
$e\cos(\omega$) |  -0.0215$\pm$0.0255 |  - |  -
$e\sin(\omega$) |  -0.0113$\pm$0.0239 |  - |  -
$\sqrt{e}$cos($\omega$) |  - |  $-0.0364\begin{subarray}{c}+0.0802\\\ -0.0756\end{subarray}$ |  $-0.0333\begin{subarray}{c}+0.0757\\\ -0.0873\end{subarray}$
$\sqrt{e}$sin($\omega$) |  - |  $-0.1776\begin{subarray}{c}+0.0816\\\ -0.0564\end{subarray}$ |  $-0.1858\begin{subarray}{c}+0.0820\\\ -0.0564\end{subarray}$

Kepler-289d
---
|  Schmitt et al. 2014 |  Kepler Reanalysis |  Kepler Reanalysis + Palomar
$P$ (days) |  66.0634$\pm$0.0114 |  $66.0281\begin{subarray}{c}+0.0048\\\ -0.0042\end{subarray}$ |  $66.0282\begin{subarray}{c}+0.0044\\\ -0.0039\end{subarray}$
$T_{0}$ (JD-2454000) |  975.6436$\pm$0.0068 |  $975.6436\begin{subarray}{c}+0.0053\\\ -0.0053\end{subarray}$ |  $975.6240\begin{subarray}{c}+0.0051\\\ -0.0052\end{subarray}$
$M_{p}$ ($M_{\oplus}$) |  4.00$\pm$0.90 |  $5.97\begin{subarray}{c}+0.50\\\ -0.50\end{subarray}$ |  $5.33\begin{subarray}{c}+0.43\\\ -0.42\end{subarray}$
$R_{p}$ $(R_{\oplus})$ |  2.68$\pm$0.17 |  3.03$\pm$0.08 |  3.03$\pm$0.08
$\rho$ (g cm-3) |  1.2$\pm$0.3 |  $1.18\begin{subarray}{c}+0.1\\\ -0.1\end{subarray}$ |  $1.14\begin{subarray}{c}+0.09\\\ -0.09\end{subarray}$
$R_{p}$/$R_{\star}$ |  $0.0246\begin{subarray}{c}+0.0022\\\ -0.0009\end{subarray}$ |  $0.0270\begin{subarray}{c}+0.0005\\\ -0.0005\end{subarray}$ |  $0.0270\begin{subarray}{c}+0.0005\\\ -0.0005\end{subarray}$
$M_{p}$/$M_{\star}$(×10-5) |  1.1$\pm$0.2 |  $1.66\begin{subarray}{c}+0.14\\\ -0.14\end{subarray}$ |  $1.6\begin{subarray}{c}+0.13\\\ -0.13\end{subarray}$
$a/R_{\star}$ |  $117.8\begin{subarray}{c}+21\\\ -42\end{subarray}$ |  $70.52\begin{subarray}{c}+2.95\\\ -2.85\end{subarray}$ |  $70.52\begin{subarray}{c}+2.95\\\ -2.85\end{subarray}$
$i(^{\circ}$) |  $89.73\begin{subarray}{c}+0.20\\\ -0.38\end{subarray}$ |  $89.31\begin{subarray}{c}+0.04\\\ -0.04\end{subarray}$ |  $89.31\begin{subarray}{c}+0.04\\\ -0.04\end{subarray}$
$e\cos(\omega$) |  -0.0035$\pm$0.0022 |  - |  -
$e\sin(\omega$) |  -0.0108$\pm$0.0122 |  - |  -
$\sqrt{e}\cos(\omega$) |  - |  $0.0340\begin{subarray}{c}+0.0164\\\ -0.0208\end{subarray}$ |  $0.0293\begin{subarray}{c}+0.0174\\\ -0.0225\end{subarray}$
$\sqrt{e}\sin(\omega$) |  - |  $-0.0164\begin{subarray}{c}+0.0537\\\ -0.0498\end{subarray}$ |  $-0.0096\begin{subarray}{c}+0.0501\\\ -0.0500\end{subarray}$

Kepler-289c
---
|  Schmitt et al. 2014 |  Kepler Reanalysis |  Kepler Reanalysis + Palomar
$P$ (days) |  125.8518$\pm$0.0076 |  $125.8727\begin{subarray}{c}+0.0036\\\ -0.0022\end{subarray}$ |  $125.8723\begin{subarray}{c}+0.0035\\\ -0.0021\end{subarray}$
$T_{0}$ (JD-2454000) |  1069.6528$\pm$0.0077 |  $1069.6744\begin{subarray}{c}+0.0035\\\ -0.0023\end{subarray}$ |  $1069.6734\begin{subarray}{c}+0.0035\\\ -0.0021\end{subarray}$
$M_{p}(M_{\oplus})$ |  132$\pm$17 |  $170.35\begin{subarray}{c}+7.95\\\ -8.67\end{subarray}$ |  $157.18\begin{subarray}{c}+7.04\\\ -7.31\end{subarray}$111In units of $M_{Jup}$, Kepler-289c’s mass is 0.49 $\pm$ 0.02
$R_{p}(R_{\oplus})$ |  11.59$\pm$0.19 |  11.31$\pm$0.23 |  11.23$\pm$0.21
$\rho$ (g cm-3) |  0.47$\pm$0.06 |  $0.65\begin{subarray}{c}+0.03\\\ -0.03\end{subarray}$ |  $0.66\begin{subarray}{c}+0.03\\\ -0.03\end{subarray}$
$R_{p}$/$R_{\star}$ |  $0.10620\begin{subarray}{c}+0.00049\\\ -0.00050\end{subarray}$ |  $0.10373\begin{subarray}{c}+0.00040\\\ -0.00038\end{subarray}$ |  $0.10297\begin{subarray}{c}+0.00027\\\ -0.00026\end{subarray}$
$M_{p}$/$M_{\star}$(×10-5) |  36.43$\pm$4.66 |  $47.36\begin{subarray}{c}+2.21\\\ -2.41\end{subarray}$ |  $47.19\begin{subarray}{c}+2.11\\\ -2.23\end{subarray}$
$a/R_{\star}$ |  109.5$\pm$1.2 |  $104.7\begin{subarray}{c}+5.9\\\ -5.1\end{subarray}$ |  $108.95\begin{subarray}{c}+0.75\\\ -0.75\end{subarray}$
$i$ (∘) |  $89.794\begin{subarray}{c}+0.017\\\ -0.016\end{subarray}$ |  $89.74\begin{subarray}{c}+0.07\\\ -0.05\end{subarray}$ |  $89.78\begin{subarray}{c}+0.04\\\ -0.04\end{subarray}$
$e\cos(\omega$) |  0.0032$\pm$0.0066 |  - |  -
$e\sin(\omega$) |  0.0033$\pm$0.0086 |  - |  -
$\sqrt{e}$cos($\omega$) |  - |  $0.1160\begin{subarray}{c}+0.0152\\\ -0.0187\end{subarray}$ |  $0.1114\begin{subarray}{c}+0.0164\\\ -0.0194\end{subarray}$
$\sqrt{e}$sin($\omega$) |  - |  $0.0277\begin{subarray}{c}+0.0386\\\ -0.0382\end{subarray}$ |  $0.0399\begin{subarray}{c}+0.0278\\\ -0.0350\end{subarray}$

Figure 7: Total dust mass that reaches the inner disk as a function of the
assumed giant planet pebble isolation (core) mass (top x axis), and its
corresponding formation location (bottom x axis, this is set by the assumed
isolation mass). The dust mass is calculated using the disk models from
Chachan et al. (2022), where the red shading denotes the spread in predictions
caused by varying the assumed disk radii, masses, and dust-to-gas ratios. The
minimum estimated dust masses required to form the two inner planets in the
planetesimal (grey dashed line) and pebble accretion (black dashed line) are
overplotted. Kepler-289c’s current orbital semi-major axis is shown as a solid
black line.

For the outermost planet, Kepler-289c, we find an updated mass of 0.49
$\pm{0.02}$ $M_{Jup}$. We use a giant planet interior structure and evolution
model (Thorngren et al., 2016; Thorngren & Fortney, 2019) to calculate the
corresponding bulk metallicity as a function of the planet’s mass, age, and
radius. We allow these three parameters to vary freely in the fit with priors
based on their measured values from this study and Schmitt et al. (2014), and
show the resulting posterior probability distributions in Figure 6. For more
details on the giant planet structure model, see Thorngren & Fortney (2019).
We find that this planet has a bulk metallicity of 0.18 $\pm$ 0.04,
significantly higher than its approximately solar metallicity host star. This
corresponds to a bulk metal content of 30.5 $\pm$ 6.9 $M_{\oplus}$. We compare
Kepler-289c to the broader population of giant planets characterized in
Thorngren et al. (2016) and find that its bulk metallicity is slightly lower
than average based on the best-fit mass-metallicity trend model, but still
consistent with the overall population distribution.

Core accretion models for gas giant formation indicate that it is relatively
difficult to form planets like Kepler-289c in-situ on close-in orbits (Dawson
& Johnson, 2018; Poon et al., 2021). It is therefore possible that Kepler-289c
formed farther out and then migrated inward. Observational studies suggest
that the gas giant planet population peaks at separations between $1-10$ au
(Fulton et al., 2021), much farther out that Kepler-289c’s present-day orbital
separation of $\sim$0.5 au. Although the sample of outer gas giants with inner
super-Earths is much smaller, observational studies find that the gas giants
in these systems are also typically located on relatively wide orbits (Zhu &
Wu, 2018; Bryan et al., 2019). Kepler 289c’s lower-than-average bulk
metallicity also suggests a more distant formation location, as dust evolution
models indicate that the dust opacity should be lower (and the corresponding
gas accretion rates for a fixed core mass should be higher) beyond the ice
line (Chachan et al., 2021).

The fact that both planet pairs in this system have a normalized distance to
resonance $\Delta<0.05$ is also consistent with the hypothesis that they might
have migrated inward through the 2:1 resonance at some point in the past
(e.g., Weiss et al., 2022), although the current observational constraints
strongly disfavor a resonant configuration. Schmitt et al. (2014) calculated
the 3-body Laplace resonance width for this system, and found that both pairs
of planets are many tens of widths away from resonance. We empirically
estimate the three-body Laplace resonance angle using the method from Mills et
al. (2016) and find that it circulates from 0∘–360∘ over the course of the
_Kepler_ observations, with no indication of libration. We therefore conclude
that the Kepler-289 planets are not currently in an orbital resonance, in
agreement with Schmitt et al. (2014). This might be explained by post-nebular
dynamical instabilities, which can nudge resonant systems into configurations
that are either interior to or exterior to the original resonance (Goldberg &
Batygin, 2022).

###  4.1 Constraining the formation location of Kepler-289c

We next explore whether or not we can place more quantitative constraints on
the likely formation location of Kepler-289c using a pebble accretion model as
described in Chachan et al. (2022). This model assumes that the material
available in the inner disk alone is not sufficient to form a system of super-
Earths, but must instead be enhanced by a flow of small solids migrating in
from the outer disk. When the outer giant planet core reaches the pebble
isolation mass, it cuts off this flow of solids. Chachan et al. (2022) used
disk evolution models to show that the integrated dust mass that reaches the
inner disk in this scenario is primarily a function of the giant planet’s
formation location and corresponding isolation mass, with more distant
formation locations (higher isolation masses) allowing more material to reach
the inner disk.

As discussed in the previous section, the masses of the two inner planets in
the Kepler-289 system are dominated by solids. This means that they must have
accreted a combined total of approximately $\sim$10 $M_{\oplus}$ of solids. If
we compare this to the solid mass budget for the inner disk, using the minimum
mass extrasolar nebula (Chiang & Laughlin, 2013) to estimate the solid content
within the orbit of Kepler-289c yields a total solid mass of $\sim$8
$M_{\oplus}$. Clearly, some material must have been transported to the inner
disk to form Kepler-289b and d. If this material predominately came from
beyond the ice line, approximately 50% of the original solid mass would have
been lost to evaporation when the pebbles crossed the water ice line. The
remaining solids must accrete into larger bodies to avoid migrating all the
way onto the star. Drążkowska et al. (2016) combined dust evolution models
with planet formation models to show that dust is converted into planetesimals
with an efficiency of $\sim$23% in the inner disk. This implies that 85
$M_{\oplus}$ of solids must have been delivered to the inner disk in order to
form the two inner planets in the Kepler-289 system. If the inner super-Earths
formed via pebble accretion, the efficiency with which solids are accreted is
somewhat lower due to Kepler-289’s large stellar mass (see discussion in § 5.3
and Appendix A of Chachan et al., 2022), resulting in a higher mass threshold
of approximately 125 $M_{\oplus}$.

Since the total dust mass delivered to the inner disk is a function of the
giant planet’s formation location, we can use the corresponding pebble
isolation mass for a given formation location to place a lower limit on the
initial orbital distance of Kepler-289c. We run a new grid of disk models that
account for Kepler-289’s mass and its expected protostellar luminosity but is
otherwise identical to models presented in Chachan et al. (2022). This grid
spans a range of assumed disk radii, solid masses, and dust-to-gas ratios, and
we only consider the subset of disks where the giant planet core reaches the
isolation mass in less than 1 Myr. We show the resulting solid fluxes as a
function of the giant planet’s formation location and corresponding pebble
isolation mass in Figure 7. In these models, a giant planet core located just
outside 1 au, which has a corresponding pebble isolation mass of $\sim$5
$M_{\oplus}$, allows $\sim$85 $M_{\oplus}$ to be delivered to the inner disk.
This is enough material to form the inner super-Earths via planetesimal
accretion, but such a small core is unlikely to accrete enough gas to become a
gas giant (e.g., Rafikov, 2006; Lee, 2019). A core located just outside 3 au
would have an isolation mass of 10 $M_{\oplus}$, making it easier to accrete a
massive gas envelope and also providing enough solids to the inner disk to
exceed the higher pebble accretion threshold of 125 $M_{\oplus}$. A formation
location beyond 3 au would also be consistent with the semi-major axis
distribution of the broader population of giant planets measured in Fulton et
al. (2021), which peaks at 3.6${}^{+2.0}_{-1.8}$ au.

Kepler-289c has a total metal content of $\sim$30 $M_{\oplus}$ from our bulk
metallicity constraints, but if it has a structure similar to that of Jupiter
and Saturn it is likely that a significant fraction of these solids are
distributed throughout its interior (Wahl et al., 2017; Mankovich & Fuller,
2021). Such a distribution might plausibly be attributed to the continued
accretion of large solids (planetesimals) after the giant planet core reaches
the isolation mass (see Helled et al., 2022, for a review), allowing for a
core mass that is less than the planet’s total estimated solid mass. If a
larger fraction of Kepler-289c’s solids originated from its core, it would
imply a more distant formation location and a correspondingly larger flux of
solids to the inner disk. We therefore conclude that 3 au is a reasonable
lower limit for the likely formation location of Kepler-289c.

##  5 Conclusions

In this study, we used transit timing variations to obtain improved mass
measurements for the three planets in the Kepler-289 system, which have
orbital period ratios of 1.9:1 for each adjacent pair of planets. We re-
analyzed the _Kepler_ photometry for this system and observed two new transits
of the outermost planet, Kepler-289c, with WIRC on the 200" Hale Telescope at
Palomar Observatory. Our re-analysis of the _Kepler_ photometry allowed us to
better correct for the star’s high photometric variability, resulting in
modestly improved radius constraints and transit times. Our Palomar transit
observations extended the original four-year _Kepler_ TTV baseline by an
additional 7.5 years, allowing for an improved dynamical solution. Our updated
mass uncertainties for all three planets are more than a factor of two smaller
than the original values reported by Schmitt et al. (2014).

We use our improved mass and radii constraints to establish that the sub-
Neptune sized inner and middle planets, Kepler-289b and d, are both low-
density (1.45 and 1.14 g cm-3) planets. If we assume that these planets both
have Earth-like cores with solar metallicity envelopes, we calculate that they
should have atmospheric mass fractions of 3.0% and 0.9%, respectively. This is
consistent with their location on the upper side of the radius valley. Since
Kepler-289 is relatively young, it is possible that these two planets may host
present-day atmospheric outflows. Although this system is too faint to measure
helium outflow signatures with current ground-based telescopes, it might be
accessible to upcoming thirty-meter class telescopes.

We calculate the three-body Laplace resonance angles for the system and find
that Kepler-289 is not in the 1:2:4 Laplace orbital resonance, in agreement
with Schmitt et al. (2014). We argue that this system might have migrated into
a resonant chain, which was later disrupted by dynamical instabilities. This
is consistent with both observations and models of giant planet formation,
which suggest that planets like Kepler-289c (0.5 $M_{Jup}$, 0.5 au present-day
location) form more readily outside 1 au. We combine our updated planet masses
with disk evolution models to constrain the formation location of Kepler-289c,
and conclude that it likely originated at or beyond 3 au.

Transiting, dynamically interacting multi-planet systems provide us with a
wealth of information about planet formation and migration processes.
Kepler-289’s unique architecture makes it a particularly valuable test case
for exploring the connections between close-in sup-Neptune planets and outer
gas giant companions. Although its relative faintness also makes it a
challenging target for the _James Webb Space Telescope_ , atmospheric
composition studies of Kepler-289c with next-generation space telescopes could
provide additional constraints on its formation location (Madhusudhan, 2019).

##  6 Acknowledgements

We thank the Palomar Observatory telescope operators, support astronomers, and
directorate for their support of this work, especially Kajse Peffer, Kevin
Rykoski, Carolyn Heffner, Andy Boden, and Tom Barlow. Part of this program is
supported by JPL Hale telescope time allocation. Some of the data presented in
this paper were obtained from the Mikulski Archive for Space Telescopes
(MAST). STScI is operated by the Association of Universities for Research in
Astronomy, Inc., under NASA contract NAS5-26555. Support for MAST for non-HST
data is provided by the NASA Office of Space Science via grant NNX13AC07G and
by other grants and contracts. H.A.K. acknowledges support from NSF CAREER
grant 1555095. SV is supported by an NSF Graduate Research Fellowship. This
research made use of Lightkurve, a Python package for Kepler and TESS data
analysis (Lightkurve Collaboration et al., 2018). This research made use of
exoplanet (Foreman-Mackey et al., 2021) and its dependencies (Agol et al.,
2020; Kumar et al., 2019; Astropy Collaboration et al., 2013, 2018a; Kipping,
2013; Luger et al., 2019; Salvatier et al., 2016; Theano Development Team,
2016).

## References

  * sci (2001) 2001, SciPy: Open source scientific tools for Python.  http://www.scipy.org/
  * Agol & Fabrycky (2018) Agol, E., & Fabrycky, D. C. 2018, Handbook of Exoplanets, 797–816, doi: 10.1007/978-3-319-55333-7_7
  * Agol et al. (2020) Agol, E., Luger, R., & Foreman-Mackey, D. 2020, AJ, 159, 123, doi: 10.3847/1538-3881/ab4fee
  * Agol et al. (2005) Agol, E., Steffen, J., Sari, R., & Clarkson, W. 2005, MNRAS, 359, 567, doi: 10.1111/j.1365-2966.2005.08922.x
  * Akeson et al. (2013) Akeson, R. L., Chen, X., Ciardi, D., et al. 2013, Publications of the Astronomical Society of the Pacific, 125, 989–999, doi: 10.1086/672273
  * Astropy Collaboration et al. (2013) Astropy Collaboration, Robitaille, T. P., Tollerud, E. J., et al. 2013, A&A, 558, A33, doi: 10.1051/0004-6361/201322068
  * Astropy Collaboration et al. (2018a) Astropy Collaboration, Price-Whelan, A. M., Sipőcz, B. M., et al. 2018a, AJ, 156, 123, doi: 10.3847/1538-3881/aabc4f
  * Astropy Collaboration et al. (2018b) Astropy Collaboration, Price-Whelan, A. M., Sipőcz, B. M., et al. 2018b, AJ, 156, 123, doi: 10.3847/1538-3881/aabc4f
  * Borucki et al. (2010) Borucki, W. J., Koch, D., Basri, G., et al. 2010, Science, 327, 977, doi: 10.1126/science.1185402
  * Bradley et al. (2020) Bradley, L., Sipőcz, B., Robitaille, T., et al. 2020, astropy/photutils: 1.0.0, 1.0.0, Zenodo, doi: 10.5281/zenodo.4044744
  * Bryan et al. (2019) Bryan, M. L., Knutson, H. A., Lee, E. J., et al. 2019, The Astronomical Journal, 157, 52, doi: 10.3847/1538-3881/aaf57f
  * Chachan et al. (2021) Chachan, Y., Lee, E. J., & Knutson, H. A. 2021, ApJ, 919, 63, doi: 10.3847/1538-4357/ac0bb6
  * Chachan et al. (2022) Chachan, Y., Dalba, P. A., Knutson, H. A., et al. 2022, ApJ, 926, 62, doi: 10.3847/1538-4357/ac3ed6
  * Charalambous et al. (2022) Charalambous, C., Teyssandier, J., & Libert, A.-S. 2022, Monthly Notices of the Royal Astronomical Society, 514, 3844, doi: 10.1093/mnras/stac1554
  * Chiang & Laughlin (2013) Chiang, E., & Laughlin, G. 2013, Monthly Notices of the Royal Astronomical Society, 431, 3444, doi: 10.1093/mnras/stt424
  * Davenport et al. (2014) Davenport, J. R. A., Hebb, L., & Hawley, S. L. 2014, Using Transiting Planets to Model Starspot Evolution.  https://arxiv.org/abs/1408.5201
  * Dawson & Johnson (2018) Dawson, R. I., & Johnson, J. A. 2018, ARA&A, 56, 175, doi: 10.1146/annurev-astro-081817-051853
  * Deck & Agol (2016) Deck, K. M., & Agol, E. 2016, The Astrophysical Journal, 821, 96, doi: 10.3847/0004-637x/821/2/96
  * Deck et al. (2014) Deck, K. M., Agol, E., Holman, M. J., & Nesvorný, D. 2014, The Astrophysical Journal, 787, 132, doi: 10.1088/0004-637x/787/2/132
  * Dierckx (1993) Dierckx, P. 1993, Curve and surface fitting with splines (Oxford University Press)
  * Drążkowska et al. (2016) Drążkowska, J., Alibert, Y., & Moore, B. 2016, A&A, 594, A105, doi: 10.1051/0004-6361/201628983
  * Fabrycky et al. (2014) Fabrycky, D. C., Lissauer, J. J., Ragozzine, D., et al. 2014, ApJ, 790, 146, doi: 10.1088/0004-637X/790/2/146
  * Foreman-Mackey (2016) Foreman-Mackey, D. 2016, The Journal of Open Source Software, 24, doi: 10.21105/joss.00024
  * Foreman-Mackey et al. (2013) Foreman-Mackey, D., Hogg, D. W., Lang, D., & Goodman, J. 2013, Publications of the Astronomical Society of the Pacific, 125, 306–312, doi: 10.1086/670067
  * Foreman-Mackey et al. (2021) Foreman-Mackey, D., Luger, R., Agol, E., et al. 2021, arXiv e-prints, arXiv:2105.01994.  https://arxiv.org/abs/2105.01994
  * Fulton et al. (2021) Fulton, B. J., Rosenthal, L. J., Hirsch, L. A., et al. 2021, ApJS, 255, 14, doi: 10.3847/1538-4365/abfcc1
  * Goldberg & Batygin (2022) Goldberg, M., & Batygin, K. 2022, arXiv e-prints, arXiv:2203.00801.  https://arxiv.org/abs/2203.00801
  * Hadden & Lithwick (2016) Hadden, S., & Lithwick, Y. 2016, ApJ, 828, 44, doi: 10.3847/0004-637X/828/1/44
  * Hadden & Lithwick (2017) Hadden, S., & Lithwick, Y. 2017, The Astronomical Journal, 154, 5, doi: 10.3847/1538-3881/aa71ef
  * Helled et al. (2022) Helled, R., Stevenson, D. J., Lunine, J. I., et al. 2022, Icarus, 378, 114937, doi: 10.1016/j.icarus.2022.114937
  * Hoffman & Gelman (2011) Hoffman, M. D., & Gelman, A. 2011, The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo.  https://arxiv.org/abs/1111.4246
  * Holczer et al. (2016) Holczer, T., Mazeh, T., Nachmani, G., et al. 2016, The Astrophysical Journal Supplement Series, 225, 9, doi: 10.3847/0067-0049/225/1/9
  * Holman & Murray (2005) Holman, M. J., & Murray, N. W. 2005, Science, 307, 1288, doi: 10.1126/science.1107822
  * Hunter (2007) Hunter, J. D. 2007, Computing In Science & Engineering, 9, 90
  * Jontof-Hutter et al. (2021) Jontof-Hutter, D., Wolfgang, A., Ford, E. B., et al. 2021, AJ, 161, 246, doi: 10.3847/1538-3881/abd93f
  * Jontof-Hutter et al. (2016) Jontof-Hutter, D., Ford, E. B., Rowe, J. F., et al. 2016, ApJ, 820, 39, doi: 10.3847/0004-637X/820/1/39
  * Kipping (2013) Kipping, D. M. 2013, MNRAS, 435, 2152, doi: 10.1093/mnras/stt1435
  * Kreidberg (2015) Kreidberg, L. 2015, Publications of the Astronomical Society of the Pacific, 127, 1161
  * Kumar et al. (2019) Kumar, R., Carroll, C., Hartikainen, A., & Martin, O. A. 2019, The Journal of Open Source Software, doi: 10.21105/joss.01143
  * Lee (2019) Lee, E. J. 2019, The Astrophysical Journal, 878, 36, doi: 10.3847/1538-4357/ab1b40
  * Lee (2019) Lee, E. J. 2019, ApJ, 878, 36, doi: 10.3847/1538-4357/ab1b40
  * Lightkurve Collaboration et al. (2018) Lightkurve Collaboration, Cardoso, J. V. d. M., Hedges, C., et al. 2018, Lightkurve: Kepler and TESS time series analysis in Python, Astrophysics Source Code Library.  http://ascl.net/1812.013
  * Lithwick et al. (2012) Lithwick, Y., Xie, J., & Wu, Y. 2012, The Astrophysical Journal, 761, 122, doi: 10.1088/0004-637x/761/2/122
  * Lopez & Fortney (2014) Lopez, E. D., & Fortney, J. J. 2014, ApJ, 792, 1, doi: 10.1088/0004-637X/792/1/1
  * Luger et al. (2019) Luger, R., Agol, E., Foreman-Mackey, D., et al. 2019, The Astronomical Journal, 157, 64, doi: 10.3847/1538-3881/aae8e5
  * Luger et al. (2019) Luger, R., Agol, E., Foreman-Mackey, D., et al. 2019, AJ, 157, 64, doi: 10.3847/1538-3881/aae8e5
  * Madhusudhan (2019) Madhusudhan, N. 2019, ARA&A, 57, 617, doi: 10.1146/annurev-astro-081817-051846
  * Mankovich & Fuller (2021) Mankovich, C. R., & Fuller, J. 2021, Nature Astronomy, 5, 1103, doi: 10.1038/s41550-021-01448-3
  * Mills et al. (2016) Mills, S. M., Fabrycky, D. C., Migaszewski, C., et al. 2016, Nature, 533, 509–512, doi: 10.1038/nature17445
  * Nesvorný & Vokrouhlický (2014) Nesvorný, D., & Vokrouhlický, D. 2014, ApJ, 790, 58, doi: 10.1088/0004-637X/790/1/58
  * Owen & Wu (2013) Owen, J. E., & Wu, Y. 2013, ApJ, 775, 105, doi: 10.1088/0004-637X/775/2/105
  * Poon et al. (2021) Poon, S. T. S., Nelson, R. P., & Coleman, G. A. L. 2021, Monthly Notices of the Royal Astronomical Society, 505, 2500, doi: 10.1093/mnras/stab1466
  * Rafikov (2006) Rafikov, R. R. 2006, ApJ, 648, 666, doi: 10.1086/505695
  * Rein & Tamayo (2015) Rein, H., & Tamayo, D. 2015, Monthly Notices of the Royal Astronomical Society, 452, 376–388, doi: 10.1093/mnras/stv1257
  * Ricker et al. (2014) Ricker, G. R., Winn, J. N., Vanderspek, R., et al. 2014, Journal of Astronomical Telescopes, Instruments, and Systems, 1, 014003, doi: 10.1117/1.jatis.1.1.014003
  * Rowe et al. (2014) Rowe, J. F., Bryson, S. T., Marcy, G. W., et al. 2014, ApJ, 784, 45, doi: 10.1088/0004-637X/784/1/45
  * Salvatier et al. (2016) Salvatier, J., Wiecki, T. V., & Fonnesbeck, C. 2016, PeerJ Computer Science, 2, e55
  * Schmitt et al. (2014) Schmitt, J. R., Agol, E., Deck, K. M., et al. 2014, The Astrophysical Journal, 795, 167, doi: 10.1088/0004-637x/795/2/167
  * Stefansson et al. (2017) Stefansson, G., Mahadevan, S., Hebb, L., et al. 2017, ApJ, 848, 9, doi: 10.3847/1538-4357/aa88aa
  * Theano Development Team (2016) Theano Development Team. 2016, arXiv e-prints, abs/1605.02688.  http://arxiv.org/abs/1605.02688
  * Thorngren & Fortney (2019) Thorngren, D., & Fortney, J. J. 2019, The Astrophysical Journal, 874, L31, doi: 10.3847/2041-8213/ab1137
  * Thorngren et al. (2016) Thorngren, D. P., Fortney, J. J., Murray-Clay, R. A., & Lopez, E. D. 2016, ApJ, 831, 64, doi: 10.3847/0004-637X/831/1/64
  * Vissapragada et al. (2020) Vissapragada, S., Jontof-Hutter, D., Shporer, A., et al. 2020, The Astronomical Journal, 159, 108, doi: 10.3847/1538-3881/ab65c8
  * Wahl et al. (2017) Wahl, S. M., Hubbard, W. B., Militzer, B., et al. 2017, Geophys. Res. Lett., 44, 4649, doi: 10.1002/2017GL073160
  * Weiss et al. (2022) Weiss, L. M., Millholland, S. C., Petigura, E. A., et al. 2022, arXiv e-prints, arXiv:2203.10076.  https://arxiv.org/abs/2203.10076
  * Wolter et al. (2009) Wolter, U., Schmitt, J. H. M. M., Huber, K. F., et al. 2009, Astronomy & Astrophysics, 504, 561–564, doi: 10.1051/0004-6361/200912329
  * Zeng et al. (2016) Zeng, L., Sasselov, D. D., & Jacobsen, S. B. 2016, The Astrophysical Journal, 819, 127, doi: 10.3847/0004-637x/819/2/127
  * Zhao et al. (2014) Zhao, M., O’Rourke, J. G., Wright, J. T., et al. 2014, The Astrophysical Journal, 796, 115, doi: 10.1088/0004-637x/796/2/115
  * Zhu & Wu (2018) Zhu, W., & Wu, Y. 2018, The Astronomical Journal, 156, 92, doi: 10.3847/1538-3881/aad22a

##  Appendix A Transit and TTV Model Plots and Posterior Distributions

Figure 8: Gray points show the phased _Kepler_ data. The left and right panels
are the inner and middle planets, respectively. Black points are binned to 15
minutes. The red shading indicates the middle 68% range from our posterior
distribution. Figure 9: Posterior probability distributions for the transit
shape parameters from the MCMC joint fit to the _Kepler_ and WIRC data. For
ease of viewing in the plot, we label the period minus 100 days as the reduced
period. $t_{1}$ is the BJD transit center time of the first WIRC night -
2458719 days, and $t_{2}$ is the BJD transit center time of the second WIRC
night - 2459474 days. Figure 10: Posterior probability distributions for the
planet masses and eccentricity vectors from the TTV fit. We also allowed the
orbital periods and initial transit times for each planet to vary as free
parameters in this fit, but we omit them from this plot for ease of viewing.